[==========] Running 56 tests from 4 test suites.
[----------] Global test environment set-up.
[----------] 30 tests from SecurityITest
[ RUN ] SecurityITest.TestAuthorizationOnListTablets
Loading random data
Initializing database '/tmp/dist-test-taskgH5Pjm/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786277382073938-888-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 09 12:09:42 dist-test-slave-2w3w krb5kdc[897](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 09 12:09:42 dist-test-slave-2w3w krb5kdc[897](info): set up 2 sockets
Aug 09 12:09:42 dist-test-slave-2w3w krb5kdc[897](info): commencing operation
krb5kdc: starting...
WARNING: Logging before InitGoogleLogging() is written to STDERR
W20260809 12:09:43.344029 888 mini_kdc.cc:121] Time spent starting KDC: real 1.141s user 0.000s sys 0.025s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskgH5Pjm/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786277382073938-888-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskgH5Pjm/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786277382073938-888-0/krb5kdc/test-user.keytab.
Aug 09 12:09:43 dist-test-slave-2w3w krb5kdc[897](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786277383, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-09T12:09:43Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-09T12:09:43Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.222.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.222.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.222.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskgH5Pjm/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786277382073938-888-0/krb5kdc/kudu_127.0.222.62.keytab.
Entry for principal kudu/127.0.222.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskgH5Pjm/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786277382073938-888-0/krb5kdc/kudu_127.0.222.62.keytab.
I20260809 12:09:43.539731 888 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskgH5Pjm/build/asan/bin/kudu
/tmp/dist-test-taskgH5Pjm/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskgH5Pjm/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786277382073938-888-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskgH5Pjm/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786277382073938-888-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskgH5Pjm/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786277382073938-888-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskgH5Pjm/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786277382073938-888-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.222.62:42811
--webserver_interface=127.0.222.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskgH5Pjm/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786277382073938-888-0/krb5kdc/kudu_127.0.222.62.keytab
--principal=kudu/127.0.222.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.222.20:39985
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.222.62:42811
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskgH5Pjm/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786277382073938-888-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskgH5Pjm/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786277382073938-888-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskgH5Pjm/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786277382073938-888-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260809 12:09:43.898867 911 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260809 12:09:43.899292 911 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260809 12:09:43.899401 911 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260809 12:09:43.906458 911 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260809 12:09:43.906553 911 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260809 12:09:43.906598 911 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260809 12:09:43.906656 911 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260809 12:09:43.906699 911 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260809 12:09:43.917943 911 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.222.20:39985
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskgH5Pjm/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786277382073938-888-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskgH5Pjm/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786277382073938-888-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.222.62:42811
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskgH5Pjm/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786277382073938-888-0/krb5kdc/kudu_127.0.222.62.keytab
--principal=kudu/127.0.222.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.222.62:42811
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskgH5Pjm/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786277382073938-888-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.222.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskgH5Pjm/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786277382073938-888-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 09 Aug 2026 11:06:41 UTC on ccycloud.quasar-bwoogy.root.comops.site
build id 4575991
ASAN enabled
I20260809 12:09:43.919957 911 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260809 12:09:43.921770 911 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260809 12:09:43.950452 917 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 12:09:43.950445 920 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 12:09:43.950452 916 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260809 12:09:43.953547 911 server_base.cc:1034] running on GCE node
I20260809 12:09:43.954732 911 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260809 12:09:43.957106 911 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260809 12:09:43.958412 911 hybrid_clock.cc:648] HybridClock initialized: now 1786277383958341 us; error 89 us; skew 500 ppm
I20260809 12:09:43.959043 911 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 09 12:09:43 dist-test-slave-2w3w krb5kdc[897](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786277383, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.222.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260809 12:09:43.962749 911 init.cc:377] Logged in from keytab as kudu/127.0.222.62@KRBTEST.COM (short username kudu)
I20260809 12:09:43.965312 911 webserver.cc:460] Webserver started at http://127.0.222.62:40345/ using document root <none> and password file <none>
I20260809 12:09:43.966251 911 fs_manager.cc:359] Metadata directory not provided
I20260809 12:09:43.966394 911 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260809 12:09:43.966748 911 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260809 12:09:43.971073 911 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskgH5Pjm/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786277382073938-888-0/minicluster-data/master-0/data/instance:
uuid: "9bcf638ad26340dcbb12d7eec08642a3"
format_stamp: "Formatted at 2026-08-09 12:09:43 on dist-test-slave-2w3w"
server_key: "f8120e023ed4083106b8c4fd7ce7fd36"
server_key_iv: "940c407b628ac76ecc12af47b15b2efb"
server_key_version: "encryptionkey@0"
I20260809 12:09:43.972122 911 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskgH5Pjm/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786277382073938-888-0/minicluster-data/master-0/wal/instance:
uuid: "9bcf638ad26340dcbb12d7eec08642a3"
format_stamp: "Formatted at 2026-08-09 12:09:43 on dist-test-slave-2w3w"
server_key: "f8120e023ed4083106b8c4fd7ce7fd36"
server_key_iv: "940c407b628ac76ecc12af47b15b2efb"
server_key_version: "encryptionkey@0"
I20260809 12:09:43.979166 911 fs_manager.cc:691] Time spent creating directory manager: real 0.007s user 0.006s sys 0.001s
I20260809 12:09:43.983462 929 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260809 12:09:43.984952 911 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260809 12:09:43.985240 911 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskgH5Pjm/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786277382073938-888-0/minicluster-data/master-0/data,/tmp/dist-test-taskgH5Pjm/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786277382073938-888-0/minicluster-data/master-0/wal
uuid: "9bcf638ad26340dcbb12d7eec08642a3"
format_stamp: "Formatted at 2026-08-09 12:09:43 on dist-test-slave-2w3w"
server_key: "f8120e023ed4083106b8c4fd7ce7fd36"
server_key_iv: "940c407b628ac76ecc12af47b15b2efb"
server_key_version: "encryptionkey@0"
I20260809 12:09:43.985459 911 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskgH5Pjm/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786277382073938-888-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskgH5Pjm/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786277382073938-888-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskgH5Pjm/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786277382073938-888-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260809 12:09:44.013657 911 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260809 12:09:44.019086 911 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260809 12:09:44.019522 911 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260809 12:09:44.042878 911 rpc_server.cc:307] RPC server started. Bound to: 127.0.222.62:42811
I20260809 12:09:44.042922 985 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.222.62:42811 every 8 connection(s)
I20260809 12:09:44.045246 911 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskgH5Pjm/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786277382073938-888-0/minicluster-data/master-0/data/info.pb
I20260809 12:09:44.051682 986 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260809 12:09:44.052793 888 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskgH5Pjm/build/asan/bin/kudu as pid 911
I20260809 12:09:44.053153 888 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskgH5Pjm/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786277382073938-888-0/minicluster-data/master-0/wal/instance
I20260809 12:09:44.055269 888 external_mini_cluster.cc:1428] Setting key d238242814fe221b2c92eed756cdd71c
I20260809 12:09:44.066602 986 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 9bcf638ad26340dcbb12d7eec08642a3: Bootstrap starting.
I20260809 12:09:44.071915 986 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 9bcf638ad26340dcbb12d7eec08642a3: Neither blocks nor log segments found. Creating new log.
I20260809 12:09:44.074167 986 log.cc:824] T 00000000000000000000000000000000 P 9bcf638ad26340dcbb12d7eec08642a3: Log is configured to *not* fsync() on all Append() calls
I20260809 12:09:44.078415 986 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 9bcf638ad26340dcbb12d7eec08642a3: No bootstrap required, opened a new log
I20260809 12:09:44.090045 986 raft_consensus.cc:348] T 00000000000000000000000000000000 P 9bcf638ad26340dcbb12d7eec08642a3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9bcf638ad26340dcbb12d7eec08642a3" member_type: VOTER last_known_addr { host: "127.0.222.62" port: 42811 } }
I20260809 12:09:44.090786 986 raft_consensus.cc:374] T 00000000000000000000000000000000 P 9bcf638ad26340dcbb12d7eec08642a3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260809 12:09:44.091065 986 raft_consensus.cc:729] T 00000000000000000000000000000000 P 9bcf638ad26340dcbb12d7eec08642a3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9bcf638ad26340dcbb12d7eec08642a3, State: Initialized, Role: FOLLOWER
Aug 09 12:09:44 dist-test-slave-2w3w krb5kdc[897](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786277383, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.222.62@KRBTEST.COM
I20260809 12:09:44.092331 986 consensus_queue.cc:260] T 00000000000000000000000000000000 P 9bcf638ad26340dcbb12d7eec08642a3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9bcf638ad26340dcbb12d7eec08642a3" member_type: VOTER last_known_addr { host: "127.0.222.62" port: 42811 } }
I20260809 12:09:44.092701 986 raft_consensus.cc:388] T 00000000000000000000000000000000 P 9bcf638ad26340dcbb12d7eec08642a3 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260809 12:09:44.093053 986 raft_consensus.cc:482] T 00000000000000000000000000000000 P 9bcf638ad26340dcbb12d7eec08642a3 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260809 12:09:44.093339 986 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 9bcf638ad26340dcbb12d7eec08642a3 [term 0 FOLLOWER]: Advancing to term 1
I20260809 12:09:44.097334 986 raft_consensus.cc:504] T 00000000000000000000000000000000 P 9bcf638ad26340dcbb12d7eec08642a3 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9bcf638ad26340dcbb12d7eec08642a3" member_type: VOTER last_known_addr { host: "127.0.222.62" port: 42811 } }
I20260809 12:09:44.098487 986 leader_election.cc:302] T 00000000000000000000000000000000 P 9bcf638ad26340dcbb12d7eec08642a3 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 9bcf638ad26340dcbb12d7eec08642a3; no voters:
I20260809 12:09:44.099525 986 leader_election.cc:288] T 00000000000000000000000000000000 P 9bcf638ad26340dcbb12d7eec08642a3 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260809 12:09:44.099838 991 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 9bcf638ad26340dcbb12d7eec08642a3 [term 1 FOLLOWER]: Leader election won for term 1
I20260809 12:09:44.102258 991 raft_consensus.cc:686] T 00000000000000000000000000000000 P 9bcf638ad26340dcbb12d7eec08642a3 [term 1 LEADER]: Becoming Leader. State: Replica: 9bcf638ad26340dcbb12d7eec08642a3, State: Running, Role: LEADER
I20260809 12:09:44.103600 991 consensus_queue.cc:237] T 00000000000000000000000000000000 P 9bcf638ad26340dcbb12d7eec08642a3 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9bcf638ad26340dcbb12d7eec08642a3" member_type: VOTER last_known_addr { host: "127.0.222.62" port: 42811 } }
I20260809 12:09:44.104650 986 sys_catalog.cc:564] T 00000000000000000000000000000000 P 9bcf638ad26340dcbb12d7eec08642a3 [sys.catalog]: configured and running, proceeding with master startup.
I20260809 12:09:44.106395 988 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 12:09:44.060929 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51604 (local address 127.0.222.62:42811)
0809 12:09:44.062154 (+ 1225us) server_negotiation.cc:206] Beginning negotiation
0809 12:09:44.062171 (+ 17us) server_negotiation.cc:399] Waiting for connection header
0809 12:09:44.064830 (+ 2659us) server_negotiation.cc:407] Connection header received
0809 12:09:44.075267 (+ 10437us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 12:09:44.075286 (+ 19us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 12:09:44.075882 (+ 596us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 12:09:44.076356 (+ 474us) server_negotiation.cc:226] Negotiated authn=SASL
0809 12:09:44.079552 (+ 3196us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 12:09:44.082299 (+ 2747us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 12:09:44.084150 (+ 1851us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 12:09:44.085400 (+ 1250us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 12:09:44.085726 (+ 326us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 12:09:44.092993 (+ 7267us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 12:09:44.093077 (+ 84us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 12:09:44.093122 (+ 45us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0809 12:09:44.093193 (+ 71us) server_negotiation.cc:922] calling sasl_server_start()
0809 12:09:44.097525 (+ 4332us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 12:09:44.098545 (+ 1020us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 12:09:44.098555 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 12:09:44.098575 (+ 20us) server_negotiation.cc:967] calling sasl_server_step()
0809 12:09:44.098927 (+ 352us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 12:09:44.099589 (+ 662us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 12:09:44.099597 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 12:09:44.099603 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0809 12:09:44.100844 (+ 1241us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 12:09:44.101359 (+ 515us) server_negotiation.cc:1035] Waiting for connection context
0809 12:09:44.104431 (+ 3072us) server_negotiation.cc:299] Negotiation successful
0809 12:09:44.104784 (+ 353us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":802,"thread_start_us":411,"threads_started":1}
I20260809 12:09:44.115643 993 sys_catalog.cc:455] T 00000000000000000000000000000000 P 9bcf638ad26340dcbb12d7eec08642a3 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 9bcf638ad26340dcbb12d7eec08642a3. Latest consensus state: current_term: 1 leader_uuid: "9bcf638ad26340dcbb12d7eec08642a3" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9bcf638ad26340dcbb12d7eec08642a3" member_type: VOTER last_known_addr { host: "127.0.222.62" port: 42811 } } }
I20260809 12:09:44.115995 992 sys_catalog.cc:455] T 00000000000000000000000000000000 P 9bcf638ad26340dcbb12d7eec08642a3 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "9bcf638ad26340dcbb12d7eec08642a3" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9bcf638ad26340dcbb12d7eec08642a3" member_type: VOTER last_known_addr { host: "127.0.222.62" port: 42811 } } }
I20260809 12:09:44.116765 992 sys_catalog.cc:458] T 00000000000000000000000000000000 P 9bcf638ad26340dcbb12d7eec08642a3 [sys.catalog]: This master's current role is: LEADER
I20260809 12:09:44.116576 993 sys_catalog.cc:458] T 00000000000000000000000000000000 P 9bcf638ad26340dcbb12d7eec08642a3 [sys.catalog]: This master's current role is: LEADER
I20260809 12:09:44.126693 998 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260809 12:09:44.135604 998 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260809 12:09:44.149895 998 catalog_manager.cc:1347] Generated new cluster ID: 7c8c61e8f533406eb0f687c166635294
I20260809 12:09:44.150038 998 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260809 12:09:44.166281 998 catalog_manager.cc:1370] Generated new certificate authority record
I20260809 12:09:44.167867 998 catalog_manager.cc:1504] Loading token signing keys...
I20260809 12:09:44.180289 998 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 9bcf638ad26340dcbb12d7eec08642a3: Generated new TSK 0
I20260809 12:09:44.181285 998 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.0.222.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.222.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.222.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskgH5Pjm/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786277382073938-888-0/krb5kdc/kudu_127.0.222.1.keytab.
Entry for principal kudu/127.0.222.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskgH5Pjm/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786277382073938-888-0/krb5kdc/kudu_127.0.222.1.keytab.
I20260809 12:09:44.240229 888 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskgH5Pjm/build/asan/bin/kudu
/tmp/dist-test-taskgH5Pjm/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskgH5Pjm/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786277382073938-888-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskgH5Pjm/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786277382073938-888-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskgH5Pjm/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786277382073938-888-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskgH5Pjm/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786277382073938-888-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.222.1:0
--local_ip_for_outbound_sockets=127.0.222.1
--webserver_interface=127.0.222.1
--webserver_port=0
--tserver_master_addrs=127.0.222.62:42811
--keytab_file=/tmp/dist-test-taskgH5Pjm/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786277382073938-888-0/krb5kdc/kudu_127.0.222.1.keytab
--principal=kudu/127.0.222.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.222.20:39985
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_enforce_access_control with env {KRB5CCNAME=/tmp/dist-test-taskgH5Pjm/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786277382073938-888-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskgH5Pjm/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786277382073938-888-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskgH5Pjm/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786277382073938-888-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260809 12:09:44.886281 1012 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260809 12:09:44.886621 1012 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260809 12:09:44.886875 1012 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260809 12:09:44.896906 1012 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260809 12:09:44.897094 1012 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260809 12:09:44.897311 1012 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.222.1
I20260809 12:09:44.909938 1012 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.222.20:39985
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskgH5Pjm/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786277382073938-888-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskgH5Pjm/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786277382073938-888-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskgH5Pjm/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786277382073938-888-0/krb5kdc/kudu_127.0.222.1.keytab
--principal=kudu/127.0.222.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.222.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskgH5Pjm/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786277382073938-888-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.222.1
--webserver_port=0
--tserver_master_addrs=127.0.222.62:42811
--tserver_enforce_access_control=true
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.222.1
--log_dir=/tmp/dist-test-taskgH5Pjm/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786277382073938-888-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 09 Aug 2026 11:06:41 UTC on ccycloud.quasar-bwoogy.root.comops.site
build id 4575991
ASAN enabled
I20260809 12:09:44.911608 1012 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260809 12:09:44.913199 1012 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260809 12:09:44.929368 1017 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 12:09:44.930349 1020 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 12:09:44.930366 1018 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260809 12:09:44.931819 1012 server_base.cc:1034] running on GCE node
I20260809 12:09:44.932484 1012 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260809 12:09:44.933432 1012 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260809 12:09:44.934723 1012 hybrid_clock.cc:648] HybridClock initialized: now 1786277384934626 us; error 34 us; skew 500 ppm
I20260809 12:09:44.935329 1012 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 09 12:09:44 dist-test-slave-2w3w krb5kdc[897](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786277384, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.222.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260809 12:09:44.939749 1012 init.cc:377] Logged in from keytab as kudu/127.0.222.1@KRBTEST.COM (short username kudu)
I20260809 12:09:44.942057 1012 webserver.cc:460] Webserver started at http://127.0.222.1:41043/ using document root <none> and password file <none>
I20260809 12:09:44.943255 1012 fs_manager.cc:359] Metadata directory not provided
I20260809 12:09:44.943516 1012 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260809 12:09:44.943971 1012 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260809 12:09:44.948009 1012 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskgH5Pjm/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786277382073938-888-0/minicluster-data/ts-0/data/instance:
uuid: "9e7ef5854212477f84883fac2766a568"
format_stamp: "Formatted at 2026-08-09 12:09:44 on dist-test-slave-2w3w"
server_key: "329b837911cda6f75ca7bf4765366f82"
server_key_iv: "5082d2bf3380769035c1041411ac332f"
server_key_version: "encryptionkey@0"
I20260809 12:09:44.949131 1012 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskgH5Pjm/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786277382073938-888-0/minicluster-data/ts-0/wal/instance:
uuid: "9e7ef5854212477f84883fac2766a568"
format_stamp: "Formatted at 2026-08-09 12:09:44 on dist-test-slave-2w3w"
server_key: "329b837911cda6f75ca7bf4765366f82"
server_key_iv: "5082d2bf3380769035c1041411ac332f"
server_key_version: "encryptionkey@0"
I20260809 12:09:44.956310 1012 fs_manager.cc:691] Time spent creating directory manager: real 0.007s user 0.003s sys 0.005s
I20260809 12:09:44.960736 1028 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260809 12:09:44.962093 1012 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.000s
I20260809 12:09:44.962308 1012 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskgH5Pjm/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786277382073938-888-0/minicluster-data/ts-0/data,/tmp/dist-test-taskgH5Pjm/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786277382073938-888-0/minicluster-data/ts-0/wal
uuid: "9e7ef5854212477f84883fac2766a568"
format_stamp: "Formatted at 2026-08-09 12:09:44 on dist-test-slave-2w3w"
server_key: "329b837911cda6f75ca7bf4765366f82"
server_key_iv: "5082d2bf3380769035c1041411ac332f"
server_key_version: "encryptionkey@0"
I20260809 12:09:44.962554 1012 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskgH5Pjm/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786277382073938-888-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskgH5Pjm/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786277382073938-888-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskgH5Pjm/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786277382073938-888-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260809 12:09:45.005852 1012 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260809 12:09:45.011763 1012 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260809 12:09:45.012275 1012 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260809 12:09:45.013818 1012 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260809 12:09:45.016270 1012 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260809 12:09:45.016474 1012 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260809 12:09:45.016782 1012 ts_tablet_manager.cc:610] Registered 0 tablets
I20260809 12:09:45.016927 1012 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260809 12:09:45.132054 1012 rpc_server.cc:307] RPC server started. Bound to: 127.0.222.1:34503
I20260809 12:09:45.132182 1154 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.222.1:34503 every 8 connection(s)
I20260809 12:09:45.133617 1012 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskgH5Pjm/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786277382073938-888-0/minicluster-data/ts-0/data/info.pb
I20260809 12:09:45.140940 888 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskgH5Pjm/build/asan/bin/kudu as pid 1012
I20260809 12:09:45.141355 888 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskgH5Pjm/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786277382073938-888-0/minicluster-data/ts-0/wal/instance
I20260809 12:09:45.142530 888 external_mini_cluster.cc:1428] Setting key 18b1a9533be78cdd768d956d4f1c45a8
Aug 09 12:09:45 dist-test-slave-2w3w krb5kdc[897](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786277384, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.222.1@KRBTEST.COM for kudu/127.0.222.62@KRBTEST.COM
I20260809 12:09:45.165480 1157 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 12:09:45.137239 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.222.1:60541 (local address 127.0.222.62:42811)
0809 12:09:45.137962 (+ 723us) server_negotiation.cc:206] Beginning negotiation
0809 12:09:45.137966 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0809 12:09:45.138909 (+ 943us) server_negotiation.cc:407] Connection header received
0809 12:09:45.141921 (+ 3012us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 12:09:45.141926 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 12:09:45.141995 (+ 69us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 12:09:45.142201 (+ 206us) server_negotiation.cc:226] Negotiated authn=SASL
0809 12:09:45.144701 (+ 2500us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 12:09:45.145653 (+ 952us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 12:09:45.149486 (+ 3833us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 12:09:45.149948 (+ 462us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 12:09:45.150044 (+ 96us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 12:09:45.156895 (+ 6851us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 12:09:45.156900 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 12:09:45.156903 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0809 12:09:45.156936 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0809 12:09:45.161246 (+ 4310us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 12:09:45.162047 (+ 801us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 12:09:45.162066 (+ 19us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 12:09:45.162072 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0809 12:09:45.162313 (+ 241us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 12:09:45.162867 (+ 554us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 12:09:45.162874 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 12:09:45.162881 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0809 12:09:45.163380 (+ 499us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 12:09:45.163547 (+ 167us) server_negotiation.cc:1035] Waiting for connection context
0809 12:09:45.164391 (+ 844us) server_negotiation.cc:299] Negotiation successful
0809 12:09:45.164673 (+ 282us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":562,"thread_start_us":440,"threads_started":1}
I20260809 12:09:45.166019 1158 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 12:09:45.137746 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.222.62:42811 (local address 127.0.222.1:60541)
0809 12:09:45.138609 (+ 863us) negotiation.cc:107] Waiting for socket to connect
0809 12:09:45.138652 (+ 43us) client_negotiation.cc:174] Beginning negotiation
0809 12:09:45.141560 (+ 2908us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 12:09:45.142602 (+ 1042us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 12:09:45.142617 (+ 15us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 12:09:45.143555 (+ 938us) client_negotiation.cc:189] Negotiated authn=SASL
0809 12:09:45.144445 (+ 890us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 12:09:45.144464 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 12:09:45.145859 (+ 1395us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 12:09:45.145866 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 12:09:45.149313 (+ 3447us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 12:09:45.149332 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 12:09:45.150337 (+ 1005us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 12:09:45.150356 (+ 19us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 12:09:45.150990 (+ 634us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 12:09:45.152117 (+ 1127us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0809 12:09:45.152207 (+ 90us) client_negotiation.cc:656] calling sasl_client_start()
0809 12:09:45.156569 (+ 4362us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0809 12:09:45.161400 (+ 4831us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0809 12:09:45.161406 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0809 12:09:45.161420 (+ 14us) client_negotiation.cc:760] calling sasl_client_step()
0809 12:09:45.161902 (+ 482us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0809 12:09:45.162447 (+ 545us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0809 12:09:45.162450 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0809 12:09:45.162453 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0809 12:09:45.162736 (+ 283us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0809 12:09:45.163524 (+ 788us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0809 12:09:45.163531 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0809 12:09:45.163891 (+ 360us) client_negotiation.cc:769] Sending connection context
0809 12:09:45.164200 (+ 309us) client_negotiation.cc:240] Negotiation successful
0809 12:09:45.164665 (+ 465us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":625,"thread_start_us":410,"threads_started":1}
I20260809 12:09:45.168287 1155 heartbeater.cc:347] Connected to a master server at 127.0.222.62:42811
I20260809 12:09:45.168841 1155 heartbeater.cc:464] Registering TS with master...
I20260809 12:09:45.170506 1155 heartbeater.cc:511] Master 127.0.222.62:42811 requested a full tablet report, sending...
WARNING: no policy specified for kudu/127.0.222.2@KRBTEST.COM; defaulting to no policy
I20260809 12:09:45.174823 947 ts_manager.cc:194] Registered new tserver with Master: 9e7ef5854212477f84883fac2766a568 (127.0.222.1:34503)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.222.2@KRBTEST.COM" created.
I20260809 12:09:45.177184 947 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.222.1@KRBTEST.COM'} at 127.0.222.1:60541
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.222.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskgH5Pjm/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786277382073938-888-0/krb5kdc/kudu_127.0.222.2.keytab.
Entry for principal kudu/127.0.222.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskgH5Pjm/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786277382073938-888-0/krb5kdc/kudu_127.0.222.2.keytab.
I20260809 12:09:45.210456 888 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskgH5Pjm/build/asan/bin/kudu
/tmp/dist-test-taskgH5Pjm/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskgH5Pjm/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786277382073938-888-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskgH5Pjm/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786277382073938-888-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskgH5Pjm/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786277382073938-888-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskgH5Pjm/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786277382073938-888-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.222.2:0
--local_ip_for_outbound_sockets=127.0.222.2
--webserver_interface=127.0.222.2
--webserver_port=0
--tserver_master_addrs=127.0.222.62:42811
--keytab_file=/tmp/dist-test-taskgH5Pjm/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786277382073938-888-0/krb5kdc/kudu_127.0.222.2.keytab
--principal=kudu/127.0.222.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.222.20:39985
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_enforce_access_control with env {KRB5CCNAME=/tmp/dist-test-taskgH5Pjm/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786277382073938-888-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskgH5Pjm/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786277382073938-888-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskgH5Pjm/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786277382073938-888-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260809 12:09:45.783536 1161 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260809 12:09:45.783885 1161 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260809 12:09:45.783998 1161 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260809 12:09:45.793395 1161 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260809 12:09:45.793606 1161 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260809 12:09:45.793794 1161 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.222.2
I20260809 12:09:45.806332 1161 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.222.20:39985
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskgH5Pjm/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786277382073938-888-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskgH5Pjm/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786277382073938-888-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskgH5Pjm/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786277382073938-888-0/krb5kdc/kudu_127.0.222.2.keytab
--principal=kudu/127.0.222.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.222.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskgH5Pjm/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786277382073938-888-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.222.2
--webserver_port=0
--tserver_master_addrs=127.0.222.62:42811
--tserver_enforce_access_control=true
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.222.2
--log_dir=/tmp/dist-test-taskgH5Pjm/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786277382073938-888-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 09 Aug 2026 11:06:41 UTC on ccycloud.quasar-bwoogy.root.comops.site
build id 4575991
ASAN enabled
I20260809 12:09:45.807735 1161 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260809 12:09:45.809417 1161 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260809 12:09:45.828027 1166 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 12:09:45.829352 1169 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 12:09:45.829914 1167 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260809 12:09:45.831103 1161 server_base.cc:1034] running on GCE node
I20260809 12:09:45.831902 1161 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260809 12:09:45.832891 1161 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260809 12:09:45.834133 1161 hybrid_clock.cc:648] HybridClock initialized: now 1786277385834056 us; error 32 us; skew 500 ppm
I20260809 12:09:45.834415 1161 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 09 12:09:45 dist-test-slave-2w3w krb5kdc[897](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786277385, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.222.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260809 12:09:45.838056 1161 init.cc:377] Logged in from keytab as kudu/127.0.222.2@KRBTEST.COM (short username kudu)
I20260809 12:09:45.840044 1161 webserver.cc:460] Webserver started at http://127.0.222.2:38849/ using document root <none> and password file <none>
I20260809 12:09:45.840703 1161 fs_manager.cc:359] Metadata directory not provided
I20260809 12:09:45.840804 1161 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260809 12:09:45.841101 1161 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260809 12:09:45.845067 1161 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskgH5Pjm/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786277382073938-888-0/minicluster-data/ts-1/data/instance:
uuid: "aca215a708164ead8416a53638728e9d"
format_stamp: "Formatted at 2026-08-09 12:09:45 on dist-test-slave-2w3w"
server_key: "399e5da82ae6a5f25c77f64d908abe61"
server_key_iv: "c1385044c73cee8287cf5f1c47d5f44b"
server_key_version: "encryptionkey@0"
I20260809 12:09:45.846522 1161 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskgH5Pjm/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786277382073938-888-0/minicluster-data/ts-1/wal/instance:
uuid: "aca215a708164ead8416a53638728e9d"
format_stamp: "Formatted at 2026-08-09 12:09:45 on dist-test-slave-2w3w"
server_key: "399e5da82ae6a5f25c77f64d908abe61"
server_key_iv: "c1385044c73cee8287cf5f1c47d5f44b"
server_key_version: "encryptionkey@0"
I20260809 12:09:45.855645 1161 fs_manager.cc:691] Time spent creating directory manager: real 0.008s user 0.006s sys 0.002s
I20260809 12:09:45.859866 1176 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260809 12:09:45.861470 1161 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.003s sys 0.001s
I20260809 12:09:45.861660 1161 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskgH5Pjm/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786277382073938-888-0/minicluster-data/ts-1/data,/tmp/dist-test-taskgH5Pjm/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786277382073938-888-0/minicluster-data/ts-1/wal
uuid: "aca215a708164ead8416a53638728e9d"
format_stamp: "Formatted at 2026-08-09 12:09:45 on dist-test-slave-2w3w"
server_key: "399e5da82ae6a5f25c77f64d908abe61"
server_key_iv: "c1385044c73cee8287cf5f1c47d5f44b"
server_key_version: "encryptionkey@0"
I20260809 12:09:45.861913 1161 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskgH5Pjm/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786277382073938-888-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskgH5Pjm/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786277382073938-888-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskgH5Pjm/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786277382073938-888-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260809 12:09:45.880977 1161 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260809 12:09:45.886139 1161 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260809 12:09:45.886591 1161 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260809 12:09:45.888226 1161 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260809 12:09:45.890532 1161 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260809 12:09:45.890679 1161 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260809 12:09:45.890856 1161 ts_tablet_manager.cc:610] Registered 0 tablets
I20260809 12:09:45.890945 1161 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260809 12:09:45.969051 1161 rpc_server.cc:307] RPC server started. Bound to: 127.0.222.2:39469
I20260809 12:09:45.969228 1297 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.222.2:39469 every 8 connection(s)
I20260809 12:09:45.970553 1161 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskgH5Pjm/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786277382073938-888-0/minicluster-data/ts-1/data/info.pb
I20260809 12:09:45.971971 888 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskgH5Pjm/build/asan/bin/kudu as pid 1161
I20260809 12:09:45.972311 888 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskgH5Pjm/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786277382073938-888-0/minicluster-data/ts-1/wal/instance
I20260809 12:09:45.973102 888 external_mini_cluster.cc:1428] Setting key 13b4778200cc8fd8765ddc67baa0944b
Aug 09 12:09:45 dist-test-slave-2w3w krb5kdc[897](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786277385, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.222.2@KRBTEST.COM for kudu/127.0.222.62@KRBTEST.COM
WARNING: no policy specified for kudu/127.0.222.3@KRBTEST.COM; defaulting to no policy
I20260809 12:09:46.004392 1300 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 12:09:45.974354 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.222.2:51261 (local address 127.0.222.62:42811)
0809 12:09:45.975340 (+ 986us) server_negotiation.cc:206] Beginning negotiation
0809 12:09:45.975345 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0809 12:09:45.977005 (+ 1660us) server_negotiation.cc:407] Connection header received
0809 12:09:45.981935 (+ 4930us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 12:09:45.981939 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 12:09:45.982007 (+ 68us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 12:09:45.982244 (+ 237us) server_negotiation.cc:226] Negotiated authn=SASL
0809 12:09:45.985764 (+ 3520us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 12:09:45.986684 (+ 920us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 12:09:45.990099 (+ 3415us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 12:09:45.991306 (+ 1207us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 12:09:45.991453 (+ 147us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 12:09:45.996385 (+ 4932us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 12:09:45.996395 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 12:09:45.996403 (+ 8us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0809 12:09:45.996453 (+ 50us) server_negotiation.cc:922] calling sasl_server_start()
0809 12:09:46.000822 (+ 4369us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 12:09:46.001673 (+ 851us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 12:09:46.001679 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 12:09:46.001685 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0809 12:09:46.001854 (+ 169us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 12:09:46.002384 (+ 530us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 12:09:46.002391 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 12:09:46.002396 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0809 12:09:46.002785 (+ 389us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 12:09:46.002984 (+ 199us) server_negotiation.cc:1035] Waiting for connection context
0809 12:09:46.003927 (+ 943us) server_negotiation.cc:299] Negotiation successful
0809 12:09:46.004109 (+ 182us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":843,"thread_start_us":707,"threads_started":1}
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.222.3@KRBTEST.COM" created.
I20260809 12:09:46.005772 1301 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 12:09:45.975312 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.222.62:42811 (local address 127.0.222.2:51261)
0809 12:09:45.976675 (+ 1363us) negotiation.cc:107] Waiting for socket to connect
0809 12:09:45.976747 (+ 72us) client_negotiation.cc:174] Beginning negotiation
0809 12:09:45.981478 (+ 4731us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 12:09:45.982779 (+ 1301us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 12:09:45.982793 (+ 14us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 12:09:45.984141 (+ 1348us) client_negotiation.cc:189] Negotiated authn=SASL
0809 12:09:45.985565 (+ 1424us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 12:09:45.985594 (+ 29us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 12:09:45.987051 (+ 1457us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 12:09:45.987059 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 12:09:45.989928 (+ 2869us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 12:09:45.989941 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 12:09:45.991552 (+ 1611us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 12:09:45.991557 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 12:09:45.991828 (+ 271us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 12:09:45.992594 (+ 766us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0809 12:09:45.992618 (+ 24us) client_negotiation.cc:656] calling sasl_client_start()
0809 12:09:45.996001 (+ 3383us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0809 12:09:46.000991 (+ 4990us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0809 12:09:46.000998 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0809 12:09:46.001016 (+ 18us) client_negotiation.cc:760] calling sasl_client_step()
0809 12:09:46.001516 (+ 500us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0809 12:09:46.001971 (+ 455us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0809 12:09:46.001975 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0809 12:09:46.001977 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0809 12:09:46.002242 (+ 265us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0809 12:09:46.002939 (+ 697us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0809 12:09:46.002946 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0809 12:09:46.003387 (+ 441us) client_negotiation.cc:769] Sending connection context
0809 12:09:46.003674 (+ 287us) client_negotiation.cc:240] Negotiation successful
0809 12:09:46.004200 (+ 526us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":994,"thread_start_us":588,"threads_started":1}
I20260809 12:09:46.007735 1298 heartbeater.cc:347] Connected to a master server at 127.0.222.62:42811
I20260809 12:09:46.008083 1298 heartbeater.cc:464] Registering TS with master...
I20260809 12:09:46.008888 1298 heartbeater.cc:511] Master 127.0.222.62:42811 requested a full tablet report, sending...
I20260809 12:09:46.010608 947 ts_manager.cc:194] Registered new tserver with Master: aca215a708164ead8416a53638728e9d (127.0.222.2:39469)
I20260809 12:09:46.011543 947 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.222.2@KRBTEST.COM'} at 127.0.222.2:51261
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.222.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskgH5Pjm/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786277382073938-888-0/krb5kdc/kudu_127.0.222.3.keytab.
Entry for principal kudu/127.0.222.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskgH5Pjm/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786277382073938-888-0/krb5kdc/kudu_127.0.222.3.keytab.
I20260809 12:09:46.033525 888 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskgH5Pjm/build/asan/bin/kudu
/tmp/dist-test-taskgH5Pjm/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskgH5Pjm/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786277382073938-888-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskgH5Pjm/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786277382073938-888-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskgH5Pjm/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786277382073938-888-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskgH5Pjm/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786277382073938-888-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.222.3:0
--local_ip_for_outbound_sockets=127.0.222.3
--webserver_interface=127.0.222.3
--webserver_port=0
--tserver_master_addrs=127.0.222.62:42811
--keytab_file=/tmp/dist-test-taskgH5Pjm/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786277382073938-888-0/krb5kdc/kudu_127.0.222.3.keytab
--principal=kudu/127.0.222.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.222.20:39985
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_enforce_access_control with env {KRB5CCNAME=/tmp/dist-test-taskgH5Pjm/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786277382073938-888-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskgH5Pjm/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786277382073938-888-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskgH5Pjm/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786277382073938-888-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260809 12:09:46.181025 1155 heartbeater.cc:503] Master 127.0.222.62:42811 was elected leader, sending a full tablet report...
W20260809 12:09:46.579910 1304 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260809 12:09:46.580446 1304 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260809 12:09:46.580685 1304 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260809 12:09:46.590999 1304 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260809 12:09:46.591253 1304 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260809 12:09:46.591480 1304 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.222.3
I20260809 12:09:46.604244 1304 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.222.20:39985
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskgH5Pjm/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786277382073938-888-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskgH5Pjm/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786277382073938-888-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskgH5Pjm/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786277382073938-888-0/krb5kdc/kudu_127.0.222.3.keytab
--principal=kudu/127.0.222.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.222.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskgH5Pjm/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786277382073938-888-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.222.3
--webserver_port=0
--tserver_master_addrs=127.0.222.62:42811
--tserver_enforce_access_control=true
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.222.3
--log_dir=/tmp/dist-test-taskgH5Pjm/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786277382073938-888-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 09 Aug 2026 11:06:41 UTC on ccycloud.quasar-bwoogy.root.comops.site
build id 4575991
ASAN enabled
I20260809 12:09:46.605692 1304 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260809 12:09:46.607342 1304 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260809 12:09:46.624541 1309 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 12:09:46.624751 1312 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 12:09:46.625118 1310 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260809 12:09:46.626758 1304 server_base.cc:1034] running on GCE node
I20260809 12:09:46.627645 1304 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260809 12:09:46.628722 1304 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260809 12:09:46.630000 1304 hybrid_clock.cc:648] HybridClock initialized: now 1786277386629915 us; error 59 us; skew 500 ppm
I20260809 12:09:46.630374 1304 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 09 12:09:46 dist-test-slave-2w3w krb5kdc[897](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786277386, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.222.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260809 12:09:46.634631 1304 init.cc:377] Logged in from keytab as kudu/127.0.222.3@KRBTEST.COM (short username kudu)
I20260809 12:09:46.636445 1304 webserver.cc:460] Webserver started at http://127.0.222.3:37475/ using document root <none> and password file <none>
I20260809 12:09:46.637305 1304 fs_manager.cc:359] Metadata directory not provided
I20260809 12:09:46.637542 1304 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260809 12:09:46.638044 1304 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260809 12:09:46.641639 1304 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskgH5Pjm/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786277382073938-888-0/minicluster-data/ts-2/data/instance:
uuid: "404406f3954d474f9024eb1453256ac1"
format_stamp: "Formatted at 2026-08-09 12:09:46 on dist-test-slave-2w3w"
server_key: "fec16bd8e45aca851e7bd95359051217"
server_key_iv: "120af0d1d36d8c56f806af9ae0024edf"
server_key_version: "encryptionkey@0"
I20260809 12:09:46.642678 1304 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskgH5Pjm/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786277382073938-888-0/minicluster-data/ts-2/wal/instance:
uuid: "404406f3954d474f9024eb1453256ac1"
format_stamp: "Formatted at 2026-08-09 12:09:46 on dist-test-slave-2w3w"
server_key: "fec16bd8e45aca851e7bd95359051217"
server_key_iv: "120af0d1d36d8c56f806af9ae0024edf"
server_key_version: "encryptionkey@0"
I20260809 12:09:46.649142 1304 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.000s sys 0.004s
I20260809 12:09:46.653237 1319 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260809 12:09:46.654646 1304 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260809 12:09:46.654865 1304 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskgH5Pjm/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786277382073938-888-0/minicluster-data/ts-2/data,/tmp/dist-test-taskgH5Pjm/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786277382073938-888-0/minicluster-data/ts-2/wal
uuid: "404406f3954d474f9024eb1453256ac1"
format_stamp: "Formatted at 2026-08-09 12:09:46 on dist-test-slave-2w3w"
server_key: "fec16bd8e45aca851e7bd95359051217"
server_key_iv: "120af0d1d36d8c56f806af9ae0024edf"
server_key_version: "encryptionkey@0"
I20260809 12:09:46.655222 1304 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskgH5Pjm/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786277382073938-888-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskgH5Pjm/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786277382073938-888-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskgH5Pjm/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786277382073938-888-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260809 12:09:46.672959 1304 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260809 12:09:46.679390 1304 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260809 12:09:46.679850 1304 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260809 12:09:46.681499 1304 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260809 12:09:46.683660 1304 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260809 12:09:46.683804 1304 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260809 12:09:46.684049 1304 ts_tablet_manager.cc:610] Registered 0 tablets
I20260809 12:09:46.684168 1304 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260809 12:09:46.760396 1304 rpc_server.cc:307] RPC server started. Bound to: 127.0.222.3:36859
I20260809 12:09:46.760516 1441 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.222.3:36859 every 8 connection(s)
I20260809 12:09:46.761915 1304 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskgH5Pjm/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786277382073938-888-0/minicluster-data/ts-2/data/info.pb
I20260809 12:09:46.763756 888 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskgH5Pjm/build/asan/bin/kudu as pid 1304
I20260809 12:09:46.764043 888 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskgH5Pjm/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786277382073938-888-0/minicluster-data/ts-2/wal/instance
I20260809 12:09:46.764710 888 external_mini_cluster.cc:1428] Setting key d4eb41f2ce70e0af3451f379732f383d
Aug 09 12:09:46 dist-test-slave-2w3w krb5kdc[897](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786277386, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.222.3@KRBTEST.COM for kudu/127.0.222.62@KRBTEST.COM
I20260809 12:09:46.792632 1444 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 12:09:46.765973 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.222.3:42431 (local address 127.0.222.62:42811)
0809 12:09:46.766789 (+ 816us) server_negotiation.cc:206] Beginning negotiation
0809 12:09:46.766796 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0809 12:09:46.768808 (+ 2012us) server_negotiation.cc:407] Connection header received
0809 12:09:46.773015 (+ 4207us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 12:09:46.773019 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 12:09:46.773087 (+ 68us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 12:09:46.773173 (+ 86us) server_negotiation.cc:226] Negotiated authn=SASL
0809 12:09:46.776090 (+ 2917us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 12:09:46.777131 (+ 1041us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 12:09:46.778900 (+ 1769us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 12:09:46.779617 (+ 717us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 12:09:46.779720 (+ 103us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 12:09:46.784908 (+ 5188us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 12:09:46.784912 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 12:09:46.784915 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0809 12:09:46.784949 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0809 12:09:46.788986 (+ 4037us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 12:09:46.789905 (+ 919us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 12:09:46.789928 (+ 23us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 12:09:46.789933 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0809 12:09:46.790167 (+ 234us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 12:09:46.790703 (+ 536us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 12:09:46.790706 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 12:09:46.790708 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0809 12:09:46.791158 (+ 450us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 12:09:46.791303 (+ 145us) server_negotiation.cc:1035] Waiting for connection context
0809 12:09:46.792069 (+ 766us) server_negotiation.cc:299] Negotiation successful
0809 12:09:46.792236 (+ 167us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":652,"thread_start_us":486,"threads_started":1}
I20260809 12:09:46.793746 1446 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 12:09:46.766949 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.222.62:42811 (local address 127.0.222.3:42431)
0809 12:09:46.768601 (+ 1652us) negotiation.cc:107] Waiting for socket to connect
0809 12:09:46.768650 (+ 49us) client_negotiation.cc:174] Beginning negotiation
0809 12:09:46.772656 (+ 4006us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 12:09:46.773569 (+ 913us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 12:09:46.773583 (+ 14us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 12:09:46.774800 (+ 1217us) client_negotiation.cc:189] Negotiated authn=SASL
0809 12:09:46.775873 (+ 1073us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 12:09:46.775902 (+ 29us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 12:09:46.777323 (+ 1421us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 12:09:46.777327 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 12:09:46.778770 (+ 1443us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 12:09:46.778782 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 12:09:46.779875 (+ 1093us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 12:09:46.779881 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 12:09:46.780117 (+ 236us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 12:09:46.780997 (+ 880us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0809 12:09:46.781021 (+ 24us) client_negotiation.cc:656] calling sasl_client_start()
0809 12:09:46.784656 (+ 3635us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0809 12:09:46.789116 (+ 4460us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0809 12:09:46.789123 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0809 12:09:46.789136 (+ 13us) client_negotiation.cc:760] calling sasl_client_step()
0809 12:09:46.789723 (+ 587us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0809 12:09:46.790290 (+ 567us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0809 12:09:46.790294 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0809 12:09:46.790297 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0809 12:09:46.790591 (+ 294us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0809 12:09:46.791283 (+ 692us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0809 12:09:46.791289 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0809 12:09:46.791637 (+ 348us) client_negotiation.cc:769] Sending connection context
0809 12:09:46.791951 (+ 314us) client_negotiation.cc:240] Negotiation successful
0809 12:09:46.792360 (+ 409us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1188,"thread_start_us":745,"threads_started":1}
I20260809 12:09:46.795502 1442 heartbeater.cc:347] Connected to a master server at 127.0.222.62:42811
I20260809 12:09:46.796002 1442 heartbeater.cc:464] Registering TS with master...
I20260809 12:09:46.797077 1442 heartbeater.cc:511] Master 127.0.222.62:42811 requested a full tablet report, sending...
I20260809 12:09:46.798851 946 ts_manager.cc:194] Registered new tserver with Master: 404406f3954d474f9024eb1453256ac1 (127.0.222.3:36859)
I20260809 12:09:46.800041 946 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.222.3@KRBTEST.COM'} at 127.0.222.3:42431
I20260809 12:09:46.809636 888 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
Aug 09 12:09:46 dist-test-slave-2w3w krb5kdc[897](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786277386, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-user@KRBTEST.COM:
Aug 09 12:09:46 dist-test-slave-2w3w krb5kdc[897](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786277386, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.0.222.1@KRBTEST.COM
I20260809 12:09:46.871802 1451 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 12:09:46.849590 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33980 (local address 127.0.222.1:34503)
0809 12:09:46.850772 (+ 1182us) server_negotiation.cc:206] Beginning negotiation
0809 12:09:46.850804 (+ 32us) server_negotiation.cc:399] Waiting for connection header
0809 12:09:46.850842 (+ 38us) server_negotiation.cc:407] Connection header received
0809 12:09:46.851073 (+ 231us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 12:09:46.851081 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 12:09:46.851349 (+ 268us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 12:09:46.851509 (+ 160us) server_negotiation.cc:226] Negotiated authn=SASL
0809 12:09:46.853762 (+ 2253us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 12:09:46.855206 (+ 1444us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 12:09:46.856897 (+ 1691us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 12:09:46.857652 (+ 755us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 12:09:46.857757 (+ 105us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 12:09:46.863915 (+ 6158us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 12:09:46.863929 (+ 14us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 12:09:46.863944 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0809 12:09:46.863986 (+ 42us) server_negotiation.cc:922] calling sasl_server_start()
0809 12:09:46.868271 (+ 4285us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 12:09:46.869041 (+ 770us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 12:09:46.869048 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 12:09:46.869050 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0809 12:09:46.869246 (+ 196us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 12:09:46.869698 (+ 452us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 12:09:46.869701 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 12:09:46.869703 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0809 12:09:46.870154 (+ 451us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 12:09:46.870480 (+ 326us) server_negotiation.cc:1035] Waiting for connection context
0809 12:09:46.870886 (+ 406us) server_negotiation.cc:299] Negotiation successful
0809 12:09:46.871209 (+ 323us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1027,"thread_start_us":768,"threads_started":1}
W20260809 12:09:46.873471 1060 server_base.cc:1116] Unauthorized access attempt to method kudu.tserver.TabletServerService.ListTablets from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:33980
Aug 09 12:09:46 dist-test-slave-2w3w krb5kdc[897](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786277386, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
Aug 09 12:09:46 dist-test-slave-2w3w krb5kdc[897](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786277386, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.222.1@KRBTEST.COM
I20260809 12:09:46.932443 1451 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 12:09:46.915350 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33992 (local address 127.0.222.1:34503)
0809 12:09:46.915627 (+ 277us) server_negotiation.cc:206] Beginning negotiation
0809 12:09:46.915634 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0809 12:09:46.916012 (+ 378us) server_negotiation.cc:407] Connection header received
0809 12:09:46.916177 (+ 165us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 12:09:46.916181 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 12:09:46.916262 (+ 81us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 12:09:46.916350 (+ 88us) server_negotiation.cc:226] Negotiated authn=SASL
0809 12:09:46.917940 (+ 1590us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 12:09:46.918638 (+ 698us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 12:09:46.919947 (+ 1309us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 12:09:46.920417 (+ 470us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 12:09:46.920575 (+ 158us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 12:09:46.925549 (+ 4974us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 12:09:46.925557 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 12:09:46.925565 (+ 8us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0809 12:09:46.925721 (+ 156us) server_negotiation.cc:922] calling sasl_server_start()
0809 12:09:46.929703 (+ 3982us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 12:09:46.930336 (+ 633us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 12:09:46.930340 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 12:09:46.930343 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0809 12:09:46.930554 (+ 211us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 12:09:46.930883 (+ 329us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 12:09:46.930887 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 12:09:46.930889 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0809 12:09:46.931204 (+ 315us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 12:09:46.931462 (+ 258us) server_negotiation.cc:1035] Waiting for connection context
0809 12:09:46.931664 (+ 202us) server_negotiation.cc:299] Negotiation successful
0809 12:09:46.931872 (+ 208us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":104}
I20260809 12:09:46.934998 888 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskgH5Pjm/build/asan/bin/kudu pid 1012
I20260809 12:09:46.952203 1451 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 12:09:46.936106 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34002 (local address 127.0.222.1:34503)
0809 12:09:46.936297 (+ 191us) server_negotiation.cc:206] Beginning negotiation
0809 12:09:46.936301 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0809 12:09:46.936762 (+ 461us) server_negotiation.cc:407] Connection header received
0809 12:09:46.936854 (+ 92us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 12:09:46.936858 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 12:09:46.936922 (+ 64us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 12:09:46.937009 (+ 87us) server_negotiation.cc:226] Negotiated authn=SASL
0809 12:09:46.938878 (+ 1869us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 12:09:46.939909 (+ 1031us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 12:09:46.941705 (+ 1796us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 12:09:46.942253 (+ 548us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 12:09:46.942353 (+ 100us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 12:09:46.945697 (+ 3344us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 12:09:46.945702 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 12:09:46.945705 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0809 12:09:46.945737 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0809 12:09:46.949145 (+ 3408us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 12:09:46.950007 (+ 862us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 12:09:46.950011 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 12:09:46.950013 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0809 12:09:46.950226 (+ 213us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 12:09:46.950628 (+ 402us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 12:09:46.950631 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 12:09:46.950633 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0809 12:09:46.951074 (+ 441us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 12:09:46.951315 (+ 241us) server_negotiation.cc:1035] Waiting for connection context
0809 12:09:46.951622 (+ 307us) server_negotiation.cc:299] Negotiation successful
0809 12:09:46.951792 (+ 170us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":50}
I20260809 12:09:46.953260 1150 generic_service.cc:196] Checking for leaks (request via RPC)
I20260809 12:09:47.014256 1298 heartbeater.cc:503] Master 127.0.222.62:42811 was elected leader, sending a full tablet report...
I20260809 12:09:47.050756 888 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskgH5Pjm/build/asan/bin/kudu with pid 1012
I20260809 12:09:47.075104 888 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskgH5Pjm/build/asan/bin/kudu pid 1161
Aug 09 12:09:47 dist-test-slave-2w3w krb5kdc[897](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786277386, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.222.2@KRBTEST.COM
I20260809 12:09:47.092206 1458 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 12:09:47.076184 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60990 (local address 127.0.222.2:39469)
0809 12:09:47.076810 (+ 626us) server_negotiation.cc:206] Beginning negotiation
0809 12:09:47.076817 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0809 12:09:47.076842 (+ 25us) server_negotiation.cc:407] Connection header received
0809 12:09:47.076900 (+ 58us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 12:09:47.076907 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 12:09:47.077111 (+ 204us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 12:09:47.077235 (+ 124us) server_negotiation.cc:226] Negotiated authn=SASL
0809 12:09:47.078468 (+ 1233us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 12:09:47.079547 (+ 1079us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 12:09:47.080775 (+ 1228us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 12:09:47.081315 (+ 540us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 12:09:47.081412 (+ 97us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 12:09:47.085509 (+ 4097us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 12:09:47.085520 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 12:09:47.085534 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0809 12:09:47.085567 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0809 12:09:47.088544 (+ 2977us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 12:09:47.089371 (+ 827us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 12:09:47.089378 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 12:09:47.089381 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0809 12:09:47.089517 (+ 136us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 12:09:47.089924 (+ 407us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 12:09:47.089928 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 12:09:47.089929 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0809 12:09:47.090465 (+ 536us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 12:09:47.090844 (+ 379us) server_negotiation.cc:1035] Waiting for connection context
0809 12:09:47.091612 (+ 768us) server_negotiation.cc:299] Negotiation successful
0809 12:09:47.091909 (+ 297us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":484,"thread_start_us":327,"threads_started":1}
I20260809 12:09:47.093467 1293 generic_service.cc:196] Checking for leaks (request via RPC)
I20260809 12:09:47.178512 888 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskgH5Pjm/build/asan/bin/kudu with pid 1161
I20260809 12:09:47.197775 888 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskgH5Pjm/build/asan/bin/kudu pid 1304
Aug 09 12:09:47 dist-test-slave-2w3w krb5kdc[897](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786277386, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.222.3@KRBTEST.COM
I20260809 12:09:47.214422 1460 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 12:09:47.198803 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45278 (local address 127.0.222.3:36859)
0809 12:09:47.199428 (+ 625us) server_negotiation.cc:206] Beginning negotiation
0809 12:09:47.199435 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0809 12:09:47.199464 (+ 29us) server_negotiation.cc:407] Connection header received
0809 12:09:47.199529 (+ 65us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 12:09:47.199536 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 12:09:47.199746 (+ 210us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 12:09:47.199871 (+ 125us) server_negotiation.cc:226] Negotiated authn=SASL
0809 12:09:47.201078 (+ 1207us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 12:09:47.202093 (+ 1015us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 12:09:47.203612 (+ 1519us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 12:09:47.204187 (+ 575us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 12:09:47.204291 (+ 104us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 12:09:47.208770 (+ 4479us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 12:09:47.208784 (+ 14us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 12:09:47.208801 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0809 12:09:47.208844 (+ 43us) server_negotiation.cc:922] calling sasl_server_start()
0809 12:09:47.211845 (+ 3001us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 12:09:47.212502 (+ 657us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 12:09:47.212509 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 12:09:47.212511 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0809 12:09:47.212634 (+ 123us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 12:09:47.212953 (+ 319us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 12:09:47.212956 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 12:09:47.212958 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0809 12:09:47.213305 (+ 347us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 12:09:47.213527 (+ 222us) server_negotiation.cc:1035] Waiting for connection context
0809 12:09:47.213789 (+ 262us) server_negotiation.cc:299] Negotiation successful
0809 12:09:47.214000 (+ 211us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":470,"thread_start_us":321,"threads_started":1}
I20260809 12:09:47.215847 1435 generic_service.cc:196] Checking for leaks (request via RPC)
I20260809 12:09:47.299844 888 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskgH5Pjm/build/asan/bin/kudu with pid 1304
I20260809 12:09:47.319135 888 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskgH5Pjm/build/asan/bin/kudu pid 911
I20260809 12:09:47.320375 980 generic_service.cc:196] Checking for leaks (request via RPC)
I20260809 12:09:47.428186 888 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskgH5Pjm/build/asan/bin/kudu with pid 911
2026-08-09T12:09:47Z chronyd exiting
[ OK ] SecurityITest.TestAuthorizationOnListTablets (5290 ms)
[ RUN ] SecurityITest.TestAuthorizationOnChecksum
Loading random data
Initializing database '/tmp/dist-test-taskgH5Pjm/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786277382073938-888-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 09 12:09:47 dist-test-slave-2w3w krb5kdc[1465](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 09 12:09:47 dist-test-slave-2w3w krb5kdc[1465](info): set up 2 sockets
Aug 09 12:09:47 dist-test-slave-2w3w krb5kdc[1465](info): commencing operation
krb5kdc: starting...
W20260809 12:09:48.576848 888 mini_kdc.cc:121] Time spent starting KDC: real 1.101s user 0.000s sys 0.019s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskgH5Pjm/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786277382073938-888-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskgH5Pjm/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786277382073938-888-0/krb5kdc/test-user.keytab.
Aug 09 12:09:48 dist-test-slave-2w3w krb5kdc[1465](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786277388, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-09T12:09:48Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-09T12:09:48Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.222.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.222.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.222.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskgH5Pjm/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786277382073938-888-0/krb5kdc/kudu_127.0.222.62.keytab.
Entry for principal kudu/127.0.222.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskgH5Pjm/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786277382073938-888-0/krb5kdc/kudu_127.0.222.62.keytab.
I20260809 12:09:48.778056 888 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskgH5Pjm/build/asan/bin/kudu
/tmp/dist-test-taskgH5Pjm/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskgH5Pjm/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786277382073938-888-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskgH5Pjm/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786277382073938-888-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskgH5Pjm/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786277382073938-888-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskgH5Pjm/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786277382073938-888-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.222.62:34593
--webserver_interface=127.0.222.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskgH5Pjm/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786277382073938-888-0/krb5kdc/kudu_127.0.222.62.keytab
--principal=kudu/127.0.222.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.222.20:38943
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.222.62:34593
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskgH5Pjm/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786277382073938-888-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskgH5Pjm/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786277382073938-888-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskgH5Pjm/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786277382073938-888-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260809 12:09:49.108460 1480 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260809 12:09:49.108961 1480 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260809 12:09:49.109131 1480 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260809 12:09:49.117343 1480 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260809 12:09:49.117488 1480 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260809 12:09:49.117596 1480 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260809 12:09:49.117704 1480 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260809 12:09:49.117797 1480 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260809 12:09:49.127997 1480 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.222.20:38943
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskgH5Pjm/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786277382073938-888-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskgH5Pjm/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786277382073938-888-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.222.62:34593
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskgH5Pjm/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786277382073938-888-0/krb5kdc/kudu_127.0.222.62.keytab
--principal=kudu/127.0.222.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.222.62:34593
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskgH5Pjm/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786277382073938-888-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.222.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskgH5Pjm/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786277382073938-888-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 09 Aug 2026 11:06:41 UTC on ccycloud.quasar-bwoogy.root.comops.site
build id 4575991
ASAN enabled
I20260809 12:09:49.129704 1480 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260809 12:09:49.131220 1480 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260809 12:09:49.142992 1486 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 12:09:49.142920 1487 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 12:09:49.142930 1489 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260809 12:09:49.143913 1480 server_base.cc:1034] running on GCE node
I20260809 12:09:49.144755 1480 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260809 12:09:49.146128 1480 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260809 12:09:49.147362 1480 hybrid_clock.cc:648] HybridClock initialized: now 1786277389147312 us; error 57 us; skew 500 ppm
I20260809 12:09:49.147713 1480 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 09 12:09:49 dist-test-slave-2w3w krb5kdc[1465](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786277389, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.222.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260809 12:09:49.150776 1480 init.cc:377] Logged in from keytab as kudu/127.0.222.62@KRBTEST.COM (short username kudu)
I20260809 12:09:49.152302 1480 webserver.cc:460] Webserver started at http://127.0.222.62:38063/ using document root <none> and password file <none>
I20260809 12:09:49.152992 1480 fs_manager.cc:359] Metadata directory not provided
I20260809 12:09:49.153143 1480 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260809 12:09:49.153455 1480 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260809 12:09:49.156422 1480 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskgH5Pjm/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786277382073938-888-0/minicluster-data/master-0/data/instance:
uuid: "aa483a268fb147afbbfa935435493e40"
format_stamp: "Formatted at 2026-08-09 12:09:49 on dist-test-slave-2w3w"
server_key: "98824edf1c90a0541824a4700346a7c1"
server_key_iv: "1b044f3048aec8b55dd57adb945c861c"
server_key_version: "encryptionkey@0"
I20260809 12:09:49.157188 1480 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskgH5Pjm/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786277382073938-888-0/minicluster-data/master-0/wal/instance:
uuid: "aa483a268fb147afbbfa935435493e40"
format_stamp: "Formatted at 2026-08-09 12:09:49 on dist-test-slave-2w3w"
server_key: "98824edf1c90a0541824a4700346a7c1"
server_key_iv: "1b044f3048aec8b55dd57adb945c861c"
server_key_version: "encryptionkey@0"
I20260809 12:09:49.162338 1480 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.000s
I20260809 12:09:49.166342 1496 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260809 12:09:49.167730 1480 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.002s
I20260809 12:09:49.167959 1480 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskgH5Pjm/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786277382073938-888-0/minicluster-data/master-0/data,/tmp/dist-test-taskgH5Pjm/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786277382073938-888-0/minicluster-data/master-0/wal
uuid: "aa483a268fb147afbbfa935435493e40"
format_stamp: "Formatted at 2026-08-09 12:09:49 on dist-test-slave-2w3w"
server_key: "98824edf1c90a0541824a4700346a7c1"
server_key_iv: "1b044f3048aec8b55dd57adb945c861c"
server_key_version: "encryptionkey@0"
I20260809 12:09:49.168169 1480 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskgH5Pjm/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786277382073938-888-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskgH5Pjm/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786277382073938-888-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskgH5Pjm/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786277382073938-888-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260809 12:09:49.187175 1480 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260809 12:09:49.191331 1480 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260809 12:09:49.191641 1480 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260809 12:09:49.210325 1480 rpc_server.cc:307] RPC server started. Bound to: 127.0.222.62:34593
I20260809 12:09:49.210362 1549 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.222.62:34593 every 8 connection(s)
I20260809 12:09:49.211771 1480 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskgH5Pjm/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786277382073938-888-0/minicluster-data/master-0/data/info.pb
I20260809 12:09:49.215595 1550 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260809 12:09:49.217166 888 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskgH5Pjm/build/asan/bin/kudu as pid 1480
I20260809 12:09:49.217414 888 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskgH5Pjm/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786277382073938-888-0/minicluster-data/master-0/wal/instance
I20260809 12:09:49.218089 888 external_mini_cluster.cc:1428] Setting key b2a864f536ba8a7e320e8e5a296c8deb
I20260809 12:09:49.227820 1550 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P aa483a268fb147afbbfa935435493e40: Bootstrap starting.
I20260809 12:09:49.231819 1550 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P aa483a268fb147afbbfa935435493e40: Neither blocks nor log segments found. Creating new log.
I20260809 12:09:49.233120 1550 log.cc:824] T 00000000000000000000000000000000 P aa483a268fb147afbbfa935435493e40: Log is configured to *not* fsync() on all Append() calls
Aug 09 12:09:49 dist-test-slave-2w3w krb5kdc[1465](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786277388, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.222.62@KRBTEST.COM
I20260809 12:09:49.236770 1550 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P aa483a268fb147afbbfa935435493e40: No bootstrap required, opened a new log
I20260809 12:09:49.243630 1553 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 12:09:49.220128 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43136 (local address 127.0.222.62:34593)
0809 12:09:49.221038 (+ 910us) server_negotiation.cc:206] Beginning negotiation
0809 12:09:49.221053 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0809 12:09:49.221098 (+ 45us) server_negotiation.cc:407] Connection header received
0809 12:09:49.222782 (+ 1684us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 12:09:49.222812 (+ 30us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 12:09:49.223250 (+ 438us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 12:09:49.223561 (+ 311us) server_negotiation.cc:226] Negotiated authn=SASL
0809 12:09:49.225485 (+ 1924us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 12:09:49.226882 (+ 1397us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 12:09:49.228632 (+ 1750us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 12:09:49.229321 (+ 689us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 12:09:49.229460 (+ 139us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 12:09:49.234660 (+ 5200us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 12:09:49.234674 (+ 14us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 12:09:49.234693 (+ 19us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0809 12:09:49.234734 (+ 41us) server_negotiation.cc:922] calling sasl_server_start()
0809 12:09:49.238787 (+ 4053us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 12:09:49.239519 (+ 732us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 12:09:49.239529 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 12:09:49.239542 (+ 13us) server_negotiation.cc:967] calling sasl_server_step()
0809 12:09:49.239762 (+ 220us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 12:09:49.240107 (+ 345us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 12:09:49.240116 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 12:09:49.240120 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0809 12:09:49.240848 (+ 728us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 12:09:49.241155 (+ 307us) server_negotiation.cc:1035] Waiting for connection context
0809 12:09:49.241576 (+ 421us) server_negotiation.cc:299] Negotiation successful
0809 12:09:49.242223 (+ 647us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":687,"thread_start_us":428,"threads_started":1}
I20260809 12:09:49.247427 1550 raft_consensus.cc:348] T 00000000000000000000000000000000 P aa483a268fb147afbbfa935435493e40 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "aa483a268fb147afbbfa935435493e40" member_type: VOTER last_known_addr { host: "127.0.222.62" port: 34593 } }
I20260809 12:09:49.247751 1550 raft_consensus.cc:374] T 00000000000000000000000000000000 P aa483a268fb147afbbfa935435493e40 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260809 12:09:49.247876 1550 raft_consensus.cc:729] T 00000000000000000000000000000000 P aa483a268fb147afbbfa935435493e40 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: aa483a268fb147afbbfa935435493e40, State: Initialized, Role: FOLLOWER
I20260809 12:09:49.248410 1550 consensus_queue.cc:260] T 00000000000000000000000000000000 P aa483a268fb147afbbfa935435493e40 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "aa483a268fb147afbbfa935435493e40" member_type: VOTER last_known_addr { host: "127.0.222.62" port: 34593 } }
I20260809 12:09:49.248627 1550 raft_consensus.cc:388] T 00000000000000000000000000000000 P aa483a268fb147afbbfa935435493e40 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260809 12:09:49.248728 1550 raft_consensus.cc:482] T 00000000000000000000000000000000 P aa483a268fb147afbbfa935435493e40 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260809 12:09:49.248852 1550 raft_consensus.cc:3037] T 00000000000000000000000000000000 P aa483a268fb147afbbfa935435493e40 [term 0 FOLLOWER]: Advancing to term 1
I20260809 12:09:49.251374 1550 raft_consensus.cc:504] T 00000000000000000000000000000000 P aa483a268fb147afbbfa935435493e40 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "aa483a268fb147afbbfa935435493e40" member_type: VOTER last_known_addr { host: "127.0.222.62" port: 34593 } }
I20260809 12:09:49.251822 1550 leader_election.cc:302] T 00000000000000000000000000000000 P aa483a268fb147afbbfa935435493e40 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: aa483a268fb147afbbfa935435493e40; no voters:
I20260809 12:09:49.252378 1550 leader_election.cc:288] T 00000000000000000000000000000000 P aa483a268fb147afbbfa935435493e40 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260809 12:09:49.252483 1555 raft_consensus.cc:2781] T 00000000000000000000000000000000 P aa483a268fb147afbbfa935435493e40 [term 1 FOLLOWER]: Leader election won for term 1
I20260809 12:09:49.253268 1555 raft_consensus.cc:686] T 00000000000000000000000000000000 P aa483a268fb147afbbfa935435493e40 [term 1 LEADER]: Becoming Leader. State: Replica: aa483a268fb147afbbfa935435493e40, State: Running, Role: LEADER
I20260809 12:09:49.253747 1555 consensus_queue.cc:237] T 00000000000000000000000000000000 P aa483a268fb147afbbfa935435493e40 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "aa483a268fb147afbbfa935435493e40" member_type: VOTER last_known_addr { host: "127.0.222.62" port: 34593 } }
I20260809 12:09:49.254505 1550 sys_catalog.cc:564] T 00000000000000000000000000000000 P aa483a268fb147afbbfa935435493e40 [sys.catalog]: configured and running, proceeding with master startup.
I20260809 12:09:49.256851 1557 sys_catalog.cc:455] T 00000000000000000000000000000000 P aa483a268fb147afbbfa935435493e40 [sys.catalog]: SysCatalogTable state changed. Reason: New leader aa483a268fb147afbbfa935435493e40. Latest consensus state: current_term: 1 leader_uuid: "aa483a268fb147afbbfa935435493e40" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "aa483a268fb147afbbfa935435493e40" member_type: VOTER last_known_addr { host: "127.0.222.62" port: 34593 } } }
I20260809 12:09:49.257297 1557 sys_catalog.cc:458] T 00000000000000000000000000000000 P aa483a268fb147afbbfa935435493e40 [sys.catalog]: This master's current role is: LEADER
I20260809 12:09:49.258145 1556 sys_catalog.cc:455] T 00000000000000000000000000000000 P aa483a268fb147afbbfa935435493e40 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "aa483a268fb147afbbfa935435493e40" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "aa483a268fb147afbbfa935435493e40" member_type: VOTER last_known_addr { host: "127.0.222.62" port: 34593 } } }
I20260809 12:09:49.258515 1556 sys_catalog.cc:458] T 00000000000000000000000000000000 P aa483a268fb147afbbfa935435493e40 [sys.catalog]: This master's current role is: LEADER
I20260809 12:09:49.260385 1562 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260809 12:09:49.266618 1562 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260809 12:09:49.274806 1562 catalog_manager.cc:1347] Generated new cluster ID: 7e57db65031f4183afaf69c850d1e0dc
I20260809 12:09:49.274926 1562 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260809 12:09:49.294638 1562 catalog_manager.cc:1370] Generated new certificate authority record
I20260809 12:09:49.296629 1562 catalog_manager.cc:1504] Loading token signing keys...
I20260809 12:09:49.304013 1562 catalog_manager.cc:5868] T 00000000000000000000000000000000 P aa483a268fb147afbbfa935435493e40: Generated new TSK 0
I20260809 12:09:49.304998 1562 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.0.222.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.222.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.222.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskgH5Pjm/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786277382073938-888-0/krb5kdc/kudu_127.0.222.1.keytab.
Entry for principal kudu/127.0.222.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskgH5Pjm/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786277382073938-888-0/krb5kdc/kudu_127.0.222.1.keytab.
I20260809 12:09:49.365329 888 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskgH5Pjm/build/asan/bin/kudu
/tmp/dist-test-taskgH5Pjm/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskgH5Pjm/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786277382073938-888-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskgH5Pjm/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786277382073938-888-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskgH5Pjm/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786277382073938-888-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskgH5Pjm/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786277382073938-888-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.222.1:0
--local_ip_for_outbound_sockets=127.0.222.1
--webserver_interface=127.0.222.1
--webserver_port=0
--tserver_master_addrs=127.0.222.62:34593
--keytab_file=/tmp/dist-test-taskgH5Pjm/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786277382073938-888-0/krb5kdc/kudu_127.0.222.1.keytab
--principal=kudu/127.0.222.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.222.20:38943
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_enforce_access_control with env {KRB5CCNAME=/tmp/dist-test-taskgH5Pjm/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786277382073938-888-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskgH5Pjm/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786277382073938-888-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskgH5Pjm/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786277382073938-888-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260809 12:09:49.695533 1576 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260809 12:09:49.695853 1576 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260809 12:09:49.696035 1576 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260809 12:09:49.703190 1576 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260809 12:09:49.703342 1576 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260809 12:09:49.703536 1576 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.222.1
I20260809 12:09:49.713258 1576 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.222.20:38943
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskgH5Pjm/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786277382073938-888-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskgH5Pjm/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786277382073938-888-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskgH5Pjm/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786277382073938-888-0/krb5kdc/kudu_127.0.222.1.keytab
--principal=kudu/127.0.222.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.222.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskgH5Pjm/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786277382073938-888-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.222.1
--webserver_port=0
--tserver_master_addrs=127.0.222.62:34593
--tserver_enforce_access_control=true
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.222.1
--log_dir=/tmp/dist-test-taskgH5Pjm/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786277382073938-888-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 09 Aug 2026 11:06:41 UTC on ccycloud.quasar-bwoogy.root.comops.site
build id 4575991
ASAN enabled
I20260809 12:09:49.714552 1576 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260809 12:09:49.716027 1576 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260809 12:09:49.729164 1582 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 12:09:49.729310 1583 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 12:09:49.729298 1586 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260809 12:09:49.730122 1576 server_base.cc:1034] running on GCE node
I20260809 12:09:49.730819 1576 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260809 12:09:49.731853 1576 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260809 12:09:49.733139 1576 hybrid_clock.cc:648] HybridClock initialized: now 1786277389733072 us; error 37 us; skew 500 ppm
I20260809 12:09:49.733471 1576 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 09 12:09:49 dist-test-slave-2w3w krb5kdc[1465](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786277389, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.222.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260809 12:09:49.736992 1576 init.cc:377] Logged in from keytab as kudu/127.0.222.1@KRBTEST.COM (short username kudu)
I20260809 12:09:49.738703 1576 webserver.cc:460] Webserver started at http://127.0.222.1:42975/ using document root <none> and password file <none>
I20260809 12:09:49.739496 1576 fs_manager.cc:359] Metadata directory not provided
I20260809 12:09:49.739599 1576 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260809 12:09:49.739959 1576 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260809 12:09:49.742918 1576 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskgH5Pjm/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786277382073938-888-0/minicluster-data/ts-0/data/instance:
uuid: "d8e905ef1c864d1ab535707a4951a95b"
format_stamp: "Formatted at 2026-08-09 12:09:49 on dist-test-slave-2w3w"
server_key: "0688addf5babe6f2cf78d248cb7e5a9b"
server_key_iv: "f7bb640463848418084036035a515a0a"
server_key_version: "encryptionkey@0"
I20260809 12:09:49.744135 1576 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskgH5Pjm/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786277382073938-888-0/minicluster-data/ts-0/wal/instance:
uuid: "d8e905ef1c864d1ab535707a4951a95b"
format_stamp: "Formatted at 2026-08-09 12:09:49 on dist-test-slave-2w3w"
server_key: "0688addf5babe6f2cf78d248cb7e5a9b"
server_key_iv: "f7bb640463848418084036035a515a0a"
server_key_version: "encryptionkey@0"
I20260809 12:09:49.749647 1576 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.000s
I20260809 12:09:49.753247 1594 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260809 12:09:49.754455 1576 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260809 12:09:49.754644 1576 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskgH5Pjm/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786277382073938-888-0/minicluster-data/ts-0/data,/tmp/dist-test-taskgH5Pjm/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786277382073938-888-0/minicluster-data/ts-0/wal
uuid: "d8e905ef1c864d1ab535707a4951a95b"
format_stamp: "Formatted at 2026-08-09 12:09:49 on dist-test-slave-2w3w"
server_key: "0688addf5babe6f2cf78d248cb7e5a9b"
server_key_iv: "f7bb640463848418084036035a515a0a"
server_key_version: "encryptionkey@0"
I20260809 12:09:49.754890 1576 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskgH5Pjm/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786277382073938-888-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskgH5Pjm/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786277382073938-888-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskgH5Pjm/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786277382073938-888-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260809 12:09:49.787029 1576 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260809 12:09:49.791527 1576 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260809 12:09:49.791842 1576 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260809 12:09:49.792892 1576 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260809 12:09:49.794263 1576 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260809 12:09:49.794371 1576 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260809 12:09:49.794533 1576 ts_tablet_manager.cc:610] Registered 0 tablets
I20260809 12:09:49.794610 1576 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260809 12:09:49.831427 1576 rpc_server.cc:307] RPC server started. Bound to: 127.0.222.1:43353
I20260809 12:09:49.831488 1713 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.222.1:43353 every 8 connection(s)
I20260809 12:09:49.832839 1576 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskgH5Pjm/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786277382073938-888-0/minicluster-data/ts-0/data/info.pb
I20260809 12:09:49.837668 888 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskgH5Pjm/build/asan/bin/kudu as pid 1576
I20260809 12:09:49.837898 888 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskgH5Pjm/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786277382073938-888-0/minicluster-data/ts-0/wal/instance
I20260809 12:09:49.838505 888 external_mini_cluster.cc:1428] Setting key 2ca287f57181ccd8e552f862e15470b1
Aug 09 12:09:49 dist-test-slave-2w3w krb5kdc[1465](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786277389, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.222.1@KRBTEST.COM for kudu/127.0.222.62@KRBTEST.COM
I20260809 12:09:49.853718 1716 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 12:09:49.835506 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.222.1:58307 (local address 127.0.222.62:34593)
0809 12:09:49.836075 (+ 569us) server_negotiation.cc:206] Beginning negotiation
0809 12:09:49.836080 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0809 12:09:49.836997 (+ 917us) server_negotiation.cc:407] Connection header received
0809 12:09:49.839019 (+ 2022us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 12:09:49.839024 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 12:09:49.839121 (+ 97us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 12:09:49.839246 (+ 125us) server_negotiation.cc:226] Negotiated authn=SASL
0809 12:09:49.841014 (+ 1768us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 12:09:49.841739 (+ 725us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 12:09:49.843387 (+ 1648us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 12:09:49.843793 (+ 406us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 12:09:49.843941 (+ 148us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 12:09:49.847409 (+ 3468us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 12:09:49.847414 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 12:09:49.847418 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0809 12:09:49.847450 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0809 12:09:49.850402 (+ 2952us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 12:09:49.851361 (+ 959us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 12:09:49.851365 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 12:09:49.851367 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0809 12:09:49.851504 (+ 137us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 12:09:49.851995 (+ 491us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 12:09:49.851999 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 12:09:49.852000 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0809 12:09:49.852256 (+ 256us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 12:09:49.852371 (+ 115us) server_negotiation.cc:1035] Waiting for connection context
0809 12:09:49.853290 (+ 919us) server_negotiation.cc:299] Negotiation successful
0809 12:09:49.853465 (+ 175us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":458,"thread_start_us":334,"threads_started":1}
I20260809 12:09:49.855254 1717 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 12:09:49.836129 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.222.62:34593 (local address 127.0.222.1:58307)
0809 12:09:49.836831 (+ 702us) negotiation.cc:107] Waiting for socket to connect
0809 12:09:49.836876 (+ 45us) client_negotiation.cc:174] Beginning negotiation
0809 12:09:49.838738 (+ 1862us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 12:09:49.839516 (+ 778us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 12:09:49.839529 (+ 13us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 12:09:49.840271 (+ 742us) client_negotiation.cc:189] Negotiated authn=SASL
0809 12:09:49.840851 (+ 580us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 12:09:49.840868 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 12:09:49.841921 (+ 1053us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 12:09:49.841925 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 12:09:49.843234 (+ 1309us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 12:09:49.843246 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 12:09:49.844050 (+ 804us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 12:09:49.844058 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 12:09:49.844287 (+ 229us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 12:09:49.844973 (+ 686us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0809 12:09:49.844995 (+ 22us) client_negotiation.cc:656] calling sasl_client_start()
0809 12:09:49.847190 (+ 2195us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0809 12:09:49.850539 (+ 3349us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0809 12:09:49.850550 (+ 11us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0809 12:09:49.850569 (+ 19us) client_negotiation.cc:760] calling sasl_client_step()
0809 12:09:49.851237 (+ 668us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0809 12:09:49.851622 (+ 385us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0809 12:09:49.851629 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0809 12:09:49.851636 (+ 7us) client_negotiation.cc:760] calling sasl_client_step()
0809 12:09:49.851882 (+ 246us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0809 12:09:49.852387 (+ 505us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0809 12:09:49.852398 (+ 11us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0809 12:09:49.852843 (+ 445us) client_negotiation.cc:769] Sending connection context
0809 12:09:49.853161 (+ 318us) client_negotiation.cc:240] Negotiation successful
0809 12:09:49.853710 (+ 549us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":472,"thread_start_us":299,"threads_started":1}
I20260809 12:09:49.856993 1714 heartbeater.cc:347] Connected to a master server at 127.0.222.62:34593
I20260809 12:09:49.857313 1714 heartbeater.cc:464] Registering TS with master...
I20260809 12:09:49.858258 1714 heartbeater.cc:511] Master 127.0.222.62:34593 requested a full tablet report, sending...
I20260809 12:09:49.860893 1513 ts_manager.cc:194] Registered new tserver with Master: d8e905ef1c864d1ab535707a4951a95b (127.0.222.1:43353)
WARNING: no policy specified for kudu/127.0.222.2@KRBTEST.COM; defaulting to no policy
I20260809 12:09:49.863131 1513 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.222.1@KRBTEST.COM'} at 127.0.222.1:58307
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.222.2@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.222.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskgH5Pjm/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786277382073938-888-0/krb5kdc/kudu_127.0.222.2.keytab.
Entry for principal kudu/127.0.222.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskgH5Pjm/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786277382073938-888-0/krb5kdc/kudu_127.0.222.2.keytab.
I20260809 12:09:49.888283 888 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskgH5Pjm/build/asan/bin/kudu
/tmp/dist-test-taskgH5Pjm/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskgH5Pjm/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786277382073938-888-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskgH5Pjm/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786277382073938-888-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskgH5Pjm/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786277382073938-888-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskgH5Pjm/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786277382073938-888-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.222.2:0
--local_ip_for_outbound_sockets=127.0.222.2
--webserver_interface=127.0.222.2
--webserver_port=0
--tserver_master_addrs=127.0.222.62:34593
--keytab_file=/tmp/dist-test-taskgH5Pjm/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786277382073938-888-0/krb5kdc/kudu_127.0.222.2.keytab
--principal=kudu/127.0.222.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.222.20:38943
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_enforce_access_control with env {KRB5CCNAME=/tmp/dist-test-taskgH5Pjm/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786277382073938-888-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskgH5Pjm/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786277382073938-888-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskgH5Pjm/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786277382073938-888-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260809 12:09:50.207883 1720 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260809 12:09:50.208241 1720 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260809 12:09:50.208413 1720 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260809 12:09:50.215886 1720 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260809 12:09:50.216044 1720 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260809 12:09:50.216223 1720 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.222.2
I20260809 12:09:50.225996 1720 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.222.20:38943
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskgH5Pjm/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786277382073938-888-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskgH5Pjm/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786277382073938-888-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskgH5Pjm/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786277382073938-888-0/krb5kdc/kudu_127.0.222.2.keytab
--principal=kudu/127.0.222.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.222.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskgH5Pjm/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786277382073938-888-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.222.2
--webserver_port=0
--tserver_master_addrs=127.0.222.62:34593
--tserver_enforce_access_control=true
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.222.2
--log_dir=/tmp/dist-test-taskgH5Pjm/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786277382073938-888-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 09 Aug 2026 11:06:41 UTC on ccycloud.quasar-bwoogy.root.comops.site
build id 4575991
ASAN enabled
I20260809 12:09:50.227414 1720 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260809 12:09:50.228730 1720 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260809 12:09:50.242070 1727 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 12:09:50.242154 1730 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 12:09:50.242033 1728 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260809 12:09:50.243287 1720 server_base.cc:1034] running on GCE node
I20260809 12:09:50.243948 1720 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260809 12:09:50.245013 1720 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260809 12:09:50.246241 1720 hybrid_clock.cc:648] HybridClock initialized: now 1786277390246197 us; error 50 us; skew 500 ppm
I20260809 12:09:50.246567 1720 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 09 12:09:50 dist-test-slave-2w3w krb5kdc[1465](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786277390, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.222.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260809 12:09:50.249774 1720 init.cc:377] Logged in from keytab as kudu/127.0.222.2@KRBTEST.COM (short username kudu)
I20260809 12:09:50.251351 1720 webserver.cc:460] Webserver started at http://127.0.222.2:40127/ using document root <none> and password file <none>
I20260809 12:09:50.252071 1720 fs_manager.cc:359] Metadata directory not provided
I20260809 12:09:50.252209 1720 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260809 12:09:50.252558 1720 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260809 12:09:50.255762 1720 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskgH5Pjm/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786277382073938-888-0/minicluster-data/ts-1/data/instance:
uuid: "8d1775c7b3c54f22bf644d4557c6d106"
format_stamp: "Formatted at 2026-08-09 12:09:50 on dist-test-slave-2w3w"
server_key: "c09d4e838fa13005ad13d0735c93130f"
server_key_iv: "f784be9a56f96accd39eb52100e70bd3"
server_key_version: "encryptionkey@0"
I20260809 12:09:50.256654 1720 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskgH5Pjm/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786277382073938-888-0/minicluster-data/ts-1/wal/instance:
uuid: "8d1775c7b3c54f22bf644d4557c6d106"
format_stamp: "Formatted at 2026-08-09 12:09:50 on dist-test-slave-2w3w"
server_key: "c09d4e838fa13005ad13d0735c93130f"
server_key_iv: "f784be9a56f96accd39eb52100e70bd3"
server_key_version: "encryptionkey@0"
I20260809 12:09:50.262032 1720 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.000s
I20260809 12:09:50.265398 1737 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260809 12:09:50.266669 1720 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260809 12:09:50.266845 1720 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskgH5Pjm/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786277382073938-888-0/minicluster-data/ts-1/data,/tmp/dist-test-taskgH5Pjm/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786277382073938-888-0/minicluster-data/ts-1/wal
uuid: "8d1775c7b3c54f22bf644d4557c6d106"
format_stamp: "Formatted at 2026-08-09 12:09:50 on dist-test-slave-2w3w"
server_key: "c09d4e838fa13005ad13d0735c93130f"
server_key_iv: "f784be9a56f96accd39eb52100e70bd3"
server_key_version: "encryptionkey@0"
I20260809 12:09:50.267117 1720 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskgH5Pjm/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786277382073938-888-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskgH5Pjm/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786277382073938-888-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskgH5Pjm/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786277382073938-888-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260809 12:09:50.303956 1720 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260809 12:09:50.308475 1720 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260809 12:09:50.308789 1720 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260809 12:09:50.309760 1720 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260809 12:09:50.311157 1720 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260809 12:09:50.311266 1720 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260809 12:09:50.311420 1720 ts_tablet_manager.cc:610] Registered 0 tablets
I20260809 12:09:50.311493 1720 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260809 12:09:50.344729 1720 rpc_server.cc:307] RPC server started. Bound to: 127.0.222.2:44865
I20260809 12:09:50.344817 1851 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.222.2:44865 every 8 connection(s)
I20260809 12:09:50.346011 1720 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskgH5Pjm/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786277382073938-888-0/minicluster-data/ts-1/data/info.pb
I20260809 12:09:50.346365 888 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskgH5Pjm/build/asan/bin/kudu as pid 1720
I20260809 12:09:50.346576 888 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskgH5Pjm/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786277382073938-888-0/minicluster-data/ts-1/wal/instance
I20260809 12:09:50.347226 888 external_mini_cluster.cc:1428] Setting key eab764a9a58b1a2f8739fa5976b93925
Aug 09 12:09:50 dist-test-slave-2w3w krb5kdc[1465](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786277390, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.222.2@KRBTEST.COM for kudu/127.0.222.62@KRBTEST.COM
I20260809 12:09:50.367540 1716 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 12:09:50.349300 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.222.2:44817 (local address 127.0.222.62:34593)
0809 12:09:50.349511 (+ 211us) server_negotiation.cc:206] Beginning negotiation
0809 12:09:50.349520 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0809 12:09:50.350225 (+ 705us) server_negotiation.cc:407] Connection header received
0809 12:09:50.352117 (+ 1892us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 12:09:50.352125 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 12:09:50.352222 (+ 97us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 12:09:50.352339 (+ 117us) server_negotiation.cc:226] Negotiated authn=SASL
0809 12:09:50.353748 (+ 1409us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 12:09:50.354930 (+ 1182us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 12:09:50.356993 (+ 2063us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 12:09:50.357423 (+ 430us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 12:09:50.357528 (+ 105us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 12:09:50.361525 (+ 3997us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 12:09:50.361530 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 12:09:50.361533 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0809 12:09:50.361566 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0809 12:09:50.364402 (+ 2836us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 12:09:50.365162 (+ 760us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 12:09:50.365172 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 12:09:50.365174 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0809 12:09:50.365305 (+ 131us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 12:09:50.365756 (+ 451us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 12:09:50.365761 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 12:09:50.365764 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0809 12:09:50.366045 (+ 281us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 12:09:50.366204 (+ 159us) server_negotiation.cc:1035] Waiting for connection context
0809 12:09:50.366905 (+ 701us) server_negotiation.cc:299] Negotiation successful
0809 12:09:50.367175 (+ 270us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":70}
WARNING: no policy specified for kudu/127.0.222.3@KRBTEST.COM; defaulting to no policy
I20260809 12:09:50.368587 1855 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 12:09:50.349361 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.222.62:34593 (local address 127.0.222.2:44817)
0809 12:09:50.350066 (+ 705us) negotiation.cc:107] Waiting for socket to connect
0809 12:09:50.350101 (+ 35us) client_negotiation.cc:174] Beginning negotiation
0809 12:09:50.351850 (+ 1749us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 12:09:50.352536 (+ 686us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 12:09:50.352543 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 12:09:50.353032 (+ 489us) client_negotiation.cc:189] Negotiated authn=SASL
0809 12:09:50.353578 (+ 546us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 12:09:50.353595 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 12:09:50.355113 (+ 1518us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 12:09:50.355117 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 12:09:50.356824 (+ 1707us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 12:09:50.356840 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 12:09:50.357654 (+ 814us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 12:09:50.357663 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 12:09:50.357959 (+ 296us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 12:09:50.358679 (+ 720us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0809 12:09:50.358701 (+ 22us) client_negotiation.cc:656] calling sasl_client_start()
0809 12:09:50.361291 (+ 2590us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0809 12:09:50.364549 (+ 3258us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0809 12:09:50.364559 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0809 12:09:50.364576 (+ 17us) client_negotiation.cc:760] calling sasl_client_step()
0809 12:09:50.365027 (+ 451us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0809 12:09:50.365429 (+ 402us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0809 12:09:50.365435 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0809 12:09:50.365438 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0809 12:09:50.365639 (+ 201us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0809 12:09:50.366176 (+ 537us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0809 12:09:50.366184 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0809 12:09:50.366539 (+ 355us) client_negotiation.cc:769] Sending connection context
0809 12:09:50.366806 (+ 267us) client_negotiation.cc:240] Negotiation successful
0809 12:09:50.367391 (+ 585us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":480,"thread_start_us":1561,"threads_started":1}
I20260809 12:09:50.370114 1852 heartbeater.cc:347] Connected to a master server at 127.0.222.62:34593
I20260809 12:09:50.370515 1852 heartbeater.cc:464] Registering TS with master...
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.222.3@KRBTEST.COM" created.
I20260809 12:09:50.371490 1852 heartbeater.cc:511] Master 127.0.222.62:34593 requested a full tablet report, sending...
I20260809 12:09:50.373061 1513 ts_manager.cc:194] Registered new tserver with Master: 8d1775c7b3c54f22bf644d4557c6d106 (127.0.222.2:44865)
I20260809 12:09:50.374349 1513 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.222.2@KRBTEST.COM'} at 127.0.222.2:44817
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.222.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskgH5Pjm/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786277382073938-888-0/krb5kdc/kudu_127.0.222.3.keytab.
Entry for principal kudu/127.0.222.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskgH5Pjm/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786277382073938-888-0/krb5kdc/kudu_127.0.222.3.keytab.
I20260809 12:09:50.396255 888 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskgH5Pjm/build/asan/bin/kudu
/tmp/dist-test-taskgH5Pjm/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskgH5Pjm/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786277382073938-888-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskgH5Pjm/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786277382073938-888-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskgH5Pjm/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786277382073938-888-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskgH5Pjm/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786277382073938-888-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.222.3:0
--local_ip_for_outbound_sockets=127.0.222.3
--webserver_interface=127.0.222.3
--webserver_port=0
--tserver_master_addrs=127.0.222.62:34593
--keytab_file=/tmp/dist-test-taskgH5Pjm/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786277382073938-888-0/krb5kdc/kudu_127.0.222.3.keytab
--principal=kudu/127.0.222.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.222.20:38943
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_enforce_access_control with env {KRB5CCNAME=/tmp/dist-test-taskgH5Pjm/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786277382073938-888-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskgH5Pjm/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786277382073938-888-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskgH5Pjm/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786277382073938-888-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260809 12:09:50.736050 1858 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260809 12:09:50.736344 1858 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260809 12:09:50.736444 1858 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260809 12:09:50.743767 1858 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260809 12:09:50.743876 1858 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260809 12:09:50.743995 1858 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.222.3
I20260809 12:09:50.753592 1858 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.222.20:38943
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskgH5Pjm/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786277382073938-888-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskgH5Pjm/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786277382073938-888-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskgH5Pjm/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786277382073938-888-0/krb5kdc/kudu_127.0.222.3.keytab
--principal=kudu/127.0.222.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.222.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskgH5Pjm/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786277382073938-888-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.222.3
--webserver_port=0
--tserver_master_addrs=127.0.222.62:34593
--tserver_enforce_access_control=true
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.222.3
--log_dir=/tmp/dist-test-taskgH5Pjm/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786277382073938-888-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 09 Aug 2026 11:06:41 UTC on ccycloud.quasar-bwoogy.root.comops.site
build id 4575991
ASAN enabled
I20260809 12:09:50.754894 1858 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260809 12:09:50.756421 1858 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260809 12:09:50.769431 1867 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 12:09:50.769445 1864 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 12:09:50.769734 1865 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260809 12:09:50.770169 1858 server_base.cc:1034] running on GCE node
I20260809 12:09:50.770872 1858 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260809 12:09:50.772002 1858 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260809 12:09:50.773221 1858 hybrid_clock.cc:648] HybridClock initialized: now 1786277390773158 us; error 65 us; skew 500 ppm
I20260809 12:09:50.773525 1858 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 09 12:09:50 dist-test-slave-2w3w krb5kdc[1465](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786277390, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.222.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260809 12:09:50.776868 1858 init.cc:377] Logged in from keytab as kudu/127.0.222.3@KRBTEST.COM (short username kudu)
I20260809 12:09:50.778280 1858 webserver.cc:460] Webserver started at http://127.0.222.3:40377/ using document root <none> and password file <none>
I20260809 12:09:50.778918 1858 fs_manager.cc:359] Metadata directory not provided
I20260809 12:09:50.779073 1858 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260809 12:09:50.779330 1858 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260809 12:09:50.782097 1858 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskgH5Pjm/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786277382073938-888-0/minicluster-data/ts-2/data/instance:
uuid: "a7fa8439659a45ee8807f9f491a96627"
format_stamp: "Formatted at 2026-08-09 12:09:50 on dist-test-slave-2w3w"
server_key: "d585b99229db3e19157c5a9b74e078dd"
server_key_iv: "a61a48fbd28e014835dfd6a3d52bde09"
server_key_version: "encryptionkey@0"
I20260809 12:09:50.782835 1858 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskgH5Pjm/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786277382073938-888-0/minicluster-data/ts-2/wal/instance:
uuid: "a7fa8439659a45ee8807f9f491a96627"
format_stamp: "Formatted at 2026-08-09 12:09:50 on dist-test-slave-2w3w"
server_key: "d585b99229db3e19157c5a9b74e078dd"
server_key_iv: "a61a48fbd28e014835dfd6a3d52bde09"
server_key_version: "encryptionkey@0"
I20260809 12:09:50.787961 1858 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.003s sys 0.001s
I20260809 12:09:50.791379 1874 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260809 12:09:50.792689 1858 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.002s
I20260809 12:09:50.792865 1858 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskgH5Pjm/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786277382073938-888-0/minicluster-data/ts-2/data,/tmp/dist-test-taskgH5Pjm/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786277382073938-888-0/minicluster-data/ts-2/wal
uuid: "a7fa8439659a45ee8807f9f491a96627"
format_stamp: "Formatted at 2026-08-09 12:09:50 on dist-test-slave-2w3w"
server_key: "d585b99229db3e19157c5a9b74e078dd"
server_key_iv: "a61a48fbd28e014835dfd6a3d52bde09"
server_key_version: "encryptionkey@0"
I20260809 12:09:50.793043 1858 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskgH5Pjm/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786277382073938-888-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskgH5Pjm/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786277382073938-888-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskgH5Pjm/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786277382073938-888-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260809 12:09:50.840868 1858 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260809 12:09:50.845206 1858 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260809 12:09:50.845445 1858 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260809 12:09:50.846344 1858 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260809 12:09:50.847674 1858 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260809 12:09:50.847760 1858 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260809 12:09:50.847909 1858 ts_tablet_manager.cc:610] Registered 0 tablets
I20260809 12:09:50.847963 1858 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260809 12:09:50.866045 1714 heartbeater.cc:503] Master 127.0.222.62:34593 was elected leader, sending a full tablet report...
I20260809 12:09:50.885187 1858 rpc_server.cc:307] RPC server started. Bound to: 127.0.222.3:35771
I20260809 12:09:50.885370 1996 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.222.3:35771 every 8 connection(s)
I20260809 12:09:50.886637 1858 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskgH5Pjm/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786277382073938-888-0/minicluster-data/ts-2/data/info.pb
I20260809 12:09:50.888646 888 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskgH5Pjm/build/asan/bin/kudu as pid 1858
I20260809 12:09:50.888932 888 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskgH5Pjm/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786277382073938-888-0/minicluster-data/ts-2/wal/instance
I20260809 12:09:50.889614 888 external_mini_cluster.cc:1428] Setting key ffaf93b803f114333f5670b15eca52f7
Aug 09 12:09:50 dist-test-slave-2w3w krb5kdc[1465](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786277390, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.222.3@KRBTEST.COM for kudu/127.0.222.62@KRBTEST.COM
I20260809 12:09:50.912439 1999 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 12:09:50.890300 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.222.3:59641 (local address 127.0.222.62:34593)
0809 12:09:50.891096 (+ 796us) server_negotiation.cc:206] Beginning negotiation
0809 12:09:50.891102 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0809 12:09:50.892732 (+ 1630us) server_negotiation.cc:407] Connection header received
0809 12:09:50.895518 (+ 2786us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 12:09:50.895522 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 12:09:50.895590 (+ 68us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 12:09:50.895690 (+ 100us) server_negotiation.cc:226] Negotiated authn=SASL
0809 12:09:50.897580 (+ 1890us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 12:09:50.898757 (+ 1177us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 12:09:50.900726 (+ 1969us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 12:09:50.901271 (+ 545us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 12:09:50.901374 (+ 103us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 12:09:50.905766 (+ 4392us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 12:09:50.905772 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 12:09:50.905776 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0809 12:09:50.905822 (+ 46us) server_negotiation.cc:922] calling sasl_server_start()
0809 12:09:50.909051 (+ 3229us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 12:09:50.909882 (+ 831us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 12:09:50.909886 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 12:09:50.909888 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0809 12:09:50.910030 (+ 142us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 12:09:50.910533 (+ 503us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 12:09:50.910536 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 12:09:50.910539 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0809 12:09:50.910834 (+ 295us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 12:09:50.910947 (+ 113us) server_negotiation.cc:1035] Waiting for connection context
0809 12:09:50.911820 (+ 873us) server_negotiation.cc:299] Negotiation successful
0809 12:09:50.911989 (+ 169us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":660,"thread_start_us":527,"threads_started":1}
I20260809 12:09:50.913403 2000 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 12:09:50.891105 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.222.62:34593 (local address 127.0.222.3:59641)
0809 12:09:50.892559 (+ 1454us) negotiation.cc:107] Waiting for socket to connect
0809 12:09:50.892608 (+ 49us) client_negotiation.cc:174] Beginning negotiation
0809 12:09:50.895248 (+ 2640us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 12:09:50.895908 (+ 660us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 12:09:50.895916 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 12:09:50.896566 (+ 650us) client_negotiation.cc:189] Negotiated authn=SASL
0809 12:09:50.897398 (+ 832us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 12:09:50.897418 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 12:09:50.898952 (+ 1534us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 12:09:50.899004 (+ 52us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 12:09:50.900591 (+ 1587us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 12:09:50.900603 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 12:09:50.901515 (+ 912us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 12:09:50.901521 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 12:09:50.901736 (+ 215us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 12:09:50.902569 (+ 833us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0809 12:09:50.902604 (+ 35us) client_negotiation.cc:656] calling sasl_client_start()
0809 12:09:50.905555 (+ 2951us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0809 12:09:50.909182 (+ 3627us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0809 12:09:50.909189 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0809 12:09:50.909201 (+ 12us) client_negotiation.cc:760] calling sasl_client_step()
0809 12:09:50.909768 (+ 567us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0809 12:09:50.910131 (+ 363us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0809 12:09:50.910135 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0809 12:09:50.910137 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0809 12:09:50.910422 (+ 285us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0809 12:09:50.910948 (+ 526us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0809 12:09:50.910954 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0809 12:09:50.911430 (+ 476us) client_negotiation.cc:769] Sending connection context
0809 12:09:50.911662 (+ 232us) client_negotiation.cc:240] Negotiation successful
0809 12:09:50.912068 (+ 406us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1082,"thread_start_us":893,"threads_started":1}
I20260809 12:09:50.914880 1997 heartbeater.cc:347] Connected to a master server at 127.0.222.62:34593
I20260809 12:09:50.915232 1997 heartbeater.cc:464] Registering TS with master...
I20260809 12:09:50.916134 1997 heartbeater.cc:511] Master 127.0.222.62:34593 requested a full tablet report, sending...
I20260809 12:09:50.917965 1513 ts_manager.cc:194] Registered new tserver with Master: a7fa8439659a45ee8807f9f491a96627 (127.0.222.3:35771)
I20260809 12:09:50.918670 888 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260809 12:09:50.919088 1513 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.222.3@KRBTEST.COM'} at 127.0.222.3:59641
I20260809 12:09:50.945411 1999 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 12:09:50.927919 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43140 (local address 127.0.222.62:34593)
0809 12:09:50.928107 (+ 188us) server_negotiation.cc:206] Beginning negotiation
0809 12:09:50.928112 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0809 12:09:50.929152 (+ 1040us) server_negotiation.cc:407] Connection header received
0809 12:09:50.929389 (+ 237us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 12:09:50.929393 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 12:09:50.929456 (+ 63us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 12:09:50.929563 (+ 107us) server_negotiation.cc:226] Negotiated authn=SASL
0809 12:09:50.931898 (+ 2335us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 12:09:50.932938 (+ 1040us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 12:09:50.934610 (+ 1672us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 12:09:50.935196 (+ 586us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 12:09:50.935336 (+ 140us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 12:09:50.938588 (+ 3252us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 12:09:50.938593 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 12:09:50.938596 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0809 12:09:50.938628 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0809 12:09:50.942034 (+ 3406us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 12:09:50.942774 (+ 740us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 12:09:50.942782 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 12:09:50.942788 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0809 12:09:50.943007 (+ 219us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 12:09:50.943457 (+ 450us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 12:09:50.943463 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 12:09:50.943466 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0809 12:09:50.943923 (+ 457us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 12:09:50.944296 (+ 373us) server_negotiation.cc:1035] Waiting for connection context
0809 12:09:50.944678 (+ 382us) server_negotiation.cc:299] Negotiation successful
0809 12:09:50.944944 (+ 266us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":80}
I20260809 12:09:50.954754 1513 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:43140:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260809 12:09:50.965534 1513 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260809 12:09:51.002521 2011 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 12:09:50.985572 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42410 (local address 127.0.222.2:44865)
0809 12:09:50.986839 (+ 1267us) server_negotiation.cc:206] Beginning negotiation
0809 12:09:50.986849 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0809 12:09:50.987743 (+ 894us) server_negotiation.cc:407] Connection header received
0809 12:09:50.987852 (+ 109us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 12:09:50.987859 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 12:09:50.988256 (+ 397us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 12:09:50.988584 (+ 328us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0809 12:09:50.992707 (+ 4123us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 12:09:50.994562 (+ 1855us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 12:09:50.999160 (+ 4598us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 12:09:51.001036 (+ 1876us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 12:09:51.001223 (+ 187us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 12:09:51.001452 (+ 229us) server_negotiation.cc:1035] Waiting for connection context
0809 12:09:51.001890 (+ 438us) server_negotiation.cc:299] Negotiation successful
0809 12:09:51.002072 (+ 182us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1006,"thread_start_us":801,"threads_started":1}
I20260809 12:09:51.002750 2012 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 12:09:50.985304 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.222.2:44865 (local address 127.0.0.1:42410)
0809 12:09:50.987119 (+ 1815us) negotiation.cc:107] Waiting for socket to connect
0809 12:09:50.987165 (+ 46us) client_negotiation.cc:174] Beginning negotiation
0809 12:09:50.987456 (+ 291us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 12:09:50.989077 (+ 1621us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 12:09:50.989085 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 12:09:50.989118 (+ 33us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0809 12:09:50.992431 (+ 3313us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 12:09:50.992448 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 12:09:50.995588 (+ 3140us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 12:09:50.995594 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 12:09:50.998927 (+ 3333us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 12:09:50.998939 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 12:09:51.001324 (+ 2385us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 12:09:51.001331 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 12:09:51.001633 (+ 302us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 12:09:51.001673 (+ 40us) client_negotiation.cc:769] Sending connection context
0809 12:09:51.001811 (+ 138us) client_negotiation.cc:240] Negotiation successful
0809 12:09:51.002056 (+ 245us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1389,"mutex_wait_us":1,"thread_start_us":1708,"threads_started":1}
I20260809 12:09:51.003299 2013 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 12:09:50.986458 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56852 (local address 127.0.222.1:43353)
0809 12:09:50.989026 (+ 2568us) server_negotiation.cc:206] Beginning negotiation
0809 12:09:50.989034 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0809 12:09:50.989060 (+ 26us) server_negotiation.cc:407] Connection header received
0809 12:09:50.989166 (+ 106us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 12:09:50.989175 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 12:09:50.989465 (+ 290us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 12:09:50.989720 (+ 255us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0809 12:09:50.991800 (+ 2080us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 12:09:50.994365 (+ 2565us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 12:09:50.998585 (+ 4220us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 12:09:51.000840 (+ 2255us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 12:09:51.001040 (+ 200us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 12:09:51.001386 (+ 346us) server_negotiation.cc:1035] Waiting for connection context
0809 12:09:51.002282 (+ 896us) server_negotiation.cc:299] Negotiation successful
0809 12:09:51.002535 (+ 253us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":2367,"thread_start_us":1973,"threads_started":1}
I20260809 12:09:51.004616 2010 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 12:09:50.985595 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.222.1:43353 (local address 127.0.0.1:56852)
0809 12:09:50.987696 (+ 2101us) negotiation.cc:107] Waiting for socket to connect
0809 12:09:50.987715 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0809 12:09:50.987841 (+ 126us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 12:09:50.989750 (+ 1909us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 12:09:50.989759 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 12:09:50.989800 (+ 41us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0809 12:09:50.991479 (+ 1679us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 12:09:50.991497 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 12:09:50.994506 (+ 3009us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 12:09:50.994512 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 12:09:50.998353 (+ 3841us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 12:09:50.998371 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 12:09:51.001762 (+ 3391us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 12:09:51.001772 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 12:09:51.002054 (+ 282us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 12:09:51.002086 (+ 32us) client_negotiation.cc:769] Sending connection context
0809 12:09:51.003584 (+ 1498us) client_negotiation.cc:240] Negotiation successful
0809 12:09:51.003763 (+ 179us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1904,"mutex_wait_us":44,"thread_start_us":1216,"threads_started":1}
I20260809 12:09:51.005463 2014 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 12:09:50.987756 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.222.3:35771 (local address 127.0.0.1:45320)
0809 12:09:50.991638 (+ 3882us) negotiation.cc:107] Waiting for socket to connect
0809 12:09:50.991658 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0809 12:09:50.991797 (+ 139us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 12:09:50.992734 (+ 937us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 12:09:50.992740 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 12:09:50.992776 (+ 36us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0809 12:09:50.993774 (+ 998us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 12:09:50.993790 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 12:09:50.996682 (+ 2892us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 12:09:50.996692 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 12:09:51.000170 (+ 3478us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 12:09:51.000186 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 12:09:51.003509 (+ 3323us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 12:09:51.003516 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 12:09:51.003779 (+ 263us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 12:09:51.003823 (+ 44us) client_negotiation.cc:769] Sending connection context
0809 12:09:51.003930 (+ 107us) client_negotiation.cc:240] Negotiation successful
0809 12:09:51.004115 (+ 185us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":3701,"mutex_wait_us":26,"thread_start_us":3317,"threads_started":1}
I20260809 12:09:51.006228 2015 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 12:09:50.989004 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45320 (local address 127.0.222.3:35771)
0809 12:09:50.992113 (+ 3109us) server_negotiation.cc:206] Beginning negotiation
0809 12:09:50.992121 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0809 12:09:50.992187 (+ 66us) server_negotiation.cc:407] Connection header received
0809 12:09:50.992329 (+ 142us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 12:09:50.992335 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 12:09:50.992475 (+ 140us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 12:09:50.992742 (+ 267us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0809 12:09:50.994127 (+ 1385us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 12:09:50.996487 (+ 2360us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 12:09:51.000376 (+ 3889us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 12:09:51.002553 (+ 2177us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 12:09:51.003021 (+ 468us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 12:09:51.003339 (+ 318us) server_negotiation.cc:1035] Waiting for connection context
0809 12:09:51.004326 (+ 987us) server_negotiation.cc:299] Negotiation successful
0809 12:09:51.004596 (+ 270us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":2133,"thread_start_us":1721,"threads_started":1}
I20260809 12:09:51.008309 1787 tablet_service.cc:1467] Processing CreateTablet for tablet 1e69bcaf3f22453b8e7f32bf6821701a (DEFAULT_TABLE table=test-table [id=22592d0ce7834ec6b8c867093b26b858]), partition=RANGE (key) PARTITION UNBOUNDED
I20260809 12:09:51.010339 1787 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 1e69bcaf3f22453b8e7f32bf6821701a. 1 dirs total, 0 dirs full, 0 dirs failed
I20260809 12:09:51.009918 1926 tablet_service.cc:1467] Processing CreateTablet for tablet 1e69bcaf3f22453b8e7f32bf6821701a (DEFAULT_TABLE table=test-table [id=22592d0ce7834ec6b8c867093b26b858]), partition=RANGE (key) PARTITION UNBOUNDED
I20260809 12:09:51.011379 1926 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 1e69bcaf3f22453b8e7f32bf6821701a. 1 dirs total, 0 dirs full, 0 dirs failed
I20260809 12:09:51.012494 1647 tablet_service.cc:1467] Processing CreateTablet for tablet 1e69bcaf3f22453b8e7f32bf6821701a (DEFAULT_TABLE table=test-table [id=22592d0ce7834ec6b8c867093b26b858]), partition=RANGE (key) PARTITION UNBOUNDED
I20260809 12:09:51.014374 1647 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 1e69bcaf3f22453b8e7f32bf6821701a. 1 dirs total, 0 dirs full, 0 dirs failed
I20260809 12:09:51.030368 2016 tablet_bootstrap.cc:492] T 1e69bcaf3f22453b8e7f32bf6821701a P a7fa8439659a45ee8807f9f491a96627: Bootstrap starting.
I20260809 12:09:51.032166 2017 tablet_bootstrap.cc:492] T 1e69bcaf3f22453b8e7f32bf6821701a P 8d1775c7b3c54f22bf644d4557c6d106: Bootstrap starting.
I20260809 12:09:51.035118 2018 tablet_bootstrap.cc:492] T 1e69bcaf3f22453b8e7f32bf6821701a P d8e905ef1c864d1ab535707a4951a95b: Bootstrap starting.
I20260809 12:09:51.037563 2016 tablet_bootstrap.cc:654] T 1e69bcaf3f22453b8e7f32bf6821701a P a7fa8439659a45ee8807f9f491a96627: Neither blocks nor log segments found. Creating new log.
I20260809 12:09:51.038693 2017 tablet_bootstrap.cc:654] T 1e69bcaf3f22453b8e7f32bf6821701a P 8d1775c7b3c54f22bf644d4557c6d106: Neither blocks nor log segments found. Creating new log.
I20260809 12:09:51.040050 2016 log.cc:824] T 1e69bcaf3f22453b8e7f32bf6821701a P a7fa8439659a45ee8807f9f491a96627: Log is configured to *not* fsync() on all Append() calls
I20260809 12:09:51.041626 2017 log.cc:824] T 1e69bcaf3f22453b8e7f32bf6821701a P 8d1775c7b3c54f22bf644d4557c6d106: Log is configured to *not* fsync() on all Append() calls
I20260809 12:09:51.042186 2018 tablet_bootstrap.cc:654] T 1e69bcaf3f22453b8e7f32bf6821701a P d8e905ef1c864d1ab535707a4951a95b: Neither blocks nor log segments found. Creating new log.
I20260809 12:09:51.044116 2018 log.cc:824] T 1e69bcaf3f22453b8e7f32bf6821701a P d8e905ef1c864d1ab535707a4951a95b: Log is configured to *not* fsync() on all Append() calls
I20260809 12:09:51.059350 2016 tablet_bootstrap.cc:492] T 1e69bcaf3f22453b8e7f32bf6821701a P a7fa8439659a45ee8807f9f491a96627: No bootstrap required, opened a new log
I20260809 12:09:51.059906 2016 ts_tablet_manager.cc:1397] T 1e69bcaf3f22453b8e7f32bf6821701a P a7fa8439659a45ee8807f9f491a96627: Time spent bootstrapping tablet: real 0.032s user 0.013s sys 0.015s
I20260809 12:09:51.065445 2017 tablet_bootstrap.cc:492] T 1e69bcaf3f22453b8e7f32bf6821701a P 8d1775c7b3c54f22bf644d4557c6d106: No bootstrap required, opened a new log
I20260809 12:09:51.065888 2017 ts_tablet_manager.cc:1397] T 1e69bcaf3f22453b8e7f32bf6821701a P 8d1775c7b3c54f22bf644d4557c6d106: Time spent bootstrapping tablet: real 0.034s user 0.011s sys 0.016s
I20260809 12:09:51.068189 2018 tablet_bootstrap.cc:492] T 1e69bcaf3f22453b8e7f32bf6821701a P d8e905ef1c864d1ab535707a4951a95b: No bootstrap required, opened a new log
I20260809 12:09:51.068625 2018 ts_tablet_manager.cc:1397] T 1e69bcaf3f22453b8e7f32bf6821701a P d8e905ef1c864d1ab535707a4951a95b: Time spent bootstrapping tablet: real 0.034s user 0.008s sys 0.008s
I20260809 12:09:51.078665 2016 raft_consensus.cc:348] T 1e69bcaf3f22453b8e7f32bf6821701a P a7fa8439659a45ee8807f9f491a96627 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8d1775c7b3c54f22bf644d4557c6d106" member_type: VOTER last_known_addr { host: "127.0.222.2" port: 44865 } } peers { permanent_uuid: "d8e905ef1c864d1ab535707a4951a95b" member_type: VOTER last_known_addr { host: "127.0.222.1" port: 43353 } } peers { permanent_uuid: "a7fa8439659a45ee8807f9f491a96627" member_type: VOTER last_known_addr { host: "127.0.222.3" port: 35771 } }
I20260809 12:09:51.079484 2016 raft_consensus.cc:374] T 1e69bcaf3f22453b8e7f32bf6821701a P a7fa8439659a45ee8807f9f491a96627 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260809 12:09:51.079744 2016 raft_consensus.cc:729] T 1e69bcaf3f22453b8e7f32bf6821701a P a7fa8439659a45ee8807f9f491a96627 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a7fa8439659a45ee8807f9f491a96627, State: Initialized, Role: FOLLOWER
I20260809 12:09:51.080760 2016 consensus_queue.cc:260] T 1e69bcaf3f22453b8e7f32bf6821701a P a7fa8439659a45ee8807f9f491a96627 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8d1775c7b3c54f22bf644d4557c6d106" member_type: VOTER last_known_addr { host: "127.0.222.2" port: 44865 } } peers { permanent_uuid: "d8e905ef1c864d1ab535707a4951a95b" member_type: VOTER last_known_addr { host: "127.0.222.1" port: 43353 } } peers { permanent_uuid: "a7fa8439659a45ee8807f9f491a96627" member_type: VOTER last_known_addr { host: "127.0.222.3" port: 35771 } }
I20260809 12:09:51.083192 1997 heartbeater.cc:503] Master 127.0.222.62:34593 was elected leader, sending a full tablet report...
I20260809 12:09:51.082782 2018 raft_consensus.cc:348] T 1e69bcaf3f22453b8e7f32bf6821701a P d8e905ef1c864d1ab535707a4951a95b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8d1775c7b3c54f22bf644d4557c6d106" member_type: VOTER last_known_addr { host: "127.0.222.2" port: 44865 } } peers { permanent_uuid: "d8e905ef1c864d1ab535707a4951a95b" member_type: VOTER last_known_addr { host: "127.0.222.1" port: 43353 } } peers { permanent_uuid: "a7fa8439659a45ee8807f9f491a96627" member_type: VOTER last_known_addr { host: "127.0.222.3" port: 35771 } }
I20260809 12:09:51.083632 2018 raft_consensus.cc:374] T 1e69bcaf3f22453b8e7f32bf6821701a P d8e905ef1c864d1ab535707a4951a95b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260809 12:09:51.083809 2016 ts_tablet_manager.cc:1428] T 1e69bcaf3f22453b8e7f32bf6821701a P a7fa8439659a45ee8807f9f491a96627: Time spent starting tablet: real 0.024s user 0.014s sys 0.008s
I20260809 12:09:51.083838 2018 raft_consensus.cc:729] T 1e69bcaf3f22453b8e7f32bf6821701a P d8e905ef1c864d1ab535707a4951a95b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d8e905ef1c864d1ab535707a4951a95b, State: Initialized, Role: FOLLOWER
I20260809 12:09:51.084475 2017 raft_consensus.cc:348] T 1e69bcaf3f22453b8e7f32bf6821701a P 8d1775c7b3c54f22bf644d4557c6d106 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8d1775c7b3c54f22bf644d4557c6d106" member_type: VOTER last_known_addr { host: "127.0.222.2" port: 44865 } } peers { permanent_uuid: "d8e905ef1c864d1ab535707a4951a95b" member_type: VOTER last_known_addr { host: "127.0.222.1" port: 43353 } } peers { permanent_uuid: "a7fa8439659a45ee8807f9f491a96627" member_type: VOTER last_known_addr { host: "127.0.222.3" port: 35771 } }
I20260809 12:09:51.084725 2018 consensus_queue.cc:260] T 1e69bcaf3f22453b8e7f32bf6821701a P d8e905ef1c864d1ab535707a4951a95b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8d1775c7b3c54f22bf644d4557c6d106" member_type: VOTER last_known_addr { host: "127.0.222.2" port: 44865 } } peers { permanent_uuid: "d8e905ef1c864d1ab535707a4951a95b" member_type: VOTER last_known_addr { host: "127.0.222.1" port: 43353 } } peers { permanent_uuid: "a7fa8439659a45ee8807f9f491a96627" member_type: VOTER last_known_addr { host: "127.0.222.3" port: 35771 } }
I20260809 12:09:51.085223 2017 raft_consensus.cc:374] T 1e69bcaf3f22453b8e7f32bf6821701a P 8d1775c7b3c54f22bf644d4557c6d106 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260809 12:09:51.085424 2017 raft_consensus.cc:729] T 1e69bcaf3f22453b8e7f32bf6821701a P 8d1775c7b3c54f22bf644d4557c6d106 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8d1775c7b3c54f22bf644d4557c6d106, State: Initialized, Role: FOLLOWER
I20260809 12:09:51.086354 2017 consensus_queue.cc:260] T 1e69bcaf3f22453b8e7f32bf6821701a P 8d1775c7b3c54f22bf644d4557c6d106 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8d1775c7b3c54f22bf644d4557c6d106" member_type: VOTER last_known_addr { host: "127.0.222.2" port: 44865 } } peers { permanent_uuid: "d8e905ef1c864d1ab535707a4951a95b" member_type: VOTER last_known_addr { host: "127.0.222.1" port: 43353 } } peers { permanent_uuid: "a7fa8439659a45ee8807f9f491a96627" member_type: VOTER last_known_addr { host: "127.0.222.3" port: 35771 } }
I20260809 12:09:51.088707 1852 heartbeater.cc:503] Master 127.0.222.62:34593 was elected leader, sending a full tablet report...
I20260809 12:09:51.089768 2017 ts_tablet_manager.cc:1428] T 1e69bcaf3f22453b8e7f32bf6821701a P 8d1775c7b3c54f22bf644d4557c6d106: Time spent starting tablet: real 0.024s user 0.013s sys 0.011s
I20260809 12:09:51.092121 2018 ts_tablet_manager.cc:1428] T 1e69bcaf3f22453b8e7f32bf6821701a P d8e905ef1c864d1ab535707a4951a95b: Time spent starting tablet: real 0.023s user 0.020s sys 0.000s
W20260809 12:09:51.098268 1853 tablet.cc:2369] T 1e69bcaf3f22453b8e7f32bf6821701a P 8d1775c7b3c54f22bf644d4557c6d106: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260809 12:09:51.139282 1998 tablet.cc:2369] T 1e69bcaf3f22453b8e7f32bf6821701a P a7fa8439659a45ee8807f9f491a96627: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260809 12:09:51.335497 1715 tablet.cc:2369] T 1e69bcaf3f22453b8e7f32bf6821701a P d8e905ef1c864d1ab535707a4951a95b: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260809 12:09:51.375375 2024 raft_consensus.cc:482] T 1e69bcaf3f22453b8e7f32bf6821701a P 8d1775c7b3c54f22bf644d4557c6d106 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260809 12:09:51.375890 2024 raft_consensus.cc:504] T 1e69bcaf3f22453b8e7f32bf6821701a P 8d1775c7b3c54f22bf644d4557c6d106 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8d1775c7b3c54f22bf644d4557c6d106" member_type: VOTER last_known_addr { host: "127.0.222.2" port: 44865 } } peers { permanent_uuid: "d8e905ef1c864d1ab535707a4951a95b" member_type: VOTER last_known_addr { host: "127.0.222.1" port: 43353 } } peers { permanent_uuid: "a7fa8439659a45ee8807f9f491a96627" member_type: VOTER last_known_addr { host: "127.0.222.3" port: 35771 } }
I20260809 12:09:51.378589 2024 leader_election.cc:288] T 1e69bcaf3f22453b8e7f32bf6821701a P 8d1775c7b3c54f22bf644d4557c6d106 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers d8e905ef1c864d1ab535707a4951a95b (127.0.222.1:43353), a7fa8439659a45ee8807f9f491a96627 (127.0.222.3:35771)
I20260809 12:09:51.388543 2015 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 12:09:51.378993 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.222.2:57081 (local address 127.0.222.3:35771)
0809 12:09:51.379204 (+ 211us) server_negotiation.cc:206] Beginning negotiation
0809 12:09:51.379210 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0809 12:09:51.381002 (+ 1792us) server_negotiation.cc:407] Connection header received
0809 12:09:51.381254 (+ 252us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 12:09:51.381264 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 12:09:51.381341 (+ 77us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 12:09:51.381449 (+ 108us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0809 12:09:51.382405 (+ 956us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 12:09:51.383370 (+ 965us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 12:09:51.385925 (+ 2555us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 12:09:51.387267 (+ 1342us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 12:09:51.387421 (+ 154us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 12:09:51.387588 (+ 167us) server_negotiation.cc:1035] Waiting for connection context
0809 12:09:51.387981 (+ 393us) server_negotiation.cc:299] Negotiation successful
0809 12:09:51.388109 (+ 128us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":75}
I20260809 12:09:51.388973 2025 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 12:09:51.378810 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.222.3:35771 (local address 127.0.222.2:57081)
0809 12:09:51.380865 (+ 2055us) negotiation.cc:107] Waiting for socket to connect
0809 12:09:51.380881 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0809 12:09:51.381091 (+ 210us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 12:09:51.381475 (+ 384us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 12:09:51.381480 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 12:09:51.381552 (+ 72us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0809 12:09:51.382227 (+ 675us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 12:09:51.382238 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 12:09:51.383569 (+ 1331us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 12:09:51.383573 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 12:09:51.385757 (+ 2184us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 12:09:51.385769 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 12:09:51.387588 (+ 1819us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 12:09:51.387596 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 12:09:51.387813 (+ 217us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 12:09:51.387838 (+ 25us) client_negotiation.cc:769] Sending connection context
0809 12:09:51.387954 (+ 116us) client_negotiation.cc:240] Negotiation successful
0809 12:09:51.388090 (+ 136us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1879,"spinlock_wait_cycles":37248,"thread_start_us":1572,"threads_started":1}
I20260809 12:09:51.389444 2013 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 12:09:51.379103 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.222.2:50543 (local address 127.0.222.1:43353)
0809 12:09:51.379310 (+ 207us) server_negotiation.cc:206] Beginning negotiation
0809 12:09:51.379315 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0809 12:09:51.381849 (+ 2534us) server_negotiation.cc:407] Connection header received
0809 12:09:51.382106 (+ 257us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 12:09:51.382112 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 12:09:51.382184 (+ 72us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 12:09:51.382285 (+ 101us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0809 12:09:51.383338 (+ 1053us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 12:09:51.384361 (+ 1023us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 12:09:51.386867 (+ 2506us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 12:09:51.388044 (+ 1177us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 12:09:51.388216 (+ 172us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 12:09:51.388419 (+ 203us) server_negotiation.cc:1035] Waiting for connection context
0809 12:09:51.389001 (+ 582us) server_negotiation.cc:299] Negotiation successful
0809 12:09:51.389133 (+ 132us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":51}
I20260809 12:09:51.389915 2026 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 12:09:51.379624 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.222.1:43353 (local address 127.0.222.2:50543)
0809 12:09:51.381730 (+ 2106us) negotiation.cc:107] Waiting for socket to connect
0809 12:09:51.381745 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0809 12:09:51.381961 (+ 216us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 12:09:51.382312 (+ 351us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 12:09:51.382317 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 12:09:51.382340 (+ 23us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0809 12:09:51.383146 (+ 806us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 12:09:51.383158 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 12:09:51.384492 (+ 1334us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 12:09:51.384500 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 12:09:51.386662 (+ 2162us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 12:09:51.386675 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 12:09:51.388561 (+ 1886us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 12:09:51.388568 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 12:09:51.388837 (+ 269us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 12:09:51.388870 (+ 33us) client_negotiation.cc:769] Sending connection context
0809 12:09:51.388974 (+ 104us) client_negotiation.cc:240] Negotiation successful
0809 12:09:51.389258 (+ 284us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1903,"thread_start_us":1581,"threads_started":1}
I20260809 12:09:51.390496 1947 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1e69bcaf3f22453b8e7f32bf6821701a" candidate_uuid: "8d1775c7b3c54f22bf644d4557c6d106" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a7fa8439659a45ee8807f9f491a96627" is_pre_election: true
I20260809 12:09:51.391382 1947 raft_consensus.cc:2454] T 1e69bcaf3f22453b8e7f32bf6821701a P a7fa8439659a45ee8807f9f491a96627 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 8d1775c7b3c54f22bf644d4557c6d106 in term 0.
I20260809 12:09:51.391274 1668 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1e69bcaf3f22453b8e7f32bf6821701a" candidate_uuid: "8d1775c7b3c54f22bf644d4557c6d106" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d8e905ef1c864d1ab535707a4951a95b" is_pre_election: true
I20260809 12:09:51.392045 1668 raft_consensus.cc:2454] T 1e69bcaf3f22453b8e7f32bf6821701a P d8e905ef1c864d1ab535707a4951a95b [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 8d1775c7b3c54f22bf644d4557c6d106 in term 0.
I20260809 12:09:51.392338 1741 leader_election.cc:302] T 1e69bcaf3f22453b8e7f32bf6821701a P 8d1775c7b3c54f22bf644d4557c6d106 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 8d1775c7b3c54f22bf644d4557c6d106, a7fa8439659a45ee8807f9f491a96627; no voters:
I20260809 12:09:51.392941 2024 raft_consensus.cc:2781] T 1e69bcaf3f22453b8e7f32bf6821701a P 8d1775c7b3c54f22bf644d4557c6d106 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260809 12:09:51.393159 2024 raft_consensus.cc:482] T 1e69bcaf3f22453b8e7f32bf6821701a P 8d1775c7b3c54f22bf644d4557c6d106 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260809 12:09:51.393316 2024 raft_consensus.cc:3037] T 1e69bcaf3f22453b8e7f32bf6821701a P 8d1775c7b3c54f22bf644d4557c6d106 [term 0 FOLLOWER]: Advancing to term 1
I20260809 12:09:51.396495 2024 raft_consensus.cc:504] T 1e69bcaf3f22453b8e7f32bf6821701a P 8d1775c7b3c54f22bf644d4557c6d106 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8d1775c7b3c54f22bf644d4557c6d106" member_type: VOTER last_known_addr { host: "127.0.222.2" port: 44865 } } peers { permanent_uuid: "d8e905ef1c864d1ab535707a4951a95b" member_type: VOTER last_known_addr { host: "127.0.222.1" port: 43353 } } peers { permanent_uuid: "a7fa8439659a45ee8807f9f491a96627" member_type: VOTER last_known_addr { host: "127.0.222.3" port: 35771 } }
I20260809 12:09:51.397230 2024 leader_election.cc:288] T 1e69bcaf3f22453b8e7f32bf6821701a P 8d1775c7b3c54f22bf644d4557c6d106 [CANDIDATE]: Term 1 election: Requested vote from peers d8e905ef1c864d1ab535707a4951a95b (127.0.222.1:43353), a7fa8439659a45ee8807f9f491a96627 (127.0.222.3:35771)
I20260809 12:09:51.397784 1947 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1e69bcaf3f22453b8e7f32bf6821701a" candidate_uuid: "8d1775c7b3c54f22bf644d4557c6d106" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a7fa8439659a45ee8807f9f491a96627"
I20260809 12:09:51.397866 1668 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1e69bcaf3f22453b8e7f32bf6821701a" candidate_uuid: "8d1775c7b3c54f22bf644d4557c6d106" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d8e905ef1c864d1ab535707a4951a95b"
I20260809 12:09:51.398061 1947 raft_consensus.cc:3037] T 1e69bcaf3f22453b8e7f32bf6821701a P a7fa8439659a45ee8807f9f491a96627 [term 0 FOLLOWER]: Advancing to term 1
I20260809 12:09:51.398109 1668 raft_consensus.cc:3037] T 1e69bcaf3f22453b8e7f32bf6821701a P d8e905ef1c864d1ab535707a4951a95b [term 0 FOLLOWER]: Advancing to term 1
I20260809 12:09:51.400858 1947 raft_consensus.cc:2454] T 1e69bcaf3f22453b8e7f32bf6821701a P a7fa8439659a45ee8807f9f491a96627 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 8d1775c7b3c54f22bf644d4557c6d106 in term 1.
I20260809 12:09:51.401379 1741 leader_election.cc:302] T 1e69bcaf3f22453b8e7f32bf6821701a P 8d1775c7b3c54f22bf644d4557c6d106 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 8d1775c7b3c54f22bf644d4557c6d106, a7fa8439659a45ee8807f9f491a96627; no voters:
I20260809 12:09:51.401654 1668 raft_consensus.cc:2454] T 1e69bcaf3f22453b8e7f32bf6821701a P d8e905ef1c864d1ab535707a4951a95b [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 8d1775c7b3c54f22bf644d4557c6d106 in term 1.
I20260809 12:09:51.401769 2024 raft_consensus.cc:2781] T 1e69bcaf3f22453b8e7f32bf6821701a P 8d1775c7b3c54f22bf644d4557c6d106 [term 1 FOLLOWER]: Leader election won for term 1
I20260809 12:09:51.402648 2024 raft_consensus.cc:686] T 1e69bcaf3f22453b8e7f32bf6821701a P 8d1775c7b3c54f22bf644d4557c6d106 [term 1 LEADER]: Becoming Leader. State: Replica: 8d1775c7b3c54f22bf644d4557c6d106, State: Running, Role: LEADER
I20260809 12:09:51.403255 2024 consensus_queue.cc:237] T 1e69bcaf3f22453b8e7f32bf6821701a P 8d1775c7b3c54f22bf644d4557c6d106 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8d1775c7b3c54f22bf644d4557c6d106" member_type: VOTER last_known_addr { host: "127.0.222.2" port: 44865 } } peers { permanent_uuid: "d8e905ef1c864d1ab535707a4951a95b" member_type: VOTER last_known_addr { host: "127.0.222.1" port: 43353 } } peers { permanent_uuid: "a7fa8439659a45ee8807f9f491a96627" member_type: VOTER last_known_addr { host: "127.0.222.3" port: 35771 } }
I20260809 12:09:51.409864 1512 catalog_manager.cc:5495] T 1e69bcaf3f22453b8e7f32bf6821701a P 8d1775c7b3c54f22bf644d4557c6d106 reported cstate change: term changed from 0 to 1, leader changed from <none> to 8d1775c7b3c54f22bf644d4557c6d106 (127.0.222.2). New cstate: current_term: 1 leader_uuid: "8d1775c7b3c54f22bf644d4557c6d106" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8d1775c7b3c54f22bf644d4557c6d106" member_type: VOTER last_known_addr { host: "127.0.222.2" port: 44865 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "d8e905ef1c864d1ab535707a4951a95b" member_type: VOTER last_known_addr { host: "127.0.222.1" port: 43353 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "a7fa8439659a45ee8807f9f491a96627" member_type: VOTER last_known_addr { host: "127.0.222.3" port: 35771 } health_report { overall_health: UNKNOWN } } }
Aug 09 12:09:51 dist-test-slave-2w3w krb5kdc[1465](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786277388, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.222.1@KRBTEST.COM
I20260809 12:09:51.470636 2013 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 12:09:51.451367 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56866 (local address 127.0.222.1:43353)
0809 12:09:51.451602 (+ 235us) server_negotiation.cc:206] Beginning negotiation
0809 12:09:51.451609 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0809 12:09:51.451891 (+ 282us) server_negotiation.cc:407] Connection header received
0809 12:09:51.452072 (+ 181us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 12:09:51.452076 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 12:09:51.452159 (+ 83us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 12:09:51.452247 (+ 88us) server_negotiation.cc:226] Negotiated authn=SASL
0809 12:09:51.454428 (+ 2181us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 12:09:51.455493 (+ 1065us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 12:09:51.457267 (+ 1774us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 12:09:51.457918 (+ 651us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 12:09:51.458022 (+ 104us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 12:09:51.463547 (+ 5525us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 12:09:51.463565 (+ 18us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 12:09:51.463581 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0809 12:09:51.463616 (+ 35us) server_negotiation.cc:922] calling sasl_server_start()
0809 12:09:51.467523 (+ 3907us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 12:09:51.468286 (+ 763us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 12:09:51.468294 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 12:09:51.468296 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0809 12:09:51.468469 (+ 173us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 12:09:51.468919 (+ 450us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 12:09:51.468922 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 12:09:51.468924 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0809 12:09:51.469350 (+ 426us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 12:09:51.469637 (+ 287us) server_negotiation.cc:1035] Waiting for connection context
0809 12:09:51.469923 (+ 286us) server_negotiation.cc:299] Negotiation successful
0809 12:09:51.470120 (+ 197us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":83}
Aug 09 12:09:51 dist-test-slave-2w3w krb5kdc[1465](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786277391, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-user@KRBTEST.COM:
Aug 09 12:09:51 dist-test-slave-2w3w krb5kdc[1465](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786277391, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.0.222.1@KRBTEST.COM
I20260809 12:09:51.526929 2013 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 12:09:51.509625 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56876 (local address 127.0.222.1:43353)
0809 12:09:51.509882 (+ 257us) server_negotiation.cc:206] Beginning negotiation
0809 12:09:51.509889 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0809 12:09:51.510175 (+ 286us) server_negotiation.cc:407] Connection header received
0809 12:09:51.510322 (+ 147us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 12:09:51.510326 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 12:09:51.510392 (+ 66us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 12:09:51.510492 (+ 100us) server_negotiation.cc:226] Negotiated authn=SASL
0809 12:09:51.512762 (+ 2270us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 12:09:51.513552 (+ 790us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 12:09:51.515174 (+ 1622us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 12:09:51.515571 (+ 397us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 12:09:51.515664 (+ 93us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 12:09:51.520356 (+ 4692us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 12:09:51.520366 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 12:09:51.520372 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0809 12:09:51.520424 (+ 52us) server_negotiation.cc:922] calling sasl_server_start()
0809 12:09:51.524086 (+ 3662us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 12:09:51.524734 (+ 648us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 12:09:51.524737 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 12:09:51.524740 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0809 12:09:51.524963 (+ 223us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 12:09:51.525308 (+ 345us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 12:09:51.525311 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 12:09:51.525313 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0809 12:09:51.525681 (+ 368us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 12:09:51.525967 (+ 286us) server_negotiation.cc:1035] Waiting for connection context
0809 12:09:51.526257 (+ 290us) server_negotiation.cc:299] Negotiation successful
0809 12:09:51.526432 (+ 175us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":91}
Aug 09 12:09:51 dist-test-slave-2w3w krb5kdc[1465](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786277391, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
Aug 09 12:09:51 dist-test-slave-2w3w krb5kdc[1465](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786277391, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.222.1@KRBTEST.COM
I20260809 12:09:51.584968 2013 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 12:09:51.566069 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56884 (local address 127.0.222.1:43353)
0809 12:09:51.566351 (+ 282us) server_negotiation.cc:206] Beginning negotiation
0809 12:09:51.566396 (+ 45us) server_negotiation.cc:399] Waiting for connection header
0809 12:09:51.567002 (+ 606us) server_negotiation.cc:407] Connection header received
0809 12:09:51.567232 (+ 230us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 12:09:51.567237 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 12:09:51.567316 (+ 79us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 12:09:51.567456 (+ 140us) server_negotiation.cc:226] Negotiated authn=SASL
0809 12:09:51.569749 (+ 2293us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 12:09:51.570445 (+ 696us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 12:09:51.572160 (+ 1715us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 12:09:51.572715 (+ 555us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 12:09:51.572889 (+ 174us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 12:09:51.578555 (+ 5666us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 12:09:51.578560 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 12:09:51.578564 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0809 12:09:51.578600 (+ 36us) server_negotiation.cc:922] calling sasl_server_start()
0809 12:09:51.581500 (+ 2900us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 12:09:51.582314 (+ 814us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 12:09:51.582319 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 12:09:51.582321 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0809 12:09:51.582446 (+ 125us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 12:09:51.582890 (+ 444us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 12:09:51.582894 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 12:09:51.582896 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0809 12:09:51.583359 (+ 463us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 12:09:51.583765 (+ 406us) server_negotiation.cc:1035] Waiting for connection context
0809 12:09:51.584114 (+ 349us) server_negotiation.cc:299] Negotiation successful
0809 12:09:51.584438 (+ 324us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":62}
W20260809 12:09:51.587505 1626 tablet_service.cc:2908] Rejecting scan request for tablet 1e69bcaf3f22453b8e7f32bf6821701a: Uninitialized: clean time has not yet been initialized
I20260809 12:09:51.591053 888 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskgH5Pjm/build/asan/bin/kudu pid 1576
I20260809 12:09:51.606109 2013 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 12:09:51.592078 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56896 (local address 127.0.222.1:43353)
0809 12:09:51.592231 (+ 153us) server_negotiation.cc:206] Beginning negotiation
0809 12:09:51.592235 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0809 12:09:51.592624 (+ 389us) server_negotiation.cc:407] Connection header received
0809 12:09:51.592794 (+ 170us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 12:09:51.592797 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 12:09:51.592915 (+ 118us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 12:09:51.592997 (+ 82us) server_negotiation.cc:226] Negotiated authn=SASL
0809 12:09:51.594408 (+ 1411us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 12:09:51.595158 (+ 750us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 12:09:51.596775 (+ 1617us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 12:09:51.597223 (+ 448us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 12:09:51.597330 (+ 107us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 12:09:51.600989 (+ 3659us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 12:09:51.601010 (+ 21us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 12:09:51.601017 (+ 7us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0809 12:09:51.601057 (+ 40us) server_negotiation.cc:922] calling sasl_server_start()
0809 12:09:51.603549 (+ 2492us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 12:09:51.604198 (+ 649us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 12:09:51.604202 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 12:09:51.604205 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0809 12:09:51.604375 (+ 170us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 12:09:51.604733 (+ 358us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 12:09:51.604736 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 12:09:51.604738 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0809 12:09:51.604998 (+ 260us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 12:09:51.605232 (+ 234us) server_negotiation.cc:1035] Waiting for connection context
0809 12:09:51.605575 (+ 343us) server_negotiation.cc:299] Negotiation successful
0809 12:09:51.605733 (+ 158us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":51}
I20260809 12:09:51.606859 1709 generic_service.cc:196] Checking for leaks (request via RPC)
I20260809 12:09:51.697835 888 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskgH5Pjm/build/asan/bin/kudu with pid 1576
I20260809 12:09:51.718636 888 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskgH5Pjm/build/asan/bin/kudu pid 1720
Aug 09 12:09:51 dist-test-slave-2w3w krb5kdc[1465](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786277391, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.222.2@KRBTEST.COM
I20260809 12:09:51.733604 2042 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 12:09:51.719589 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42426 (local address 127.0.222.2:44865)
0809 12:09:51.720130 (+ 541us) server_negotiation.cc:206] Beginning negotiation
0809 12:09:51.720135 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0809 12:09:51.720152 (+ 17us) server_negotiation.cc:407] Connection header received
0809 12:09:51.720208 (+ 56us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 12:09:51.720212 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 12:09:51.720275 (+ 63us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 12:09:51.720365 (+ 90us) server_negotiation.cc:226] Negotiated authn=SASL
0809 12:09:51.721577 (+ 1212us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 12:09:51.722333 (+ 756us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 12:09:51.723673 (+ 1340us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 12:09:51.724100 (+ 427us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 12:09:51.724200 (+ 100us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 12:09:51.728188 (+ 3988us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 12:09:51.728202 (+ 14us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 12:09:51.728218 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0809 12:09:51.728258 (+ 40us) server_negotiation.cc:922] calling sasl_server_start()
0809 12:09:51.731196 (+ 2938us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 12:09:51.731815 (+ 619us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 12:09:51.731823 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 12:09:51.731825 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0809 12:09:51.731971 (+ 146us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 12:09:51.732290 (+ 319us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 12:09:51.732294 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 12:09:51.732296 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0809 12:09:51.732612 (+ 316us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 12:09:51.732863 (+ 251us) server_negotiation.cc:1035] Waiting for connection context
0809 12:09:51.733057 (+ 194us) server_negotiation.cc:299] Negotiation successful
0809 12:09:51.733226 (+ 169us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":449,"thread_start_us":314,"threads_started":1}
I20260809 12:09:51.734259 1847 generic_service.cc:196] Checking for leaks (request via RPC)
I20260809 12:09:51.826140 888 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskgH5Pjm/build/asan/bin/kudu with pid 1720
I20260809 12:09:51.843523 888 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskgH5Pjm/build/asan/bin/kudu pid 1858
Aug 09 12:09:51 dist-test-slave-2w3w krb5kdc[1465](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786277391, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.222.3@KRBTEST.COM
I20260809 12:09:51.858089 2015 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 12:09:51.844340 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45336 (local address 127.0.222.3:35771)
0809 12:09:51.844542 (+ 202us) server_negotiation.cc:206] Beginning negotiation
0809 12:09:51.844546 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0809 12:09:51.844583 (+ 37us) server_negotiation.cc:407] Connection header received
0809 12:09:51.844641 (+ 58us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 12:09:51.844645 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 12:09:51.844707 (+ 62us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 12:09:51.844799 (+ 92us) server_negotiation.cc:226] Negotiated authn=SASL
0809 12:09:51.846025 (+ 1226us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 12:09:51.846712 (+ 687us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 12:09:51.848040 (+ 1328us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 12:09:51.848411 (+ 371us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 12:09:51.848507 (+ 96us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 12:09:51.852609 (+ 4102us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 12:09:51.852622 (+ 13us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 12:09:51.852637 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0809 12:09:51.852674 (+ 37us) server_negotiation.cc:922] calling sasl_server_start()
0809 12:09:51.855553 (+ 2879us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 12:09:51.856266 (+ 713us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 12:09:51.856273 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 12:09:51.856276 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0809 12:09:51.856405 (+ 129us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 12:09:51.856737 (+ 332us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 12:09:51.856740 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 12:09:51.856742 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0809 12:09:51.857088 (+ 346us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 12:09:51.857304 (+ 216us) server_negotiation.cc:1035] Waiting for connection context
0809 12:09:51.857532 (+ 228us) server_negotiation.cc:299] Negotiation successful
0809 12:09:51.857691 (+ 159us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":88}
I20260809 12:09:51.858740 1991 generic_service.cc:196] Checking for leaks (request via RPC)
I20260809 12:09:51.947667 888 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskgH5Pjm/build/asan/bin/kudu with pid 1858
I20260809 12:09:51.967417 888 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskgH5Pjm/build/asan/bin/kudu pid 1480
I20260809 12:09:51.968638 1545 generic_service.cc:196] Checking for leaks (request via RPC)
I20260809 12:09:52.088201 888 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskgH5Pjm/build/asan/bin/kudu with pid 1480
2026-08-09T12:09:52Z chronyd exiting
[ OK ] SecurityITest.TestAuthorizationOnChecksum (4659 ms)
[ RUN ] SecurityITest.SmokeTestAsAuthorizedUser
Loading random data
Initializing database '/tmp/dist-test-taskgH5Pjm/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786277382073938-888-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 09 12:09:52 dist-test-slave-2w3w krb5kdc[2051](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 09 12:09:52 dist-test-slave-2w3w krb5kdc[2051](info): set up 2 sockets
Aug 09 12:09:52 dist-test-slave-2w3w krb5kdc[2051](info): commencing operation
krb5kdc: starting...
W20260809 12:09:53.232606 888 mini_kdc.cc:121] Time spent starting KDC: real 1.097s user 0.002s sys 0.018s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskgH5Pjm/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786277382073938-888-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskgH5Pjm/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786277382073938-888-0/krb5kdc/test-user.keytab.
Aug 09 12:09:53 dist-test-slave-2w3w krb5kdc[2051](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786277393, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-09T12:09:53Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-09T12:09:53Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.222.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.222.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.222.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskgH5Pjm/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786277382073938-888-0/krb5kdc/kudu_127.0.222.62.keytab.
Entry for principal kudu/127.0.222.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskgH5Pjm/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786277382073938-888-0/krb5kdc/kudu_127.0.222.62.keytab.
I20260809 12:09:53.446303 888 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskgH5Pjm/build/asan/bin/kudu
/tmp/dist-test-taskgH5Pjm/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskgH5Pjm/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786277382073938-888-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskgH5Pjm/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786277382073938-888-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskgH5Pjm/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786277382073938-888-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskgH5Pjm/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786277382073938-888-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.222.62:42617
--webserver_interface=127.0.222.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskgH5Pjm/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786277382073938-888-0/krb5kdc/kudu_127.0.222.62.keytab
--principal=kudu/127.0.222.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.222.20:38989
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.222.62:42617
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--txn_manager_enabled=true with env {KRB5CCNAME=/tmp/dist-test-taskgH5Pjm/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786277382073938-888-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskgH5Pjm/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786277382073938-888-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskgH5Pjm/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786277382073938-888-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260809 12:09:53.771204 2065 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260809 12:09:53.771489 2065 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260809 12:09:53.771674 2065 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260809 12:09:53.779086 2065 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260809 12:09:53.779182 2065 flags.cc:425] Enabled experimental flag: --txn_manager_enabled=true
W20260809 12:09:53.779278 2065 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260809 12:09:53.779388 2065 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260809 12:09:53.779457 2065 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260809 12:09:53.779574 2065 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260809 12:09:53.789971 2065 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.222.20:38989
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskgH5Pjm/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786277382073938-888-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskgH5Pjm/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786277382073938-888-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.222.62:42617
--txn_manager_enabled=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskgH5Pjm/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786277382073938-888-0/krb5kdc/kudu_127.0.222.62.keytab
--principal=kudu/127.0.222.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.222.62:42617
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskgH5Pjm/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786277382073938-888-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.222.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskgH5Pjm/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786277382073938-888-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 09 Aug 2026 11:06:41 UTC on ccycloud.quasar-bwoogy.root.comops.site
build id 4575991
ASAN enabled
I20260809 12:09:53.791390 2065 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260809 12:09:53.792793 2065 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260809 12:09:53.806123 2070 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 12:09:53.806115 2071 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 12:09:53.806052 2073 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260809 12:09:53.806999 2065 server_base.cc:1034] running on GCE node
I20260809 12:09:53.807761 2065 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260809 12:09:53.809084 2065 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260809 12:09:53.810334 2065 hybrid_clock.cc:648] HybridClock initialized: now 1786277393810251 us; error 30 us; skew 500 ppm
I20260809 12:09:53.810698 2065 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 09 12:09:53 dist-test-slave-2w3w krb5kdc[2051](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786277393, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.222.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260809 12:09:53.818742 2065 init.cc:377] Logged in from keytab as kudu/127.0.222.62@KRBTEST.COM (short username kudu)
I20260809 12:09:53.820432 2065 webserver.cc:460] Webserver started at http://127.0.222.62:38555/ using document root <none> and password file <none>
I20260809 12:09:53.821197 2065 fs_manager.cc:359] Metadata directory not provided
I20260809 12:09:53.821342 2065 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260809 12:09:53.821698 2065 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260809 12:09:53.825313 2065 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskgH5Pjm/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786277382073938-888-0/minicluster-data/master-0/data/instance:
uuid: "eb0ae6c68a9b460ebbcc6098cfb950cb"
format_stamp: "Formatted at 2026-08-09 12:09:53 on dist-test-slave-2w3w"
server_key: "d1187d32ccd57a4c569e729c6363c01d"
server_key_iv: "d3935763775298437dba33caf78d3069"
server_key_version: "encryptionkey@0"
I20260809 12:09:53.826236 2065 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskgH5Pjm/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786277382073938-888-0/minicluster-data/master-0/wal/instance:
uuid: "eb0ae6c68a9b460ebbcc6098cfb950cb"
format_stamp: "Formatted at 2026-08-09 12:09:53 on dist-test-slave-2w3w"
server_key: "d1187d32ccd57a4c569e729c6363c01d"
server_key_iv: "d3935763775298437dba33caf78d3069"
server_key_version: "encryptionkey@0"
I20260809 12:09:53.831966 2065 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.007s sys 0.001s
I20260809 12:09:53.835919 2080 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260809 12:09:53.837383 2065 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260809 12:09:53.837600 2065 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskgH5Pjm/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786277382073938-888-0/minicluster-data/master-0/data,/tmp/dist-test-taskgH5Pjm/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786277382073938-888-0/minicluster-data/master-0/wal
uuid: "eb0ae6c68a9b460ebbcc6098cfb950cb"
format_stamp: "Formatted at 2026-08-09 12:09:53 on dist-test-slave-2w3w"
server_key: "d1187d32ccd57a4c569e729c6363c01d"
server_key_iv: "d3935763775298437dba33caf78d3069"
server_key_version: "encryptionkey@0"
I20260809 12:09:53.837873 2065 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskgH5Pjm/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786277382073938-888-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskgH5Pjm/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786277382073938-888-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskgH5Pjm/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786277382073938-888-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260809 12:09:53.861630 2065 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260809 12:09:53.866133 2065 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260809 12:09:53.866456 2065 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260809 12:09:53.889285 2065 rpc_server.cc:307] RPC server started. Bound to: 127.0.222.62:42617
I20260809 12:09:53.889297 2143 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.222.62:42617 every 8 connection(s)
I20260809 12:09:53.890677 2065 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskgH5Pjm/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786277382073938-888-0/minicluster-data/master-0/data/info.pb
I20260809 12:09:53.894404 2144 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260809 12:09:53.895681 888 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskgH5Pjm/build/asan/bin/kudu as pid 2065
I20260809 12:09:53.895941 888 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskgH5Pjm/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786277382073938-888-0/minicluster-data/master-0/wal/instance
I20260809 12:09:53.896780 888 external_mini_cluster.cc:1428] Setting key fb325718e6ff50667cb458b64949ea37
I20260809 12:09:53.909281 2144 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P eb0ae6c68a9b460ebbcc6098cfb950cb: Bootstrap starting.
Aug 09 12:09:53 dist-test-slave-2w3w krb5kdc[2051](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786277393, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.222.62@KRBTEST.COM
I20260809 12:09:53.913290 2144 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P eb0ae6c68a9b460ebbcc6098cfb950cb: Neither blocks nor log segments found. Creating new log.
I20260809 12:09:53.914388 2144 log.cc:824] T 00000000000000000000000000000000 P eb0ae6c68a9b460ebbcc6098cfb950cb: Log is configured to *not* fsync() on all Append() calls
I20260809 12:09:53.917346 2144 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P eb0ae6c68a9b460ebbcc6098cfb950cb: No bootstrap required, opened a new log
I20260809 12:09:53.920840 2147 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 12:09:53.899045 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49864 (local address 127.0.222.62:42617)
0809 12:09:53.899969 (+ 924us) server_negotiation.cc:206] Beginning negotiation
0809 12:09:53.899980 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0809 12:09:53.900021 (+ 41us) server_negotiation.cc:407] Connection header received
0809 12:09:53.901632 (+ 1611us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 12:09:53.901657 (+ 25us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 12:09:53.902032 (+ 375us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 12:09:53.902370 (+ 338us) server_negotiation.cc:226] Negotiated authn=SASL
0809 12:09:53.904226 (+ 1856us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 12:09:53.905595 (+ 1369us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 12:09:53.907492 (+ 1897us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 12:09:53.908177 (+ 685us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 12:09:53.908322 (+ 145us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 12:09:53.913311 (+ 4989us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 12:09:53.913320 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 12:09:53.913334 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0809 12:09:53.913406 (+ 72us) server_negotiation.cc:922] calling sasl_server_start()
0809 12:09:53.916649 (+ 3243us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 12:09:53.917381 (+ 732us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 12:09:53.917387 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 12:09:53.917414 (+ 27us) server_negotiation.cc:967] calling sasl_server_step()
0809 12:09:53.917608 (+ 194us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 12:09:53.918018 (+ 410us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 12:09:53.918021 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 12:09:53.918023 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0809 12:09:53.918541 (+ 518us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 12:09:53.918884 (+ 343us) server_negotiation.cc:1035] Waiting for connection context
0809 12:09:53.919407 (+ 523us) server_negotiation.cc:299] Negotiation successful
0809 12:09:53.919838 (+ 431us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":708,"thread_start_us":450,"threads_started":1}
I20260809 12:09:53.925448 2144 raft_consensus.cc:348] T 00000000000000000000000000000000 P eb0ae6c68a9b460ebbcc6098cfb950cb [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "eb0ae6c68a9b460ebbcc6098cfb950cb" member_type: VOTER last_known_addr { host: "127.0.222.62" port: 42617 } }
I20260809 12:09:53.925827 2144 raft_consensus.cc:374] T 00000000000000000000000000000000 P eb0ae6c68a9b460ebbcc6098cfb950cb [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260809 12:09:53.925938 2144 raft_consensus.cc:729] T 00000000000000000000000000000000 P eb0ae6c68a9b460ebbcc6098cfb950cb [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: eb0ae6c68a9b460ebbcc6098cfb950cb, State: Initialized, Role: FOLLOWER
I20260809 12:09:53.926584 2144 consensus_queue.cc:260] T 00000000000000000000000000000000 P eb0ae6c68a9b460ebbcc6098cfb950cb [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "eb0ae6c68a9b460ebbcc6098cfb950cb" member_type: VOTER last_known_addr { host: "127.0.222.62" port: 42617 } }
I20260809 12:09:53.926829 2144 raft_consensus.cc:388] T 00000000000000000000000000000000 P eb0ae6c68a9b460ebbcc6098cfb950cb [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260809 12:09:53.927016 2144 raft_consensus.cc:482] T 00000000000000000000000000000000 P eb0ae6c68a9b460ebbcc6098cfb950cb [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260809 12:09:53.927203 2144 raft_consensus.cc:3037] T 00000000000000000000000000000000 P eb0ae6c68a9b460ebbcc6098cfb950cb [term 0 FOLLOWER]: Advancing to term 1
I20260809 12:09:53.929454 2144 raft_consensus.cc:504] T 00000000000000000000000000000000 P eb0ae6c68a9b460ebbcc6098cfb950cb [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "eb0ae6c68a9b460ebbcc6098cfb950cb" member_type: VOTER last_known_addr { host: "127.0.222.62" port: 42617 } }
I20260809 12:09:53.929996 2144 leader_election.cc:302] T 00000000000000000000000000000000 P eb0ae6c68a9b460ebbcc6098cfb950cb [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: eb0ae6c68a9b460ebbcc6098cfb950cb; no voters:
I20260809 12:09:53.930589 2144 leader_election.cc:288] T 00000000000000000000000000000000 P eb0ae6c68a9b460ebbcc6098cfb950cb [CANDIDATE]: Term 1 election: Requested vote from peers
I20260809 12:09:53.930748 2149 raft_consensus.cc:2781] T 00000000000000000000000000000000 P eb0ae6c68a9b460ebbcc6098cfb950cb [term 1 FOLLOWER]: Leader election won for term 1
I20260809 12:09:53.931787 2149 raft_consensus.cc:686] T 00000000000000000000000000000000 P eb0ae6c68a9b460ebbcc6098cfb950cb [term 1 LEADER]: Becoming Leader. State: Replica: eb0ae6c68a9b460ebbcc6098cfb950cb, State: Running, Role: LEADER
I20260809 12:09:53.932477 2149 consensus_queue.cc:237] T 00000000000000000000000000000000 P eb0ae6c68a9b460ebbcc6098cfb950cb [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "eb0ae6c68a9b460ebbcc6098cfb950cb" member_type: VOTER last_known_addr { host: "127.0.222.62" port: 42617 } }
I20260809 12:09:53.932981 2144 sys_catalog.cc:564] T 00000000000000000000000000000000 P eb0ae6c68a9b460ebbcc6098cfb950cb [sys.catalog]: configured and running, proceeding with master startup.
I20260809 12:09:53.935412 2151 sys_catalog.cc:455] T 00000000000000000000000000000000 P eb0ae6c68a9b460ebbcc6098cfb950cb [sys.catalog]: SysCatalogTable state changed. Reason: New leader eb0ae6c68a9b460ebbcc6098cfb950cb. Latest consensus state: current_term: 1 leader_uuid: "eb0ae6c68a9b460ebbcc6098cfb950cb" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "eb0ae6c68a9b460ebbcc6098cfb950cb" member_type: VOTER last_known_addr { host: "127.0.222.62" port: 42617 } } }
I20260809 12:09:53.935547 2150 sys_catalog.cc:455] T 00000000000000000000000000000000 P eb0ae6c68a9b460ebbcc6098cfb950cb [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "eb0ae6c68a9b460ebbcc6098cfb950cb" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "eb0ae6c68a9b460ebbcc6098cfb950cb" member_type: VOTER last_known_addr { host: "127.0.222.62" port: 42617 } } }
I20260809 12:09:53.935783 2151 sys_catalog.cc:458] T 00000000000000000000000000000000 P eb0ae6c68a9b460ebbcc6098cfb950cb [sys.catalog]: This master's current role is: LEADER
I20260809 12:09:53.935873 2150 sys_catalog.cc:458] T 00000000000000000000000000000000 P eb0ae6c68a9b460ebbcc6098cfb950cb [sys.catalog]: This master's current role is: LEADER
I20260809 12:09:53.938490 2158 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260809 12:09:53.943887 2158 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260809 12:09:53.952888 2158 catalog_manager.cc:1347] Generated new cluster ID: 0db935dc6dde4d1aa4423979304d72f8
I20260809 12:09:53.953032 2158 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260809 12:09:53.976227 2158 catalog_manager.cc:1370] Generated new certificate authority record
I20260809 12:09:53.977754 2158 catalog_manager.cc:1504] Loading token signing keys...
I20260809 12:09:53.985042 2158 catalog_manager.cc:5868] T 00000000000000000000000000000000 P eb0ae6c68a9b460ebbcc6098cfb950cb: Generated new TSK 0
I20260809 12:09:53.986032 2158 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.0.222.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.222.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.222.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskgH5Pjm/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786277382073938-888-0/krb5kdc/kudu_127.0.222.1.keytab.
Entry for principal kudu/127.0.222.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskgH5Pjm/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786277382073938-888-0/krb5kdc/kudu_127.0.222.1.keytab.
I20260809 12:09:54.046641 888 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskgH5Pjm/build/asan/bin/kudu
/tmp/dist-test-taskgH5Pjm/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskgH5Pjm/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786277382073938-888-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskgH5Pjm/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786277382073938-888-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskgH5Pjm/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786277382073938-888-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskgH5Pjm/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786277382073938-888-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.222.1:0
--local_ip_for_outbound_sockets=127.0.222.1
--webserver_interface=127.0.222.1
--webserver_port=0
--tserver_master_addrs=127.0.222.62:42617
--keytab_file=/tmp/dist-test-taskgH5Pjm/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786277382073938-888-0/krb5kdc/kudu_127.0.222.1.keytab
--principal=kudu/127.0.222.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.222.20:38989
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true with env {KRB5CCNAME=/tmp/dist-test-taskgH5Pjm/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786277382073938-888-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskgH5Pjm/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786277382073938-888-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskgH5Pjm/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786277382073938-888-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260809 12:09:54.378118 2172 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260809 12:09:54.378398 2172 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260809 12:09:54.378500 2172 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260809 12:09:54.385731 2172 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260809 12:09:54.385846 2172 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260809 12:09:54.385934 2172 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260809 12:09:54.386022 2172 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.222.1
I20260809 12:09:54.395867 2172 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.222.20:38989
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskgH5Pjm/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786277382073938-888-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskgH5Pjm/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786277382073938-888-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskgH5Pjm/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786277382073938-888-0/krb5kdc/kudu_127.0.222.1.keytab
--principal=kudu/127.0.222.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.222.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskgH5Pjm/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786277382073938-888-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.222.1
--webserver_port=0
--enable_txn_system_client_init=true
--tserver_master_addrs=127.0.222.62:42617
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.222.1
--log_dir=/tmp/dist-test-taskgH5Pjm/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786277382073938-888-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 09 Aug 2026 11:06:41 UTC on ccycloud.quasar-bwoogy.root.comops.site
build id 4575991
ASAN enabled
I20260809 12:09:54.397161 2172 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260809 12:09:54.398468 2172 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260809 12:09:54.412217 2177 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 12:09:54.412287 2178 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 12:09:54.412293 2180 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260809 12:09:54.413399 2172 server_base.cc:1034] running on GCE node
I20260809 12:09:54.414058 2172 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260809 12:09:54.415113 2172 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260809 12:09:54.416356 2172 hybrid_clock.cc:648] HybridClock initialized: now 1786277394416273 us; error 64 us; skew 500 ppm
I20260809 12:09:54.416673 2172 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 09 12:09:54 dist-test-slave-2w3w krb5kdc[2051](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786277394, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.222.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260809 12:09:54.419858 2172 init.cc:377] Logged in from keytab as kudu/127.0.222.1@KRBTEST.COM (short username kudu)
I20260809 12:09:54.421368 2172 webserver.cc:460] Webserver started at http://127.0.222.1:42929/ using document root <none> and password file <none>
I20260809 12:09:54.422096 2172 fs_manager.cc:359] Metadata directory not provided
I20260809 12:09:54.422240 2172 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260809 12:09:54.422582 2172 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260809 12:09:54.425582 2172 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskgH5Pjm/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786277382073938-888-0/minicluster-data/ts-0/data/instance:
uuid: "36f277a9f15a47bd9843342a5fc170f8"
format_stamp: "Formatted at 2026-08-09 12:09:54 on dist-test-slave-2w3w"
server_key: "12991726be6ad2fce18e77482b6c1e50"
server_key_iv: "7dce779bf734124865fb06e368e15044"
server_key_version: "encryptionkey@0"
I20260809 12:09:54.426378 2172 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskgH5Pjm/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786277382073938-888-0/minicluster-data/ts-0/wal/instance:
uuid: "36f277a9f15a47bd9843342a5fc170f8"
format_stamp: "Formatted at 2026-08-09 12:09:54 on dist-test-slave-2w3w"
server_key: "12991726be6ad2fce18e77482b6c1e50"
server_key_iv: "7dce779bf734124865fb06e368e15044"
server_key_version: "encryptionkey@0"
I20260809 12:09:54.431653 2172 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.003s sys 0.005s
I20260809 12:09:54.435468 2188 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260809 12:09:54.436915 2172 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.001s
I20260809 12:09:54.437099 2172 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskgH5Pjm/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786277382073938-888-0/minicluster-data/ts-0/data,/tmp/dist-test-taskgH5Pjm/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786277382073938-888-0/minicluster-data/ts-0/wal
uuid: "36f277a9f15a47bd9843342a5fc170f8"
format_stamp: "Formatted at 2026-08-09 12:09:54 on dist-test-slave-2w3w"
server_key: "12991726be6ad2fce18e77482b6c1e50"
server_key_iv: "7dce779bf734124865fb06e368e15044"
server_key_version: "encryptionkey@0"
I20260809 12:09:54.437352 2172 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskgH5Pjm/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786277382073938-888-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskgH5Pjm/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786277382073938-888-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskgH5Pjm/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786277382073938-888-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260809 12:09:54.464236 2172 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260809 12:09:54.468554 2172 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260809 12:09:54.468843 2172 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260809 12:09:54.471807 2172 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260809 12:09:54.471920 2172 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260809 12:09:54.472079 2172 ts_tablet_manager.cc:610] Registered 0 tablets
I20260809 12:09:54.472158 2172 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Aug 09 12:09:54 dist-test-slave-2w3w krb5kdc[2051](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786277394, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.222.1@KRBTEST.COM for kudu/127.0.222.62@KRBTEST.COM
I20260809 12:09:54.495267 2198 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 12:09:54.472473 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.222.1:33197 (local address 127.0.222.62:42617)
0809 12:09:54.473179 (+ 706us) server_negotiation.cc:206] Beginning negotiation
0809 12:09:54.473186 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0809 12:09:54.474387 (+ 1201us) server_negotiation.cc:407] Connection header received
0809 12:09:54.476611 (+ 2224us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 12:09:54.476620 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 12:09:54.476719 (+ 99us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 12:09:54.476861 (+ 142us) server_negotiation.cc:226] Negotiated authn=SASL
0809 12:09:54.479225 (+ 2364us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 12:09:54.480344 (+ 1119us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 12:09:54.482818 (+ 2474us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 12:09:54.483451 (+ 633us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 12:09:54.483593 (+ 142us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 12:09:54.488592 (+ 4999us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 12:09:54.488597 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 12:09:54.488601 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0809 12:09:54.488635 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0809 12:09:54.491401 (+ 2766us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 12:09:54.492597 (+ 1196us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 12:09:54.492603 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 12:09:54.492606 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0809 12:09:54.492736 (+ 130us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 12:09:54.493444 (+ 708us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 12:09:54.493448 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 12:09:54.493452 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0809 12:09:54.493760 (+ 308us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 12:09:54.493924 (+ 164us) server_negotiation.cc:1035] Waiting for connection context
0809 12:09:54.494622 (+ 698us) server_negotiation.cc:299] Negotiation successful
0809 12:09:54.494873 (+ 251us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":580,"thread_start_us":462,"threads_started":1}
I20260809 12:09:54.496814 2199 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 12:09:54.473165 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.222.62:42617 (local address 127.0.222.1:33197)
0809 12:09:54.474189 (+ 1024us) negotiation.cc:107] Waiting for socket to connect
0809 12:09:54.474246 (+ 57us) client_negotiation.cc:174] Beginning negotiation
0809 12:09:54.476288 (+ 2042us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 12:09:54.477106 (+ 818us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 12:09:54.477113 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 12:09:54.477952 (+ 839us) client_negotiation.cc:189] Negotiated authn=SASL
0809 12:09:54.479011 (+ 1059us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 12:09:54.479042 (+ 31us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 12:09:54.480509 (+ 1467us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 12:09:54.480520 (+ 11us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 12:09:54.482660 (+ 2140us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 12:09:54.482672 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 12:09:54.483757 (+ 1085us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 12:09:54.483769 (+ 12us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 12:09:54.484113 (+ 344us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 12:09:54.485107 (+ 994us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0809 12:09:54.485140 (+ 33us) client_negotiation.cc:656] calling sasl_client_start()
0809 12:09:54.488358 (+ 3218us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0809 12:09:54.491569 (+ 3211us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0809 12:09:54.491577 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0809 12:09:54.491598 (+ 21us) client_negotiation.cc:760] calling sasl_client_step()
0809 12:09:54.492439 (+ 841us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0809 12:09:54.492858 (+ 419us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0809 12:09:54.492867 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0809 12:09:54.492873 (+ 6us) client_negotiation.cc:760] calling sasl_client_step()
0809 12:09:54.493202 (+ 329us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0809 12:09:54.493909 (+ 707us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0809 12:09:54.493917 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0809 12:09:54.494260 (+ 343us) client_negotiation.cc:769] Sending connection context
0809 12:09:54.494525 (+ 265us) client_negotiation.cc:240] Negotiation successful
0809 12:09:54.495118 (+ 593us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":720,"thread_start_us":433,"threads_started":1}
I20260809 12:09:54.518362 2198 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 12:09:54.503228 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.222.1:56573 (local address 127.0.222.62:42617)
0809 12:09:54.503442 (+ 214us) server_negotiation.cc:206] Beginning negotiation
0809 12:09:54.503451 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0809 12:09:54.504303 (+ 852us) server_negotiation.cc:407] Connection header received
0809 12:09:54.504578 (+ 275us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 12:09:54.504596 (+ 18us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 12:09:54.504685 (+ 89us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 12:09:54.504755 (+ 70us) server_negotiation.cc:226] Negotiated authn=SASL
0809 12:09:54.506495 (+ 1740us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 12:09:54.507517 (+ 1022us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 12:09:54.509564 (+ 2047us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 12:09:54.509973 (+ 409us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 12:09:54.510116 (+ 143us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 12:09:54.512471 (+ 2355us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 12:09:54.512477 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 12:09:54.512480 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0809 12:09:54.512510 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0809 12:09:54.515691 (+ 3181us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 12:09:54.516359 (+ 668us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 12:09:54.516365 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 12:09:54.516368 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0809 12:09:54.516542 (+ 174us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 12:09:54.516900 (+ 358us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 12:09:54.516905 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 12:09:54.516908 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0809 12:09:54.517198 (+ 290us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 12:09:54.517353 (+ 155us) server_negotiation.cc:1035] Waiting for connection context
0809 12:09:54.517763 (+ 410us) server_negotiation.cc:299] Negotiation successful
0809 12:09:54.517992 (+ 229us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":69}
I20260809 12:09:54.518467 2260 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 12:09:54.503301 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.222.62:42617 (local address 127.0.222.1:56573)
0809 12:09:54.504193 (+ 892us) negotiation.cc:107] Waiting for socket to connect
0809 12:09:54.504218 (+ 25us) client_negotiation.cc:174] Beginning negotiation
0809 12:09:54.504393 (+ 175us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 12:09:54.504939 (+ 546us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 12:09:54.504942 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 12:09:54.505658 (+ 716us) client_negotiation.cc:189] Negotiated authn=SASL
0809 12:09:54.506190 (+ 532us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 12:09:54.506202 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 12:09:54.507737 (+ 1535us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 12:09:54.507743 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 12:09:54.509386 (+ 1643us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 12:09:54.509399 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 12:09:54.510162 (+ 763us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 12:09:54.510169 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 12:09:54.510342 (+ 173us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 12:09:54.511076 (+ 734us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0809 12:09:54.511093 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0809 12:09:54.512200 (+ 1107us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0809 12:09:54.515818 (+ 3618us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0809 12:09:54.515826 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0809 12:09:54.515832 (+ 6us) client_negotiation.cc:760] calling sasl_client_step()
0809 12:09:54.516223 (+ 391us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0809 12:09:54.516645 (+ 422us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0809 12:09:54.516650 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0809 12:09:54.516654 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0809 12:09:54.516783 (+ 129us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0809 12:09:54.517326 (+ 543us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0809 12:09:54.517330 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0809 12:09:54.517513 (+ 183us) client_negotiation.cc:769] Sending connection context
0809 12:09:54.517652 (+ 139us) client_negotiation.cc:240] Negotiation successful
0809 12:09:54.517976 (+ 324us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":672,"thread_start_us":505,"threads_started":1}
I20260809 12:09:54.527797 2172 rpc_server.cc:307] RPC server started. Bound to: 127.0.222.1:41559
I20260809 12:09:54.527877 2317 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.222.1:41559 every 8 connection(s)
I20260809 12:09:54.529000 2172 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskgH5Pjm/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786277382073938-888-0/minicluster-data/ts-0/data/info.pb
I20260809 12:09:54.531056 2318 heartbeater.cc:347] Connected to a master server at 127.0.222.62:42617
I20260809 12:09:54.531272 2318 heartbeater.cc:464] Registering TS with master...
I20260809 12:09:54.531893 2318 heartbeater.cc:511] Master 127.0.222.62:42617 requested a full tablet report, sending...
I20260809 12:09:54.533914 2099 ts_manager.cc:194] Registered new tserver with Master: 36f277a9f15a47bd9843342a5fc170f8 (127.0.222.1:41559)
I20260809 12:09:54.535558 2099 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.222.1@KRBTEST.COM'} at 127.0.222.1:33197
I20260809 12:09:54.537147 888 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskgH5Pjm/build/asan/bin/kudu as pid 2172
I20260809 12:09:54.537361 888 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskgH5Pjm/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786277382073938-888-0/minicluster-data/ts-0/wal/instance
I20260809 12:09:54.538031 888 external_mini_cluster.cc:1428] Setting key 38b33d0c9440f8d6cba45d620146347a
WARNING: no policy specified for kudu/127.0.222.2@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.222.2@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.222.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskgH5Pjm/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786277382073938-888-0/krb5kdc/kudu_127.0.222.2.keytab.
Entry for principal kudu/127.0.222.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskgH5Pjm/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786277382073938-888-0/krb5kdc/kudu_127.0.222.2.keytab.
I20260809 12:09:54.583976 888 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskgH5Pjm/build/asan/bin/kudu
/tmp/dist-test-taskgH5Pjm/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskgH5Pjm/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786277382073938-888-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskgH5Pjm/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786277382073938-888-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskgH5Pjm/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786277382073938-888-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskgH5Pjm/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786277382073938-888-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.222.2:0
--local_ip_for_outbound_sockets=127.0.222.2
--webserver_interface=127.0.222.2
--webserver_port=0
--tserver_master_addrs=127.0.222.62:42617
--keytab_file=/tmp/dist-test-taskgH5Pjm/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786277382073938-888-0/krb5kdc/kudu_127.0.222.2.keytab
--principal=kudu/127.0.222.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.222.20:38989
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true with env {KRB5CCNAME=/tmp/dist-test-taskgH5Pjm/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786277382073938-888-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskgH5Pjm/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786277382073938-888-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskgH5Pjm/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786277382073938-888-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260809 12:09:54.918983 2322 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260809 12:09:54.919273 2322 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260809 12:09:54.919462 2322 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260809 12:09:54.927014 2322 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260809 12:09:54.927134 2322 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260809 12:09:54.927284 2322 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260809 12:09:54.927413 2322 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.222.2
I20260809 12:09:54.938309 2322 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.222.20:38989
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskgH5Pjm/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786277382073938-888-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskgH5Pjm/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786277382073938-888-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskgH5Pjm/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786277382073938-888-0/krb5kdc/kudu_127.0.222.2.keytab
--principal=kudu/127.0.222.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.222.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskgH5Pjm/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786277382073938-888-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.222.2
--webserver_port=0
--enable_txn_system_client_init=true
--tserver_master_addrs=127.0.222.62:42617
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.222.2
--log_dir=/tmp/dist-test-taskgH5Pjm/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786277382073938-888-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 09 Aug 2026 11:06:41 UTC on ccycloud.quasar-bwoogy.root.comops.site
build id 4575991
ASAN enabled
I20260809 12:09:54.939636 2322 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260809 12:09:54.941053 2322 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260809 12:09:54.954654 2328 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 12:09:54.954607 2327 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 12:09:54.954685 2331 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260809 12:09:54.955374 2322 server_base.cc:1034] running on GCE node
I20260809 12:09:54.956102 2322 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260809 12:09:54.957033 2322 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260809 12:09:54.958277 2322 hybrid_clock.cc:648] HybridClock initialized: now 1786277394958214 us; error 58 us; skew 500 ppm
I20260809 12:09:54.958575 2322 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 09 12:09:54 dist-test-slave-2w3w krb5kdc[2051](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786277394, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.222.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260809 12:09:54.961710 2322 init.cc:377] Logged in from keytab as kudu/127.0.222.2@KRBTEST.COM (short username kudu)
I20260809 12:09:54.963254 2322 webserver.cc:460] Webserver started at http://127.0.222.2:43423/ using document root <none> and password file <none>
I20260809 12:09:54.963944 2322 fs_manager.cc:359] Metadata directory not provided
I20260809 12:09:54.964082 2322 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260809 12:09:54.964411 2322 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260809 12:09:54.967389 2322 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskgH5Pjm/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786277382073938-888-0/minicluster-data/ts-1/data/instance:
uuid: "3842c96efca14e43898885e9664afe98"
format_stamp: "Formatted at 2026-08-09 12:09:54 on dist-test-slave-2w3w"
server_key: "9b2d5f67a7a93e3120efb304a3364f15"
server_key_iv: "3e870b179b7979b643f9f08b35859eda"
server_key_version: "encryptionkey@0"
I20260809 12:09:54.968202 2322 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskgH5Pjm/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786277382073938-888-0/minicluster-data/ts-1/wal/instance:
uuid: "3842c96efca14e43898885e9664afe98"
format_stamp: "Formatted at 2026-08-09 12:09:54 on dist-test-slave-2w3w"
server_key: "9b2d5f67a7a93e3120efb304a3364f15"
server_key_iv: "3e870b179b7979b643f9f08b35859eda"
server_key_version: "encryptionkey@0"
I20260809 12:09:54.973359 2322 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.000s
I20260809 12:09:54.976853 2339 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260809 12:09:54.978139 2322 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260809 12:09:54.978318 2322 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskgH5Pjm/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786277382073938-888-0/minicluster-data/ts-1/data,/tmp/dist-test-taskgH5Pjm/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786277382073938-888-0/minicluster-data/ts-1/wal
uuid: "3842c96efca14e43898885e9664afe98"
format_stamp: "Formatted at 2026-08-09 12:09:54 on dist-test-slave-2w3w"
server_key: "9b2d5f67a7a93e3120efb304a3364f15"
server_key_iv: "3e870b179b7979b643f9f08b35859eda"
server_key_version: "encryptionkey@0"
I20260809 12:09:54.978557 2322 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskgH5Pjm/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786277382073938-888-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskgH5Pjm/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786277382073938-888-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskgH5Pjm/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786277382073938-888-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260809 12:09:54.990844 2322 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260809 12:09:54.994899 2322 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260809 12:09:54.995206 2322 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260809 12:09:54.997905 2322 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260809 12:09:54.998024 2322 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260809 12:09:54.998188 2322 ts_tablet_manager.cc:610] Registered 0 tablets
I20260809 12:09:54.998260 2322 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Aug 09 12:09:55 dist-test-slave-2w3w krb5kdc[2051](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786277394, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.222.2@KRBTEST.COM for kudu/127.0.222.62@KRBTEST.COM
I20260809 12:09:55.023411 2198 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 12:09:54.998696 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.222.2:53897 (local address 127.0.222.62:42617)
0809 12:09:54.998892 (+ 196us) server_negotiation.cc:206] Beginning negotiation
0809 12:09:54.998898 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0809 12:09:55.000487 (+ 1589us) server_negotiation.cc:407] Connection header received
0809 12:09:55.002504 (+ 2017us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 12:09:55.002510 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 12:09:55.002604 (+ 94us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 12:09:55.002724 (+ 120us) server_negotiation.cc:226] Negotiated authn=SASL
0809 12:09:55.005194 (+ 2470us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 12:09:55.006289 (+ 1095us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 12:09:55.009294 (+ 3005us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 12:09:55.009782 (+ 488us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 12:09:55.009938 (+ 156us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 12:09:55.015571 (+ 5633us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 12:09:55.015579 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 12:09:55.015586 (+ 7us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0809 12:09:55.015637 (+ 51us) server_negotiation.cc:922] calling sasl_server_start()
0809 12:09:55.019352 (+ 3715us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 12:09:55.020343 (+ 991us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 12:09:55.020347 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 12:09:55.020349 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0809 12:09:55.020467 (+ 118us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 12:09:55.021107 (+ 640us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 12:09:55.021112 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 12:09:55.021116 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0809 12:09:55.021536 (+ 420us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 12:09:55.021704 (+ 168us) server_negotiation.cc:1035] Waiting for connection context
0809 12:09:55.022758 (+ 1054us) server_negotiation.cc:299] Negotiation successful
0809 12:09:55.023034 (+ 276us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":52}
I20260809 12:09:55.024973 2348 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 12:09:54.999369 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.222.62:42617 (local address 127.0.222.2:53897)
0809 12:09:55.000323 (+ 954us) negotiation.cc:107] Waiting for socket to connect
0809 12:09:55.000370 (+ 47us) client_negotiation.cc:174] Beginning negotiation
0809 12:09:55.002215 (+ 1845us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 12:09:55.003027 (+ 812us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 12:09:55.003039 (+ 12us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 12:09:55.003916 (+ 877us) client_negotiation.cc:189] Negotiated authn=SASL
0809 12:09:55.004900 (+ 984us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 12:09:55.004932 (+ 32us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 12:09:55.006466 (+ 1534us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 12:09:55.006473 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 12:09:55.009125 (+ 2652us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 12:09:55.009142 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 12:09:55.010129 (+ 987us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 12:09:55.010141 (+ 12us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 12:09:55.010553 (+ 412us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 12:09:55.011426 (+ 873us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0809 12:09:55.011469 (+ 43us) client_negotiation.cc:656] calling sasl_client_start()
0809 12:09:55.015325 (+ 3856us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0809 12:09:55.019495 (+ 4170us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0809 12:09:55.019505 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0809 12:09:55.019531 (+ 26us) client_negotiation.cc:760] calling sasl_client_step()
0809 12:09:55.020166 (+ 635us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0809 12:09:55.020641 (+ 475us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0809 12:09:55.020649 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0809 12:09:55.020655 (+ 6us) client_negotiation.cc:760] calling sasl_client_step()
0809 12:09:55.020967 (+ 312us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0809 12:09:55.021685 (+ 718us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0809 12:09:55.021694 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0809 12:09:55.022255 (+ 561us) client_negotiation.cc:769] Sending connection context
0809 12:09:55.022664 (+ 409us) client_negotiation.cc:240] Negotiation successful
0809 12:09:55.023285 (+ 621us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":677,"thread_start_us":370,"threads_started":1}
I20260809 12:09:55.048738 2435 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 12:09:55.032403 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.222.62:42617 (local address 127.0.222.2:35003)
0809 12:09:55.033277 (+ 874us) negotiation.cc:107] Waiting for socket to connect
0809 12:09:55.033293 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0809 12:09:55.033425 (+ 132us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 12:09:55.033939 (+ 514us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 12:09:55.033943 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 12:09:55.034458 (+ 515us) client_negotiation.cc:189] Negotiated authn=SASL
0809 12:09:55.034763 (+ 305us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 12:09:55.034796 (+ 33us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 12:09:55.036369 (+ 1573us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 12:09:55.036373 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 12:09:55.037708 (+ 1335us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 12:09:55.037727 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 12:09:55.038908 (+ 1181us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 12:09:55.038917 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 12:09:55.039167 (+ 250us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 12:09:55.039889 (+ 722us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0809 12:09:55.039906 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0809 12:09:55.040642 (+ 736us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0809 12:09:55.045118 (+ 4476us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0809 12:09:55.045126 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0809 12:09:55.045133 (+ 7us) client_negotiation.cc:760] calling sasl_client_step()
0809 12:09:55.045834 (+ 701us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0809 12:09:55.046315 (+ 481us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0809 12:09:55.046323 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0809 12:09:55.046329 (+ 6us) client_negotiation.cc:760] calling sasl_client_step()
0809 12:09:55.046537 (+ 208us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0809 12:09:55.047456 (+ 919us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0809 12:09:55.047465 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0809 12:09:55.047676 (+ 211us) client_negotiation.cc:769] Sending connection context
0809 12:09:55.047842 (+ 166us) client_negotiation.cc:240] Negotiation successful
0809 12:09:55.048207 (+ 365us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":717,"thread_start_us":506,"threads_started":1}
I20260809 12:09:55.049361 2198 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 12:09:55.032605 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.222.2:35003 (local address 127.0.222.62:42617)
0809 12:09:55.032871 (+ 266us) server_negotiation.cc:206] Beginning negotiation
0809 12:09:55.032879 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0809 12:09:55.033383 (+ 504us) server_negotiation.cc:407] Connection header received
0809 12:09:55.033559 (+ 176us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 12:09:55.033568 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 12:09:55.033678 (+ 110us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 12:09:55.033825 (+ 147us) server_negotiation.cc:226] Negotiated authn=SASL
0809 12:09:55.034949 (+ 1124us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 12:09:55.036227 (+ 1278us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 12:09:55.037902 (+ 1675us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 12:09:55.038684 (+ 782us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 12:09:55.038864 (+ 180us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 12:09:55.040882 (+ 2018us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 12:09:55.040892 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 12:09:55.040899 (+ 7us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0809 12:09:55.040953 (+ 54us) server_negotiation.cc:922] calling sasl_server_start()
0809 12:09:55.044966 (+ 4013us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 12:09:55.045982 (+ 1016us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 12:09:55.045989 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 12:09:55.045995 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0809 12:09:55.046181 (+ 186us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 12:09:55.046673 (+ 492us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 12:09:55.046681 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 12:09:55.046686 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0809 12:09:55.047203 (+ 517us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 12:09:55.047384 (+ 181us) server_negotiation.cc:1035] Waiting for connection context
0809 12:09:55.048143 (+ 759us) server_negotiation.cc:299] Negotiation successful
0809 12:09:55.048605 (+ 462us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":69}
I20260809 12:09:55.060016 2322 rpc_server.cc:307] RPC server started. Bound to: 127.0.222.2:34441
I20260809 12:09:55.060060 2463 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.222.2:34441 every 8 connection(s)
I20260809 12:09:55.061723 2322 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskgH5Pjm/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786277382073938-888-0/minicluster-data/ts-1/data/info.pb
I20260809 12:09:55.064167 2464 heartbeater.cc:347] Connected to a master server at 127.0.222.62:42617
I20260809 12:09:55.064421 2464 heartbeater.cc:464] Registering TS with master...
I20260809 12:09:55.065050 888 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskgH5Pjm/build/asan/bin/kudu as pid 2322
I20260809 12:09:55.065269 888 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskgH5Pjm/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786277382073938-888-0/minicluster-data/ts-1/wal/instance
I20260809 12:09:55.065286 2464 heartbeater.cc:511] Master 127.0.222.62:42617 requested a full tablet report, sending...
I20260809 12:09:55.065807 888 external_mini_cluster.cc:1428] Setting key b107754d8d83141b0ac5992e891c653f
I20260809 12:09:55.066874 2099 ts_manager.cc:194] Registered new tserver with Master: 3842c96efca14e43898885e9664afe98 (127.0.222.2:34441)
I20260809 12:09:55.067879 2099 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.222.2@KRBTEST.COM'} at 127.0.222.2:53897
WARNING: no policy specified for kudu/127.0.222.3@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.222.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.222.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskgH5Pjm/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786277382073938-888-0/krb5kdc/kudu_127.0.222.3.keytab.
Entry for principal kudu/127.0.222.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskgH5Pjm/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786277382073938-888-0/krb5kdc/kudu_127.0.222.3.keytab.
I20260809 12:09:55.110190 888 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskgH5Pjm/build/asan/bin/kudu
/tmp/dist-test-taskgH5Pjm/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskgH5Pjm/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786277382073938-888-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskgH5Pjm/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786277382073938-888-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskgH5Pjm/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786277382073938-888-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskgH5Pjm/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786277382073938-888-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.222.3:0
--local_ip_for_outbound_sockets=127.0.222.3
--webserver_interface=127.0.222.3
--webserver_port=0
--tserver_master_addrs=127.0.222.62:42617
--keytab_file=/tmp/dist-test-taskgH5Pjm/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786277382073938-888-0/krb5kdc/kudu_127.0.222.3.keytab
--principal=kudu/127.0.222.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.222.20:38989
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true with env {KRB5CCNAME=/tmp/dist-test-taskgH5Pjm/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786277382073938-888-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskgH5Pjm/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786277382073938-888-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskgH5Pjm/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786277382073938-888-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260809 12:09:55.447794 2468 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260809 12:09:55.448079 2468 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260809 12:09:55.448243 2468 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260809 12:09:55.455499 2468 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260809 12:09:55.455629 2468 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260809 12:09:55.455782 2468 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260809 12:09:55.455906 2468 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.222.3
I20260809 12:09:55.465993 2468 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.222.20:38989
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskgH5Pjm/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786277382073938-888-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskgH5Pjm/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786277382073938-888-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskgH5Pjm/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786277382073938-888-0/krb5kdc/kudu_127.0.222.3.keytab
--principal=kudu/127.0.222.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.222.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskgH5Pjm/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786277382073938-888-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.222.3
--webserver_port=0
--enable_txn_system_client_init=true
--tserver_master_addrs=127.0.222.62:42617
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.222.3
--log_dir=/tmp/dist-test-taskgH5Pjm/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786277382073938-888-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 09 Aug 2026 11:06:41 UTC on ccycloud.quasar-bwoogy.root.comops.site
build id 4575991
ASAN enabled
I20260809 12:09:55.467433 2468 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260809 12:09:55.468851 2468 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260809 12:09:55.482019 2477 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 12:09:55.482059 2474 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 12:09:55.482080 2473 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260809 12:09:55.483245 2468 server_base.cc:1034] running on GCE node
I20260809 12:09:55.483896 2468 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260809 12:09:55.484786 2468 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260809 12:09:55.486007 2468 hybrid_clock.cc:648] HybridClock initialized: now 1786277395485920 us; error 55 us; skew 500 ppm
I20260809 12:09:55.486264 2468 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 09 12:09:55 dist-test-slave-2w3w krb5kdc[2051](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786277395, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.222.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260809 12:09:55.489405 2468 init.cc:377] Logged in from keytab as kudu/127.0.222.3@KRBTEST.COM (short username kudu)
I20260809 12:09:55.490806 2468 webserver.cc:460] Webserver started at http://127.0.222.3:38759/ using document root <none> and password file <none>
I20260809 12:09:55.491511 2468 fs_manager.cc:359] Metadata directory not provided
I20260809 12:09:55.491602 2468 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260809 12:09:55.491875 2468 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260809 12:09:55.494649 2468 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskgH5Pjm/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786277382073938-888-0/minicluster-data/ts-2/data/instance:
uuid: "a390a8c24bbe45baa12c712e9cf3c90d"
format_stamp: "Formatted at 2026-08-09 12:09:55 on dist-test-slave-2w3w"
server_key: "a9deed1d29a3cdba55db4f236ec5fc9d"
server_key_iv: "831c79c40c46c1a1707e72cd545286f7"
server_key_version: "encryptionkey@0"
I20260809 12:09:55.495452 2468 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskgH5Pjm/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786277382073938-888-0/minicluster-data/ts-2/wal/instance:
uuid: "a390a8c24bbe45baa12c712e9cf3c90d"
format_stamp: "Formatted at 2026-08-09 12:09:55 on dist-test-slave-2w3w"
server_key: "a9deed1d29a3cdba55db4f236ec5fc9d"
server_key_iv: "831c79c40c46c1a1707e72cd545286f7"
server_key_version: "encryptionkey@0"
I20260809 12:09:55.500469 2468 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.004s sys 0.000s
I20260809 12:09:55.503777 2485 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260809 12:09:55.505023 2468 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260809 12:09:55.505182 2468 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskgH5Pjm/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786277382073938-888-0/minicluster-data/ts-2/data,/tmp/dist-test-taskgH5Pjm/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786277382073938-888-0/minicluster-data/ts-2/wal
uuid: "a390a8c24bbe45baa12c712e9cf3c90d"
format_stamp: "Formatted at 2026-08-09 12:09:55 on dist-test-slave-2w3w"
server_key: "a9deed1d29a3cdba55db4f236ec5fc9d"
server_key_iv: "831c79c40c46c1a1707e72cd545286f7"
server_key_version: "encryptionkey@0"
I20260809 12:09:55.505389 2468 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskgH5Pjm/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786277382073938-888-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskgH5Pjm/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786277382073938-888-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskgH5Pjm/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786277382073938-888-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260809 12:09:55.522330 2468 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260809 12:09:55.526681 2468 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260809 12:09:55.527061 2468 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260809 12:09:55.530134 2468 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260809 12:09:55.530242 2468 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260809 12:09:55.530424 2468 ts_tablet_manager.cc:610] Registered 0 tablets
I20260809 12:09:55.530512 2468 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260809 12:09:55.538388 2318 heartbeater.cc:503] Master 127.0.222.62:42617 was elected leader, sending a full tablet report...
Aug 09 12:09:55 dist-test-slave-2w3w krb5kdc[2051](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786277395, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.222.3@KRBTEST.COM for kudu/127.0.222.62@KRBTEST.COM
I20260809 12:09:55.558367 2198 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 12:09:55.531390 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.222.3:35455 (local address 127.0.222.62:42617)
0809 12:09:55.531633 (+ 243us) server_negotiation.cc:206] Beginning negotiation
0809 12:09:55.531639 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0809 12:09:55.532832 (+ 1193us) server_negotiation.cc:407] Connection header received
0809 12:09:55.536360 (+ 3528us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 12:09:55.536368 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 12:09:55.536471 (+ 103us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 12:09:55.536582 (+ 111us) server_negotiation.cc:226] Negotiated authn=SASL
0809 12:09:55.539341 (+ 2759us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 12:09:55.540460 (+ 1119us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 12:09:55.543870 (+ 3410us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 12:09:55.544686 (+ 816us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 12:09:55.544825 (+ 139us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 12:09:55.550601 (+ 5776us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 12:09:55.550611 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 12:09:55.550619 (+ 8us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0809 12:09:55.550677 (+ 58us) server_negotiation.cc:922] calling sasl_server_start()
0809 12:09:55.554604 (+ 3927us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 12:09:55.555822 (+ 1218us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 12:09:55.555827 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 12:09:55.555831 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0809 12:09:55.555987 (+ 156us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 12:09:55.556467 (+ 480us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 12:09:55.556472 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 12:09:55.556475 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0809 12:09:55.556772 (+ 297us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 12:09:55.556951 (+ 179us) server_negotiation.cc:1035] Waiting for connection context
0809 12:09:55.557723 (+ 772us) server_negotiation.cc:299] Negotiation successful
0809 12:09:55.557991 (+ 268us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":81}
I20260809 12:09:55.559662 2494 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 12:09:55.531669 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.222.62:42617 (local address 127.0.222.3:35455)
0809 12:09:55.532647 (+ 978us) negotiation.cc:107] Waiting for socket to connect
0809 12:09:55.532701 (+ 54us) client_negotiation.cc:174] Beginning negotiation
0809 12:09:55.535951 (+ 3250us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 12:09:55.536886 (+ 935us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 12:09:55.536902 (+ 16us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 12:09:55.538082 (+ 1180us) client_negotiation.cc:189] Negotiated authn=SASL
0809 12:09:55.539125 (+ 1043us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 12:09:55.539158 (+ 33us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 12:09:55.540644 (+ 1486us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 12:09:55.540653 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 12:09:55.543636 (+ 2983us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 12:09:55.543656 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 12:09:55.545260 (+ 1604us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 12:09:55.545276 (+ 16us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 12:09:55.545723 (+ 447us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 12:09:55.546711 (+ 988us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0809 12:09:55.546750 (+ 39us) client_negotiation.cc:656] calling sasl_client_start()
0809 12:09:55.550283 (+ 3533us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0809 12:09:55.554839 (+ 4556us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0809 12:09:55.554854 (+ 15us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0809 12:09:55.554877 (+ 23us) client_negotiation.cc:760] calling sasl_client_step()
0809 12:09:55.555670 (+ 793us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0809 12:09:55.556111 (+ 441us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0809 12:09:55.556116 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0809 12:09:55.556119 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0809 12:09:55.556330 (+ 211us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0809 12:09:55.556911 (+ 581us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0809 12:09:55.556920 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0809 12:09:55.557318 (+ 398us) client_negotiation.cc:769] Sending connection context
0809 12:09:55.557590 (+ 272us) client_negotiation.cc:240] Negotiation successful
0809 12:09:55.558149 (+ 559us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":662,"thread_start_us":396,"threads_started":1}
I20260809 12:09:55.583343 2198 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 12:09:55.569394 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.222.3:55953 (local address 127.0.222.62:42617)
0809 12:09:55.569569 (+ 175us) server_negotiation.cc:206] Beginning negotiation
0809 12:09:55.569573 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0809 12:09:55.570178 (+ 605us) server_negotiation.cc:407] Connection header received
0809 12:09:55.570380 (+ 202us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 12:09:55.570384 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 12:09:55.570455 (+ 71us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 12:09:55.570567 (+ 112us) server_negotiation.cc:226] Negotiated authn=SASL
0809 12:09:55.571955 (+ 1388us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 12:09:55.572859 (+ 904us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 12:09:55.574166 (+ 1307us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 12:09:55.574612 (+ 446us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 12:09:55.574769 (+ 157us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 12:09:55.576804 (+ 2035us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 12:09:55.576814 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 12:09:55.576818 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0809 12:09:55.576848 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0809 12:09:55.580339 (+ 3491us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 12:09:55.581212 (+ 873us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 12:09:55.581218 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 12:09:55.581222 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0809 12:09:55.581373 (+ 151us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 12:09:55.581747 (+ 374us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 12:09:55.581753 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 12:09:55.581756 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0809 12:09:55.582040 (+ 284us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 12:09:55.582197 (+ 157us) server_negotiation.cc:1035] Waiting for connection context
0809 12:09:55.582685 (+ 488us) server_negotiation.cc:299] Negotiation successful
0809 12:09:55.582925 (+ 240us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":54}
I20260809 12:09:55.583503 2553 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 12:09:55.569432 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.222.62:42617 (local address 127.0.222.3:55953)
0809 12:09:55.570074 (+ 642us) negotiation.cc:107] Waiting for socket to connect
0809 12:09:55.570093 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0809 12:09:55.570249 (+ 156us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 12:09:55.570594 (+ 345us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 12:09:55.570598 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 12:09:55.571112 (+ 514us) client_negotiation.cc:189] Negotiated authn=SASL
0809 12:09:55.571721 (+ 609us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 12:09:55.571743 (+ 22us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 12:09:55.573026 (+ 1283us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 12:09:55.573030 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 12:09:55.574027 (+ 997us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 12:09:55.574038 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 12:09:55.574812 (+ 774us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 12:09:55.574819 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 12:09:55.575071 (+ 252us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 12:09:55.575794 (+ 723us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0809 12:09:55.575827 (+ 33us) client_negotiation.cc:656] calling sasl_client_start()
0809 12:09:55.576613 (+ 786us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0809 12:09:55.580480 (+ 3867us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0809 12:09:55.580486 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0809 12:09:55.580490 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0809 12:09:55.581068 (+ 578us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0809 12:09:55.581500 (+ 432us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0809 12:09:55.581505 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0809 12:09:55.581509 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0809 12:09:55.581638 (+ 129us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0809 12:09:55.582180 (+ 542us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0809 12:09:55.582185 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0809 12:09:55.582431 (+ 246us) client_negotiation.cc:769] Sending connection context
0809 12:09:55.582590 (+ 159us) client_negotiation.cc:240] Negotiation successful
0809 12:09:55.582996 (+ 406us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":514,"thread_start_us":370,"threads_started":1}
I20260809 12:09:55.599879 2468 rpc_server.cc:307] RPC server started. Bound to: 127.0.222.3:35491
I20260809 12:09:55.599910 2612 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.222.3:35491 every 8 connection(s)
I20260809 12:09:55.601189 2468 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskgH5Pjm/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786277382073938-888-0/minicluster-data/ts-2/data/info.pb
I20260809 12:09:55.603451 2613 heartbeater.cc:347] Connected to a master server at 127.0.222.62:42617
I20260809 12:09:55.603678 2613 heartbeater.cc:464] Registering TS with master...
I20260809 12:09:55.604363 2613 heartbeater.cc:511] Master 127.0.222.62:42617 requested a full tablet report, sending...
I20260809 12:09:55.605688 2099 ts_manager.cc:194] Registered new tserver with Master: a390a8c24bbe45baa12c712e9cf3c90d (127.0.222.3:35491)
I20260809 12:09:55.606566 2099 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.222.3@KRBTEST.COM'} at 127.0.222.3:35455
I20260809 12:09:55.609277 888 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskgH5Pjm/build/asan/bin/kudu as pid 2468
I20260809 12:09:55.609467 888 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskgH5Pjm/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786277382073938-888-0/minicluster-data/ts-2/wal/instance
I20260809 12:09:55.609938 888 external_mini_cluster.cc:1428] Setting key 83f4c7370389e7907ff1650944efd6b7
I20260809 12:09:55.612789 888 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
Aug 09 12:09:55 dist-test-slave-2w3w krb5kdc[2051](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786277395, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-user@KRBTEST.COM:
Aug 09 12:09:55 dist-test-slave-2w3w krb5kdc[2051](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786277395, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.0.222.62@KRBTEST.COM
I20260809 12:09:55.666906 2198 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 12:09:55.653059 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49866 (local address 127.0.222.62:42617)
0809 12:09:55.653327 (+ 268us) server_negotiation.cc:206] Beginning negotiation
0809 12:09:55.653336 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0809 12:09:55.653679 (+ 343us) server_negotiation.cc:407] Connection header received
0809 12:09:55.653909 (+ 230us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 12:09:55.653913 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 12:09:55.653979 (+ 66us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 12:09:55.654094 (+ 115us) server_negotiation.cc:226] Negotiated authn=SASL
0809 12:09:55.655547 (+ 1453us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 12:09:55.656302 (+ 755us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 12:09:55.657568 (+ 1266us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 12:09:55.658025 (+ 457us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 12:09:55.658167 (+ 142us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 12:09:55.662021 (+ 3854us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 12:09:55.662025 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 12:09:55.662029 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0809 12:09:55.662060 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0809 12:09:55.664695 (+ 2635us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 12:09:55.665275 (+ 580us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 12:09:55.665278 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 12:09:55.665280 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0809 12:09:55.665401 (+ 121us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 12:09:55.665733 (+ 332us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 12:09:55.665736 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 12:09:55.665737 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0809 12:09:55.666019 (+ 282us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 12:09:55.666215 (+ 196us) server_negotiation.cc:1035] Waiting for connection context
0809 12:09:55.666447 (+ 232us) server_negotiation.cc:299] Negotiation successful
0809 12:09:55.666608 (+ 161us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":90}
I20260809 12:09:55.671028 2099 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:49866:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260809 12:09:55.677990 2099 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260809 12:09:55.713491 2632 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 12:09:55.691444 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.222.3:35491 (local address 127.0.0.1:33046)
0809 12:09:55.704903 (+ 13459us) negotiation.cc:107] Waiting for socket to connect
0809 12:09:55.704922 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0809 12:09:55.705051 (+ 129us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 12:09:55.705959 (+ 908us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 12:09:55.705964 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 12:09:55.705986 (+ 22us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0809 12:09:55.706309 (+ 323us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 12:09:55.706316 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 12:09:55.708049 (+ 1733us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 12:09:55.708054 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 12:09:55.711231 (+ 3177us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 12:09:55.711245 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 12:09:55.712801 (+ 1556us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 12:09:55.712806 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 12:09:55.712929 (+ 123us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 12:09:55.712962 (+ 33us) client_negotiation.cc:769] Sending connection context
0809 12:09:55.713069 (+ 107us) client_negotiation.cc:240] Negotiation successful
0809 12:09:55.713190 (+ 121us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":13322,"mutex_wait_us":82,"thread_start_us":13864,"threads_started":1}
I20260809 12:09:55.713655 2630 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 12:09:55.692118 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33046 (local address 127.0.222.3:35491)
0809 12:09:55.704220 (+ 12102us) server_negotiation.cc:206] Beginning negotiation
0809 12:09:55.704228 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0809 12:09:55.705005 (+ 777us) server_negotiation.cc:407] Connection header received
0809 12:09:55.705206 (+ 201us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 12:09:55.705217 (+ 11us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 12:09:55.705437 (+ 220us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 12:09:55.705623 (+ 186us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0809 12:09:55.706500 (+ 877us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 12:09:55.707882 (+ 1382us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 12:09:55.711402 (+ 3520us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 12:09:55.712647 (+ 1245us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 12:09:55.712748 (+ 101us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 12:09:55.712906 (+ 158us) server_negotiation.cc:1035] Waiting for connection context
0809 12:09:55.713151 (+ 245us) server_negotiation.cc:299] Negotiation successful
0809 12:09:55.713319 (+ 168us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":10817,"thread_start_us":11714,"threads_started":1}
I20260809 12:09:55.715476 2631 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 12:09:55.692528 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.222.2:34441 (local address 127.0.0.1:59500)
0809 12:09:55.705619 (+ 13091us) negotiation.cc:107] Waiting for socket to connect
0809 12:09:55.705637 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0809 12:09:55.705837 (+ 200us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 12:09:55.706509 (+ 672us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 12:09:55.706513 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 12:09:55.706540 (+ 27us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0809 12:09:55.706899 (+ 359us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 12:09:55.706910 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 12:09:55.708539 (+ 1629us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 12:09:55.708543 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 12:09:55.711231 (+ 2688us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 12:09:55.711245 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 12:09:55.714534 (+ 3289us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 12:09:55.714538 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 12:09:55.714661 (+ 123us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 12:09:55.714684 (+ 23us) client_negotiation.cc:769] Sending connection context
0809 12:09:55.714857 (+ 173us) client_negotiation.cc:240] Negotiation successful
0809 12:09:55.715043 (+ 186us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":12944,"mutex_wait_us":10,"thread_start_us":12073,"threads_started":1}
I20260809 12:09:55.715755 2628 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 12:09:55.691255 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51668 (local address 127.0.222.1:41559)
0809 12:09:55.703410 (+ 12155us) server_negotiation.cc:206] Beginning negotiation
0809 12:09:55.703423 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0809 12:09:55.704322 (+ 899us) server_negotiation.cc:407] Connection header received
0809 12:09:55.704441 (+ 119us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 12:09:55.704449 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 12:09:55.704739 (+ 290us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 12:09:55.705043 (+ 304us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0809 12:09:55.706115 (+ 1072us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 12:09:55.707508 (+ 1393us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 12:09:55.710216 (+ 2708us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 12:09:55.711255 (+ 1039us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 12:09:55.711401 (+ 146us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 12:09:55.711637 (+ 236us) server_negotiation.cc:1035] Waiting for connection context
0809 12:09:55.715264 (+ 3627us) server_negotiation.cc:299] Negotiation successful
0809 12:09:55.715429 (+ 165us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":11966,"thread_start_us":11804,"threads_started":1}
I20260809 12:09:55.716557 2629 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 12:09:55.692098 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59500 (local address 127.0.222.2:34441)
0809 12:09:55.703836 (+ 11738us) server_negotiation.cc:206] Beginning negotiation
0809 12:09:55.703844 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0809 12:09:55.705730 (+ 1886us) server_negotiation.cc:407] Connection header received
0809 12:09:55.706027 (+ 297us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 12:09:55.706034 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 12:09:55.706245 (+ 211us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 12:09:55.706450 (+ 205us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0809 12:09:55.707103 (+ 653us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 12:09:55.708372 (+ 1269us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 12:09:55.712978 (+ 4606us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 12:09:55.714350 (+ 1372us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 12:09:55.714504 (+ 154us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 12:09:55.714704 (+ 200us) server_negotiation.cc:1035] Waiting for connection context
0809 12:09:55.716115 (+ 1411us) server_negotiation.cc:299] Negotiation successful
0809 12:09:55.716287 (+ 172us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":11563,"thread_start_us":11433,"threads_started":1}
I20260809 12:09:55.715811 2627 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 12:09:55.691028 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.222.1:41559 (local address 127.0.0.1:51668)
0809 12:09:55.703938 (+ 12910us) negotiation.cc:107] Waiting for socket to connect
0809 12:09:55.703984 (+ 46us) client_negotiation.cc:174] Beginning negotiation
0809 12:09:55.704206 (+ 222us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 12:09:55.705019 (+ 813us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 12:09:55.705031 (+ 12us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 12:09:55.705080 (+ 49us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0809 12:09:55.705641 (+ 561us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 12:09:55.705656 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 12:09:55.707651 (+ 1995us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 12:09:55.707656 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 12:09:55.710069 (+ 2413us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 12:09:55.710081 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 12:09:55.714888 (+ 4807us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 12:09:55.714893 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 12:09:55.715083 (+ 190us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 12:09:55.715106 (+ 23us) client_negotiation.cc:769] Sending connection context
0809 12:09:55.715214 (+ 108us) client_negotiation.cc:240] Negotiation successful
0809 12:09:55.715348 (+ 134us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":12612,"mutex_wait_us":50,"thread_start_us":12468,"threads_started":1}
I20260809 12:09:55.717764 2538 tablet_service.cc:1467] Processing CreateTablet for tablet 0fe5ad0b3883461695535f6e3320b478 (DEFAULT_TABLE table=test-table [id=257842f46bca4f06927027d120e6d811]), partition=RANGE (key) PARTITION UNBOUNDED
I20260809 12:09:55.719679 2538 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 0fe5ad0b3883461695535f6e3320b478. 1 dirs total, 0 dirs full, 0 dirs failed
I20260809 12:09:55.720527 2392 tablet_service.cc:1467] Processing CreateTablet for tablet 0fe5ad0b3883461695535f6e3320b478 (DEFAULT_TABLE table=test-table [id=257842f46bca4f06927027d120e6d811]), partition=RANGE (key) PARTITION UNBOUNDED
I20260809 12:09:55.722003 2392 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 0fe5ad0b3883461695535f6e3320b478. 1 dirs total, 0 dirs full, 0 dirs failed
I20260809 12:09:55.721971 2244 tablet_service.cc:1467] Processing CreateTablet for tablet 0fe5ad0b3883461695535f6e3320b478 (DEFAULT_TABLE table=test-table [id=257842f46bca4f06927027d120e6d811]), partition=RANGE (key) PARTITION UNBOUNDED
I20260809 12:09:55.723134 2244 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 0fe5ad0b3883461695535f6e3320b478. 1 dirs total, 0 dirs full, 0 dirs failed
I20260809 12:09:55.735217 2633 tablet_bootstrap.cc:492] T 0fe5ad0b3883461695535f6e3320b478 P 36f277a9f15a47bd9843342a5fc170f8: Bootstrap starting.
I20260809 12:09:55.738159 2635 tablet_bootstrap.cc:492] T 0fe5ad0b3883461695535f6e3320b478 P 3842c96efca14e43898885e9664afe98: Bootstrap starting.
I20260809 12:09:55.738515 2633 tablet_bootstrap.cc:654] T 0fe5ad0b3883461695535f6e3320b478 P 36f277a9f15a47bd9843342a5fc170f8: Neither blocks nor log segments found. Creating new log.
I20260809 12:09:55.740099 2633 log.cc:824] T 0fe5ad0b3883461695535f6e3320b478 P 36f277a9f15a47bd9843342a5fc170f8: Log is configured to *not* fsync() on all Append() calls
I20260809 12:09:55.742060 2634 tablet_bootstrap.cc:492] T 0fe5ad0b3883461695535f6e3320b478 P a390a8c24bbe45baa12c712e9cf3c90d: Bootstrap starting.
I20260809 12:09:55.742170 2635 tablet_bootstrap.cc:654] T 0fe5ad0b3883461695535f6e3320b478 P 3842c96efca14e43898885e9664afe98: Neither blocks nor log segments found. Creating new log.
I20260809 12:09:55.743670 2635 log.cc:824] T 0fe5ad0b3883461695535f6e3320b478 P 3842c96efca14e43898885e9664afe98: Log is configured to *not* fsync() on all Append() calls
I20260809 12:09:55.743960 2633 tablet_bootstrap.cc:492] T 0fe5ad0b3883461695535f6e3320b478 P 36f277a9f15a47bd9843342a5fc170f8: No bootstrap required, opened a new log
I20260809 12:09:55.744333 2633 ts_tablet_manager.cc:1397] T 0fe5ad0b3883461695535f6e3320b478 P 36f277a9f15a47bd9843342a5fc170f8: Time spent bootstrapping tablet: real 0.011s user 0.000s sys 0.008s
I20260809 12:09:55.746119 2634 tablet_bootstrap.cc:654] T 0fe5ad0b3883461695535f6e3320b478 P a390a8c24bbe45baa12c712e9cf3c90d: Neither blocks nor log segments found. Creating new log.
I20260809 12:09:55.747188 2635 tablet_bootstrap.cc:492] T 0fe5ad0b3883461695535f6e3320b478 P 3842c96efca14e43898885e9664afe98: No bootstrap required, opened a new log
I20260809 12:09:55.747469 2635 ts_tablet_manager.cc:1397] T 0fe5ad0b3883461695535f6e3320b478 P 3842c96efca14e43898885e9664afe98: Time spent bootstrapping tablet: real 0.010s user 0.005s sys 0.003s
I20260809 12:09:55.747445 2634 log.cc:824] T 0fe5ad0b3883461695535f6e3320b478 P a390a8c24bbe45baa12c712e9cf3c90d: Log is configured to *not* fsync() on all Append() calls
I20260809 12:09:55.750741 2634 tablet_bootstrap.cc:492] T 0fe5ad0b3883461695535f6e3320b478 P a390a8c24bbe45baa12c712e9cf3c90d: No bootstrap required, opened a new log
I20260809 12:09:55.751019 2634 ts_tablet_manager.cc:1397] T 0fe5ad0b3883461695535f6e3320b478 P a390a8c24bbe45baa12c712e9cf3c90d: Time spent bootstrapping tablet: real 0.009s user 0.003s sys 0.003s
I20260809 12:09:55.753537 2633 raft_consensus.cc:348] T 0fe5ad0b3883461695535f6e3320b478 P 36f277a9f15a47bd9843342a5fc170f8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "36f277a9f15a47bd9843342a5fc170f8" member_type: VOTER last_known_addr { host: "127.0.222.1" port: 41559 } } peers { permanent_uuid: "a390a8c24bbe45baa12c712e9cf3c90d" member_type: VOTER last_known_addr { host: "127.0.222.3" port: 35491 } } peers { permanent_uuid: "3842c96efca14e43898885e9664afe98" member_type: VOTER last_known_addr { host: "127.0.222.2" port: 34441 } }
I20260809 12:09:55.754137 2633 raft_consensus.cc:374] T 0fe5ad0b3883461695535f6e3320b478 P 36f277a9f15a47bd9843342a5fc170f8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260809 12:09:55.754313 2633 raft_consensus.cc:729] T 0fe5ad0b3883461695535f6e3320b478 P 36f277a9f15a47bd9843342a5fc170f8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 36f277a9f15a47bd9843342a5fc170f8, State: Initialized, Role: FOLLOWER
I20260809 12:09:55.755050 2633 consensus_queue.cc:260] T 0fe5ad0b3883461695535f6e3320b478 P 36f277a9f15a47bd9843342a5fc170f8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "36f277a9f15a47bd9843342a5fc170f8" member_type: VOTER last_known_addr { host: "127.0.222.1" port: 41559 } } peers { permanent_uuid: "a390a8c24bbe45baa12c712e9cf3c90d" member_type: VOTER last_known_addr { host: "127.0.222.3" port: 35491 } } peers { permanent_uuid: "3842c96efca14e43898885e9664afe98" member_type: VOTER last_known_addr { host: "127.0.222.2" port: 34441 } }
I20260809 12:09:55.757046 2633 ts_tablet_manager.cc:1428] T 0fe5ad0b3883461695535f6e3320b478 P 36f277a9f15a47bd9843342a5fc170f8: Time spent starting tablet: real 0.013s user 0.007s sys 0.004s
I20260809 12:09:55.758819 2635 raft_consensus.cc:348] T 0fe5ad0b3883461695535f6e3320b478 P 3842c96efca14e43898885e9664afe98 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "36f277a9f15a47bd9843342a5fc170f8" member_type: VOTER last_known_addr { host: "127.0.222.1" port: 41559 } } peers { permanent_uuid: "a390a8c24bbe45baa12c712e9cf3c90d" member_type: VOTER last_known_addr { host: "127.0.222.3" port: 35491 } } peers { permanent_uuid: "3842c96efca14e43898885e9664afe98" member_type: VOTER last_known_addr { host: "127.0.222.2" port: 34441 } }
I20260809 12:09:55.760308 2635 raft_consensus.cc:374] T 0fe5ad0b3883461695535f6e3320b478 P 3842c96efca14e43898885e9664afe98 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260809 12:09:55.760509 2635 raft_consensus.cc:729] T 0fe5ad0b3883461695535f6e3320b478 P 3842c96efca14e43898885e9664afe98 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3842c96efca14e43898885e9664afe98, State: Initialized, Role: FOLLOWER
I20260809 12:09:55.761386 2635 consensus_queue.cc:260] T 0fe5ad0b3883461695535f6e3320b478 P 3842c96efca14e43898885e9664afe98 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "36f277a9f15a47bd9843342a5fc170f8" member_type: VOTER last_known_addr { host: "127.0.222.1" port: 41559 } } peers { permanent_uuid: "a390a8c24bbe45baa12c712e9cf3c90d" member_type: VOTER last_known_addr { host: "127.0.222.3" port: 35491 } } peers { permanent_uuid: "3842c96efca14e43898885e9664afe98" member_type: VOTER last_known_addr { host: "127.0.222.2" port: 34441 } }
I20260809 12:09:55.762991 2464 heartbeater.cc:503] Master 127.0.222.62:42617 was elected leader, sending a full tablet report...
I20260809 12:09:55.765794 2635 ts_tablet_manager.cc:1428] T 0fe5ad0b3883461695535f6e3320b478 P 3842c96efca14e43898885e9664afe98: Time spent starting tablet: real 0.018s user 0.010s sys 0.005s
I20260809 12:09:55.765930 2634 raft_consensus.cc:348] T 0fe5ad0b3883461695535f6e3320b478 P a390a8c24bbe45baa12c712e9cf3c90d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "36f277a9f15a47bd9843342a5fc170f8" member_type: VOTER last_known_addr { host: "127.0.222.1" port: 41559 } } peers { permanent_uuid: "a390a8c24bbe45baa12c712e9cf3c90d" member_type: VOTER last_known_addr { host: "127.0.222.3" port: 35491 } } peers { permanent_uuid: "3842c96efca14e43898885e9664afe98" member_type: VOTER last_known_addr { host: "127.0.222.2" port: 34441 } }
I20260809 12:09:55.766618 2634 raft_consensus.cc:374] T 0fe5ad0b3883461695535f6e3320b478 P a390a8c24bbe45baa12c712e9cf3c90d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260809 12:09:55.766875 2634 raft_consensus.cc:729] T 0fe5ad0b3883461695535f6e3320b478 P a390a8c24bbe45baa12c712e9cf3c90d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a390a8c24bbe45baa12c712e9cf3c90d, State: Initialized, Role: FOLLOWER
I20260809 12:09:55.767966 2634 consensus_queue.cc:260] T 0fe5ad0b3883461695535f6e3320b478 P a390a8c24bbe45baa12c712e9cf3c90d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "36f277a9f15a47bd9843342a5fc170f8" member_type: VOTER last_known_addr { host: "127.0.222.1" port: 41559 } } peers { permanent_uuid: "a390a8c24bbe45baa12c712e9cf3c90d" member_type: VOTER last_known_addr { host: "127.0.222.3" port: 35491 } } peers { permanent_uuid: "3842c96efca14e43898885e9664afe98" member_type: VOTER last_known_addr { host: "127.0.222.2" port: 34441 } }
I20260809 12:09:55.769886 2613 heartbeater.cc:503] Master 127.0.222.62:42617 was elected leader, sending a full tablet report...
I20260809 12:09:55.770460 2634 ts_tablet_manager.cc:1428] T 0fe5ad0b3883461695535f6e3320b478 P a390a8c24bbe45baa12c712e9cf3c90d: Time spent starting tablet: real 0.019s user 0.013s sys 0.004s
W20260809 12:09:55.781550 2319 tablet.cc:2369] T 0fe5ad0b3883461695535f6e3320b478 P 36f277a9f15a47bd9843342a5fc170f8: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260809 12:09:55.814036 2465 tablet.cc:2369] T 0fe5ad0b3883461695535f6e3320b478 P 3842c96efca14e43898885e9664afe98: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260809 12:09:55.852912 2614 tablet.cc:2369] T 0fe5ad0b3883461695535f6e3320b478 P a390a8c24bbe45baa12c712e9cf3c90d: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260809 12:09:56.029887 2639 raft_consensus.cc:482] T 0fe5ad0b3883461695535f6e3320b478 P 36f277a9f15a47bd9843342a5fc170f8 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260809 12:09:56.030331 2639 raft_consensus.cc:504] T 0fe5ad0b3883461695535f6e3320b478 P 36f277a9f15a47bd9843342a5fc170f8 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "36f277a9f15a47bd9843342a5fc170f8" member_type: VOTER last_known_addr { host: "127.0.222.1" port: 41559 } } peers { permanent_uuid: "a390a8c24bbe45baa12c712e9cf3c90d" member_type: VOTER last_known_addr { host: "127.0.222.3" port: 35491 } } peers { permanent_uuid: "3842c96efca14e43898885e9664afe98" member_type: VOTER last_known_addr { host: "127.0.222.2" port: 34441 } }
I20260809 12:09:56.032629 2639 leader_election.cc:288] T 0fe5ad0b3883461695535f6e3320b478 P 36f277a9f15a47bd9843342a5fc170f8 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers a390a8c24bbe45baa12c712e9cf3c90d (127.0.222.3:35491), 3842c96efca14e43898885e9664afe98 (127.0.222.2:34441)
I20260809 12:09:56.034592 2640 raft_consensus.cc:482] T 0fe5ad0b3883461695535f6e3320b478 P 3842c96efca14e43898885e9664afe98 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260809 12:09:56.035295 2640 raft_consensus.cc:504] T 0fe5ad0b3883461695535f6e3320b478 P 3842c96efca14e43898885e9664afe98 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "36f277a9f15a47bd9843342a5fc170f8" member_type: VOTER last_known_addr { host: "127.0.222.1" port: 41559 } } peers { permanent_uuid: "a390a8c24bbe45baa12c712e9cf3c90d" member_type: VOTER last_known_addr { host: "127.0.222.3" port: 35491 } } peers { permanent_uuid: "3842c96efca14e43898885e9664afe98" member_type: VOTER last_known_addr { host: "127.0.222.2" port: 34441 } }
I20260809 12:09:56.040735 2640 leader_election.cc:288] T 0fe5ad0b3883461695535f6e3320b478 P 3842c96efca14e43898885e9664afe98 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 36f277a9f15a47bd9843342a5fc170f8 (127.0.222.1:41559), a390a8c24bbe45baa12c712e9cf3c90d (127.0.222.3:35491)
I20260809 12:09:56.050257 2630 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 12:09:56.033346 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.222.1:33405 (local address 127.0.222.3:35491)
0809 12:09:56.033706 (+ 360us) server_negotiation.cc:206] Beginning negotiation
0809 12:09:56.033713 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0809 12:09:56.036791 (+ 3078us) server_negotiation.cc:407] Connection header received
0809 12:09:56.037142 (+ 351us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 12:09:56.037150 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 12:09:56.037229 (+ 79us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 12:09:56.037292 (+ 63us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0809 12:09:56.038866 (+ 1574us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 12:09:56.040411 (+ 1545us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 12:09:56.045775 (+ 5364us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 12:09:56.048240 (+ 2465us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 12:09:56.048379 (+ 139us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 12:09:56.048570 (+ 191us) server_negotiation.cc:1035] Waiting for connection context
0809 12:09:56.049554 (+ 984us) server_negotiation.cc:299] Negotiation successful
0809 12:09:56.049732 (+ 178us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":200}
I20260809 12:09:56.051331 2642 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 12:09:56.032918 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.222.3:35491 (local address 127.0.222.1:33405)
0809 12:09:56.036458 (+ 3540us) negotiation.cc:107] Waiting for socket to connect
0809 12:09:56.036519 (+ 61us) client_negotiation.cc:174] Beginning negotiation
0809 12:09:56.036942 (+ 423us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 12:09:56.037563 (+ 621us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 12:09:56.037566 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 12:09:56.037595 (+ 29us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0809 12:09:56.038658 (+ 1063us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 12:09:56.038676 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 12:09:56.040652 (+ 1976us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 12:09:56.040659 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 12:09:56.045475 (+ 4816us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 12:09:56.045498 (+ 23us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 12:09:56.048957 (+ 3459us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 12:09:56.048966 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 12:09:56.049324 (+ 358us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 12:09:56.049363 (+ 39us) client_negotiation.cc:769] Sending connection context
0809 12:09:56.049510 (+ 147us) client_negotiation.cc:240] Negotiation successful
0809 12:09:56.049735 (+ 225us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":3308,"mutex_wait_us":1,"thread_start_us":2317,"threads_started":1}
I20260809 12:09:56.052371 2629 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 12:09:56.033706 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.222.1:37015 (local address 127.0.222.2:34441)
0809 12:09:56.034008 (+ 302us) server_negotiation.cc:206] Beginning negotiation
0809 12:09:56.034017 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0809 12:09:56.036528 (+ 2511us) server_negotiation.cc:407] Connection header received
0809 12:09:56.036812 (+ 284us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 12:09:56.036820 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 12:09:56.036927 (+ 107us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 12:09:56.037030 (+ 103us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0809 12:09:56.038733 (+ 1703us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 12:09:56.043173 (+ 4440us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 12:09:56.047982 (+ 4809us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 12:09:56.050123 (+ 2141us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 12:09:56.050333 (+ 210us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 12:09:56.050486 (+ 153us) server_negotiation.cc:1035] Waiting for connection context
0809 12:09:56.051597 (+ 1111us) server_negotiation.cc:299] Negotiation successful
0809 12:09:56.051739 (+ 142us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":47}
I20260809 12:09:56.053755 2643 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 12:09:56.032837 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.222.2:34441 (local address 127.0.222.1:37015)
0809 12:09:56.036348 (+ 3511us) negotiation.cc:107] Waiting for socket to connect
0809 12:09:56.036372 (+ 24us) client_negotiation.cc:174] Beginning negotiation
0809 12:09:56.036641 (+ 269us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 12:09:56.037124 (+ 483us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 12:09:56.037165 (+ 41us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 12:09:56.037216 (+ 51us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0809 12:09:56.038531 (+ 1315us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 12:09:56.038551 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 12:09:56.043392 (+ 4841us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 12:09:56.043401 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 12:09:56.047695 (+ 4294us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 12:09:56.047719 (+ 24us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 12:09:56.050874 (+ 3155us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 12:09:56.050884 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 12:09:56.051371 (+ 487us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 12:09:56.051410 (+ 39us) client_negotiation.cc:769] Sending connection context
0809 12:09:56.052805 (+ 1395us) client_negotiation.cc:240] Negotiation successful
0809 12:09:56.052970 (+ 165us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":2968,"thread_start_us":2967,"threads_started":1}
I20260809 12:09:56.053390 2566 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0fe5ad0b3883461695535f6e3320b478" candidate_uuid: "36f277a9f15a47bd9843342a5fc170f8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a390a8c24bbe45baa12c712e9cf3c90d" is_pre_election: true
I20260809 12:09:56.054313 2566 raft_consensus.cc:2454] T 0fe5ad0b3883461695535f6e3320b478 P a390a8c24bbe45baa12c712e9cf3c90d [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 36f277a9f15a47bd9843342a5fc170f8 in term 0.
I20260809 12:09:56.055570 2192 leader_election.cc:302] T 0fe5ad0b3883461695535f6e3320b478 P 36f277a9f15a47bd9843342a5fc170f8 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 36f277a9f15a47bd9843342a5fc170f8, a390a8c24bbe45baa12c712e9cf3c90d; no voters:
I20260809 12:09:56.055400 2413 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0fe5ad0b3883461695535f6e3320b478" candidate_uuid: "36f277a9f15a47bd9843342a5fc170f8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3842c96efca14e43898885e9664afe98" is_pre_election: true
I20260809 12:09:56.056389 2639 raft_consensus.cc:2781] T 0fe5ad0b3883461695535f6e3320b478 P 36f277a9f15a47bd9843342a5fc170f8 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260809 12:09:56.056761 2639 raft_consensus.cc:482] T 0fe5ad0b3883461695535f6e3320b478 P 36f277a9f15a47bd9843342a5fc170f8 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260809 12:09:56.056629 2413 raft_consensus.cc:2454] T 0fe5ad0b3883461695535f6e3320b478 P 3842c96efca14e43898885e9664afe98 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 36f277a9f15a47bd9843342a5fc170f8 in term 0.
I20260809 12:09:56.056965 2639 raft_consensus.cc:3037] T 0fe5ad0b3883461695535f6e3320b478 P 36f277a9f15a47bd9843342a5fc170f8 [term 0 FOLLOWER]: Advancing to term 1
I20260809 12:09:56.062665 2639 raft_consensus.cc:504] T 0fe5ad0b3883461695535f6e3320b478 P 36f277a9f15a47bd9843342a5fc170f8 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "36f277a9f15a47bd9843342a5fc170f8" member_type: VOTER last_known_addr { host: "127.0.222.1" port: 41559 } } peers { permanent_uuid: "a390a8c24bbe45baa12c712e9cf3c90d" member_type: VOTER last_known_addr { host: "127.0.222.3" port: 35491 } } peers { permanent_uuid: "3842c96efca14e43898885e9664afe98" member_type: VOTER last_known_addr { host: "127.0.222.2" port: 34441 } }
I20260809 12:09:56.064589 2566 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0fe5ad0b3883461695535f6e3320b478" candidate_uuid: "36f277a9f15a47bd9843342a5fc170f8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a390a8c24bbe45baa12c712e9cf3c90d"
I20260809 12:09:56.064858 2628 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 12:09:56.039145 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.222.2:48271 (local address 127.0.222.1:41559)
0809 12:09:56.039450 (+ 305us) server_negotiation.cc:206] Beginning negotiation
0809 12:09:56.039457 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0809 12:09:56.049830 (+ 10373us) server_negotiation.cc:407] Connection header received
0809 12:09:56.050204 (+ 374us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 12:09:56.050210 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 12:09:56.050343 (+ 133us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 12:09:56.050421 (+ 78us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0809 12:09:56.053298 (+ 2877us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 12:09:56.054922 (+ 1624us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 12:09:56.060279 (+ 5357us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 12:09:56.062778 (+ 2499us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 12:09:56.062950 (+ 172us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 12:09:56.063132 (+ 182us) server_negotiation.cc:1035] Waiting for connection context
0809 12:09:56.064302 (+ 1170us) server_negotiation.cc:299] Negotiation successful
0809 12:09:56.064516 (+ 214us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":90}
I20260809 12:09:56.065033 2566 raft_consensus.cc:3037] T 0fe5ad0b3883461695535f6e3320b478 P a390a8c24bbe45baa12c712e9cf3c90d [term 0 FOLLOWER]: Advancing to term 1
I20260809 12:09:56.066293 2646 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 12:09:56.040799 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.222.1:41559 (local address 127.0.222.2:48271)
0809 12:09:56.049519 (+ 8720us) negotiation.cc:107] Waiting for socket to connect
0809 12:09:56.049539 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0809 12:09:56.050003 (+ 464us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 12:09:56.050797 (+ 794us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 12:09:56.050803 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 12:09:56.050840 (+ 37us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0809 12:09:56.053101 (+ 2261us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 12:09:56.053119 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 12:09:56.055430 (+ 2311us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 12:09:56.055435 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 12:09:56.060059 (+ 4624us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 12:09:56.060074 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 12:09:56.063777 (+ 3703us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 12:09:56.063787 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 12:09:56.064091 (+ 304us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 12:09:56.064138 (+ 47us) client_negotiation.cc:769] Sending connection context
0809 12:09:56.065403 (+ 1265us) client_negotiation.cc:240] Negotiation successful
0809 12:09:56.065604 (+ 201us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":8516,"thread_start_us":7028,"threads_started":1}
I20260809 12:09:56.067859 2270 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0fe5ad0b3883461695535f6e3320b478" candidate_uuid: "3842c96efca14e43898885e9664afe98" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "36f277a9f15a47bd9843342a5fc170f8" is_pre_election: true
I20260809 12:09:56.068857 2270 raft_consensus.cc:2379] T 0fe5ad0b3883461695535f6e3320b478 P 36f277a9f15a47bd9843342a5fc170f8 [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 3842c96efca14e43898885e9664afe98 in current term 1: Already voted for candidate 36f277a9f15a47bd9843342a5fc170f8 in this term.
I20260809 12:09:56.069581 2645 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 12:09:56.039789 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.222.2:36399 (local address 127.0.222.3:35491)
0809 12:09:56.041099 (+ 1310us) server_negotiation.cc:206] Beginning negotiation
0809 12:09:56.041106 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0809 12:09:56.051468 (+ 10362us) server_negotiation.cc:407] Connection header received
0809 12:09:56.051760 (+ 292us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 12:09:56.051766 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 12:09:56.051837 (+ 71us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 12:09:56.052159 (+ 322us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0809 12:09:56.054546 (+ 2387us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 12:09:56.057765 (+ 3219us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 12:09:56.064838 (+ 7073us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 12:09:56.067261 (+ 2423us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 12:09:56.067428 (+ 167us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 12:09:56.067703 (+ 275us) server_negotiation.cc:1035] Waiting for connection context
0809 12:09:56.068674 (+ 971us) server_negotiation.cc:299] Negotiation successful
0809 12:09:56.068947 (+ 273us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1045,"thread_start_us":1126,"threads_started":1}
I20260809 12:09:56.070784 2647 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 12:09:56.041136 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.222.3:35491 (local address 127.0.222.2:36399)
0809 12:09:56.051123 (+ 9987us) negotiation.cc:107] Waiting for socket to connect
0809 12:09:56.051154 (+ 31us) client_negotiation.cc:174] Beginning negotiation
0809 12:09:56.051621 (+ 467us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 12:09:56.052928 (+ 1307us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 12:09:56.052933 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 12:09:56.052959 (+ 26us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0809 12:09:56.054200 (+ 1241us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 12:09:56.054227 (+ 27us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 12:09:56.058026 (+ 3799us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 12:09:56.058032 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 12:09:56.064580 (+ 6548us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 12:09:56.064597 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 12:09:56.068030 (+ 3433us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 12:09:56.068037 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 12:09:56.068370 (+ 333us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 12:09:56.068408 (+ 38us) client_negotiation.cc:769] Sending connection context
0809 12:09:56.069801 (+ 1393us) client_negotiation.cc:240] Negotiation successful
0809 12:09:56.069953 (+ 152us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":9665,"mutex_wait_us":2,"thread_start_us":7797,"threads_started":1}
I20260809 12:09:56.071938 2565 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0fe5ad0b3883461695535f6e3320b478" candidate_uuid: "3842c96efca14e43898885e9664afe98" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a390a8c24bbe45baa12c712e9cf3c90d" is_pre_election: true
I20260809 12:09:56.073143 2566 raft_consensus.cc:2454] T 0fe5ad0b3883461695535f6e3320b478 P a390a8c24bbe45baa12c712e9cf3c90d [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 36f277a9f15a47bd9843342a5fc170f8 in term 1.
I20260809 12:09:56.073226 2343 leader_election.cc:302] T 0fe5ad0b3883461695535f6e3320b478 P 3842c96efca14e43898885e9664afe98 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 3842c96efca14e43898885e9664afe98; no voters: 36f277a9f15a47bd9843342a5fc170f8, a390a8c24bbe45baa12c712e9cf3c90d
I20260809 12:09:56.073836 2192 leader_election.cc:302] T 0fe5ad0b3883461695535f6e3320b478 P 36f277a9f15a47bd9843342a5fc170f8 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 36f277a9f15a47bd9843342a5fc170f8, a390a8c24bbe45baa12c712e9cf3c90d; no voters:
I20260809 12:09:56.074322 2640 raft_consensus.cc:3037] T 0fe5ad0b3883461695535f6e3320b478 P 3842c96efca14e43898885e9664afe98 [term 0 FOLLOWER]: Advancing to term 1
I20260809 12:09:56.074548 2413 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0fe5ad0b3883461695535f6e3320b478" candidate_uuid: "36f277a9f15a47bd9843342a5fc170f8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3842c96efca14e43898885e9664afe98"
I20260809 12:09:56.075372 2639 leader_election.cc:288] T 0fe5ad0b3883461695535f6e3320b478 P 36f277a9f15a47bd9843342a5fc170f8 [CANDIDATE]: Term 1 election: Requested vote from peers a390a8c24bbe45baa12c712e9cf3c90d (127.0.222.3:35491), 3842c96efca14e43898885e9664afe98 (127.0.222.2:34441)
I20260809 12:09:56.075791 2639 raft_consensus.cc:2781] T 0fe5ad0b3883461695535f6e3320b478 P 36f277a9f15a47bd9843342a5fc170f8 [term 1 FOLLOWER]: Leader election won for term 1
I20260809 12:09:56.076020 2639 raft_consensus.cc:686] T 0fe5ad0b3883461695535f6e3320b478 P 36f277a9f15a47bd9843342a5fc170f8 [term 1 LEADER]: Becoming Leader. State: Replica: 36f277a9f15a47bd9843342a5fc170f8, State: Running, Role: LEADER
I20260809 12:09:56.077066 2639 consensus_queue.cc:237] T 0fe5ad0b3883461695535f6e3320b478 P 36f277a9f15a47bd9843342a5fc170f8 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "36f277a9f15a47bd9843342a5fc170f8" member_type: VOTER last_known_addr { host: "127.0.222.1" port: 41559 } } peers { permanent_uuid: "a390a8c24bbe45baa12c712e9cf3c90d" member_type: VOTER last_known_addr { host: "127.0.222.3" port: 35491 } } peers { permanent_uuid: "3842c96efca14e43898885e9664afe98" member_type: VOTER last_known_addr { host: "127.0.222.2" port: 34441 } }
I20260809 12:09:56.079859 2640 raft_consensus.cc:2726] T 0fe5ad0b3883461695535f6e3320b478 P 3842c96efca14e43898885e9664afe98 [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260809 12:09:56.085026 2413 raft_consensus.cc:2454] T 0fe5ad0b3883461695535f6e3320b478 P 3842c96efca14e43898885e9664afe98 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 36f277a9f15a47bd9843342a5fc170f8 in term 1.
I20260809 12:09:56.085176 2099 catalog_manager.cc:5495] T 0fe5ad0b3883461695535f6e3320b478 P 36f277a9f15a47bd9843342a5fc170f8 reported cstate change: term changed from 0 to 1, leader changed from <none> to 36f277a9f15a47bd9843342a5fc170f8 (127.0.222.1). New cstate: current_term: 1 leader_uuid: "36f277a9f15a47bd9843342a5fc170f8" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "36f277a9f15a47bd9843342a5fc170f8" member_type: VOTER last_known_addr { host: "127.0.222.1" port: 41559 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "a390a8c24bbe45baa12c712e9cf3c90d" member_type: VOTER last_known_addr { host: "127.0.222.3" port: 35491 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "3842c96efca14e43898885e9664afe98" member_type: VOTER last_known_addr { host: "127.0.222.2" port: 34441 } health_report { overall_health: UNKNOWN } } }
I20260809 12:09:56.187265 2628 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 12:09:56.178695 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51678 (local address 127.0.222.1:41559)
0809 12:09:56.178865 (+ 170us) server_negotiation.cc:206] Beginning negotiation
0809 12:09:56.178869 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0809 12:09:56.179728 (+ 859us) server_negotiation.cc:407] Connection header received
0809 12:09:56.180062 (+ 334us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 12:09:56.180066 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 12:09:56.180146 (+ 80us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 12:09:56.180296 (+ 150us) server_negotiation.cc:226] Negotiated authn=TOKEN
0809 12:09:56.181262 (+ 966us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 12:09:56.182438 (+ 1176us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 12:09:56.184378 (+ 1940us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 12:09:56.184941 (+ 563us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 12:09:56.185134 (+ 193us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 12:09:56.185725 (+ 591us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0809 12:09:56.186358 (+ 633us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0809 12:09:56.186524 (+ 166us) server_negotiation.cc:1035] Waiting for connection context
0809 12:09:56.186653 (+ 129us) server_negotiation.cc:299] Negotiation successful
0809 12:09:56.186740 (+ 87us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":52}
I20260809 12:09:56.198403 2566 raft_consensus.cc:1261] T 0fe5ad0b3883461695535f6e3320b478 P a390a8c24bbe45baa12c712e9cf3c90d [term 1 FOLLOWER]: Refusing update from remote peer 36f277a9f15a47bd9843342a5fc170f8: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260809 12:09:56.199539 2648 consensus_queue.cc:1035] T 0fe5ad0b3883461695535f6e3320b478 P 36f277a9f15a47bd9843342a5fc170f8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a390a8c24bbe45baa12c712e9cf3c90d" member_type: VOTER last_known_addr { host: "127.0.222.3" port: 35491 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260809 12:09:56.200716 2413 raft_consensus.cc:1261] T 0fe5ad0b3883461695535f6e3320b478 P 3842c96efca14e43898885e9664afe98 [term 1 FOLLOWER]: Refusing update from remote peer 36f277a9f15a47bd9843342a5fc170f8: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260809 12:09:56.201642 2648 consensus_queue.cc:1035] T 0fe5ad0b3883461695535f6e3320b478 P 36f277a9f15a47bd9843342a5fc170f8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3842c96efca14e43898885e9664afe98" member_type: VOTER last_known_addr { host: "127.0.222.2" port: 34441 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260809 12:09:56.219440 2652 mvcc.cc:205] Tried to move back new op lower bound from 7316592214816354304 to 7316592214338842624. Current Snapshot: MvccSnapshot[applied={T|T < 7316592214816354304 or (T in {7316592214816354304})}]
I20260809 12:09:56.225934 2655 mvcc.cc:205] Tried to move back new op lower bound from 7316592214816354304 to 7316592214338842624. Current Snapshot: MvccSnapshot[applied={T|T < 7316592214816354304}]
I20260809 12:09:56.249296 2098 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:49866:
table { table_name: "test-table" } modify_external_catalogs: true
I20260809 12:09:56.249763 2098 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:49866:
table { table_name: "test-table" } modify_external_catalogs: true
I20260809 12:09:56.254643 2098 catalog_manager.cc:5782] T 00000000000000000000000000000000 P eb0ae6c68a9b460ebbcc6098cfb950cb: Sending DeleteTablet for 3 replicas of tablet 0fe5ad0b3883461695535f6e3320b478
I20260809 12:09:56.256305 2244 tablet_service.cc:1514] Processing DeleteTablet for tablet 0fe5ad0b3883461695535f6e3320b478 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-09 12:09:56 UTC) from {username='kudu', principal='kudu/127.0.222.62@KRBTEST.COM'} at 127.0.0.1:51668
I20260809 12:09:56.256421 2392 tablet_service.cc:1514] Processing DeleteTablet for tablet 0fe5ad0b3883461695535f6e3320b478 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-09 12:09:56 UTC) from {username='kudu', principal='kudu/127.0.222.62@KRBTEST.COM'} at 127.0.0.1:59500
I20260809 12:09:56.256410 2538 tablet_service.cc:1514] Processing DeleteTablet for tablet 0fe5ad0b3883461695535f6e3320b478 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-09 12:09:56 UTC) from {username='kudu', principal='kudu/127.0.222.62@KRBTEST.COM'} at 127.0.0.1:33046
I20260809 12:09:56.262405 2098 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:49866:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260809 12:09:56.263427 2098 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260809 12:09:56.272049 2664 tablet_replica.cc:330] T 0fe5ad0b3883461695535f6e3320b478 P 36f277a9f15a47bd9843342a5fc170f8: stopping tablet replica
I20260809 12:09:56.272815 2664 raft_consensus.cc:2229] T 0fe5ad0b3883461695535f6e3320b478 P 36f277a9f15a47bd9843342a5fc170f8 [term 1 LEADER]: Raft consensus shutting down.
I20260809 12:09:56.273804 2664 raft_consensus.cc:2258] T 0fe5ad0b3883461695535f6e3320b478 P 36f277a9f15a47bd9843342a5fc170f8 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260809 12:09:56.276738 2664 ts_tablet_manager.cc:1905] T 0fe5ad0b3883461695535f6e3320b478 P 36f277a9f15a47bd9843342a5fc170f8: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260809 12:09:56.278415 2244 tablet_service.cc:1467] Processing CreateTablet for tablet 998406d582744483ae82e04cf4be1539 (DEFAULT_TABLE table=test-table [id=ddca40f91a774027929c394b80bc3930]), partition=RANGE (key) PARTITION UNBOUNDED
I20260809 12:09:56.278455 2391 tablet_service.cc:1467] Processing CreateTablet for tablet 998406d582744483ae82e04cf4be1539 (DEFAULT_TABLE table=test-table [id=ddca40f91a774027929c394b80bc3930]), partition=RANGE (key) PARTITION UNBOUNDED
I20260809 12:09:56.279103 2391 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 998406d582744483ae82e04cf4be1539. 1 dirs total, 0 dirs full, 0 dirs failed
I20260809 12:09:56.279246 2244 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 998406d582744483ae82e04cf4be1539. 1 dirs total, 0 dirs full, 0 dirs failed
I20260809 12:09:56.284170 2537 tablet_service.cc:1467] Processing CreateTablet for tablet 998406d582744483ae82e04cf4be1539 (DEFAULT_TABLE table=test-table [id=ddca40f91a774027929c394b80bc3930]), partition=RANGE (key) PARTITION UNBOUNDED
I20260809 12:09:56.284765 2537 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 998406d582744483ae82e04cf4be1539. 1 dirs total, 0 dirs full, 0 dirs failed
I20260809 12:09:56.300096 2666 tablet_bootstrap.cc:492] T 998406d582744483ae82e04cf4be1539 P a390a8c24bbe45baa12c712e9cf3c90d: Bootstrap starting.
I20260809 12:09:56.309145 2665 tablet_bootstrap.cc:492] T 998406d582744483ae82e04cf4be1539 P 3842c96efca14e43898885e9664afe98: Bootstrap starting.
I20260809 12:09:56.309350 2664 ts_tablet_manager.cc:1918] T 0fe5ad0b3883461695535f6e3320b478 P 36f277a9f15a47bd9843342a5fc170f8: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260809 12:09:56.309695 2662 tablet_replica.cc:330] T 0fe5ad0b3883461695535f6e3320b478 P 3842c96efca14e43898885e9664afe98: stopping tablet replica
I20260809 12:09:56.309739 2664 log.cc:1196] T 0fe5ad0b3883461695535f6e3320b478 P 36f277a9f15a47bd9843342a5fc170f8: Deleting WAL directory at /tmp/dist-test-taskgH5Pjm/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786277382073938-888-0/minicluster-data/ts-0/wal/wals/0fe5ad0b3883461695535f6e3320b478
I20260809 12:09:56.310422 2662 raft_consensus.cc:2229] T 0fe5ad0b3883461695535f6e3320b478 P 3842c96efca14e43898885e9664afe98 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260809 12:09:56.310668 2667 tablet_bootstrap.cc:492] T 998406d582744483ae82e04cf4be1539 P 36f277a9f15a47bd9843342a5fc170f8: Bootstrap starting.
I20260809 12:09:56.310904 2664 ts_tablet_manager.cc:1939] T 0fe5ad0b3883461695535f6e3320b478 P 36f277a9f15a47bd9843342a5fc170f8: Deleting consensus metadata
I20260809 12:09:56.311667 2663 tablet_replica.cc:330] T 0fe5ad0b3883461695535f6e3320b478 P a390a8c24bbe45baa12c712e9cf3c90d: stopping tablet replica
I20260809 12:09:56.312526 2663 raft_consensus.cc:2229] T 0fe5ad0b3883461695535f6e3320b478 P a390a8c24bbe45baa12c712e9cf3c90d [term 1 FOLLOWER]: Raft consensus shutting down.
I20260809 12:09:56.313900 2663 raft_consensus.cc:2258] T 0fe5ad0b3883461695535f6e3320b478 P a390a8c24bbe45baa12c712e9cf3c90d [term 1 FOLLOWER]: Raft consensus is shut down!
I20260809 12:09:56.314673 2666 tablet_bootstrap.cc:654] T 998406d582744483ae82e04cf4be1539 P a390a8c24bbe45baa12c712e9cf3c90d: Neither blocks nor log segments found. Creating new log.
I20260809 12:09:56.314851 2665 tablet_bootstrap.cc:654] T 998406d582744483ae82e04cf4be1539 P 3842c96efca14e43898885e9664afe98: Neither blocks nor log segments found. Creating new log.
I20260809 12:09:56.316826 2663 ts_tablet_manager.cc:1905] T 0fe5ad0b3883461695535f6e3320b478 P a390a8c24bbe45baa12c712e9cf3c90d: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260809 12:09:56.317179 2662 raft_consensus.cc:2258] T 0fe5ad0b3883461695535f6e3320b478 P 3842c96efca14e43898885e9664afe98 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260809 12:09:56.319976 2667 tablet_bootstrap.cc:654] T 998406d582744483ae82e04cf4be1539 P 36f277a9f15a47bd9843342a5fc170f8: Neither blocks nor log segments found. Creating new log.
I20260809 12:09:56.320072 2662 ts_tablet_manager.cc:1905] T 0fe5ad0b3883461695535f6e3320b478 P 3842c96efca14e43898885e9664afe98: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260809 12:09:56.326092 2083 catalog_manager.cc:4841] TS 36f277a9f15a47bd9843342a5fc170f8 (127.0.222.1:41559): tablet 0fe5ad0b3883461695535f6e3320b478 (table test-table [id=257842f46bca4f06927027d120e6d811]) successfully deleted
I20260809 12:09:56.339092 2665 tablet_bootstrap.cc:492] T 998406d582744483ae82e04cf4be1539 P 3842c96efca14e43898885e9664afe98: No bootstrap required, opened a new log
I20260809 12:09:56.339396 2665 ts_tablet_manager.cc:1397] T 998406d582744483ae82e04cf4be1539 P 3842c96efca14e43898885e9664afe98: Time spent bootstrapping tablet: real 0.030s user 0.007s sys 0.001s
I20260809 12:09:56.341064 2665 raft_consensus.cc:348] T 998406d582744483ae82e04cf4be1539 P 3842c96efca14e43898885e9664afe98 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "36f277a9f15a47bd9843342a5fc170f8" member_type: VOTER last_known_addr { host: "127.0.222.1" port: 41559 } } peers { permanent_uuid: "3842c96efca14e43898885e9664afe98" member_type: VOTER last_known_addr { host: "127.0.222.2" port: 34441 } } peers { permanent_uuid: "a390a8c24bbe45baa12c712e9cf3c90d" member_type: VOTER last_known_addr { host: "127.0.222.3" port: 35491 } }
I20260809 12:09:56.341507 2665 raft_consensus.cc:374] T 998406d582744483ae82e04cf4be1539 P 3842c96efca14e43898885e9664afe98 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260809 12:09:56.341746 2665 raft_consensus.cc:729] T 998406d582744483ae82e04cf4be1539 P 3842c96efca14e43898885e9664afe98 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3842c96efca14e43898885e9664afe98, State: Initialized, Role: FOLLOWER
I20260809 12:09:56.342219 2665 consensus_queue.cc:260] T 998406d582744483ae82e04cf4be1539 P 3842c96efca14e43898885e9664afe98 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "36f277a9f15a47bd9843342a5fc170f8" member_type: VOTER last_known_addr { host: "127.0.222.1" port: 41559 } } peers { permanent_uuid: "3842c96efca14e43898885e9664afe98" member_type: VOTER last_known_addr { host: "127.0.222.2" port: 34441 } } peers { permanent_uuid: "a390a8c24bbe45baa12c712e9cf3c90d" member_type: VOTER last_known_addr { host: "127.0.222.3" port: 35491 } }
I20260809 12:09:56.343120 2665 ts_tablet_manager.cc:1428] T 998406d582744483ae82e04cf4be1539 P 3842c96efca14e43898885e9664afe98: Time spent starting tablet: real 0.004s user 0.000s sys 0.004s
I20260809 12:09:56.349434 2666 tablet_bootstrap.cc:492] T 998406d582744483ae82e04cf4be1539 P a390a8c24bbe45baa12c712e9cf3c90d: No bootstrap required, opened a new log
I20260809 12:09:56.349704 2666 ts_tablet_manager.cc:1397] T 998406d582744483ae82e04cf4be1539 P a390a8c24bbe45baa12c712e9cf3c90d: Time spent bootstrapping tablet: real 0.050s user 0.018s sys 0.017s
I20260809 12:09:56.349859 2662 ts_tablet_manager.cc:1918] T 0fe5ad0b3883461695535f6e3320b478 P 3842c96efca14e43898885e9664afe98: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260809 12:09:56.350119 2662 log.cc:1196] T 0fe5ad0b3883461695535f6e3320b478 P 3842c96efca14e43898885e9664afe98: Deleting WAL directory at /tmp/dist-test-taskgH5Pjm/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786277382073938-888-0/minicluster-data/ts-1/wal/wals/0fe5ad0b3883461695535f6e3320b478
I20260809 12:09:56.351099 2662 ts_tablet_manager.cc:1939] T 0fe5ad0b3883461695535f6e3320b478 P 3842c96efca14e43898885e9664afe98: Deleting consensus metadata
I20260809 12:09:56.351202 2663 ts_tablet_manager.cc:1918] T 0fe5ad0b3883461695535f6e3320b478 P a390a8c24bbe45baa12c712e9cf3c90d: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260809 12:09:56.351464 2663 log.cc:1196] T 0fe5ad0b3883461695535f6e3320b478 P a390a8c24bbe45baa12c712e9cf3c90d: Deleting WAL directory at /tmp/dist-test-taskgH5Pjm/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786277382073938-888-0/minicluster-data/ts-2/wal/wals/0fe5ad0b3883461695535f6e3320b478
I20260809 12:09:56.351397 2666 raft_consensus.cc:348] T 998406d582744483ae82e04cf4be1539 P a390a8c24bbe45baa12c712e9cf3c90d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "36f277a9f15a47bd9843342a5fc170f8" member_type: VOTER last_known_addr { host: "127.0.222.1" port: 41559 } } peers { permanent_uuid: "3842c96efca14e43898885e9664afe98" member_type: VOTER last_known_addr { host: "127.0.222.2" port: 34441 } } peers { permanent_uuid: "a390a8c24bbe45baa12c712e9cf3c90d" member_type: VOTER last_known_addr { host: "127.0.222.3" port: 35491 } }
I20260809 12:09:56.353178 2667 tablet_bootstrap.cc:492] T 998406d582744483ae82e04cf4be1539 P 36f277a9f15a47bd9843342a5fc170f8: No bootstrap required, opened a new log
I20260809 12:09:56.353206 2081 catalog_manager.cc:4841] TS 3842c96efca14e43898885e9664afe98 (127.0.222.2:34441): tablet 0fe5ad0b3883461695535f6e3320b478 (table test-table [id=257842f46bca4f06927027d120e6d811]) successfully deleted
I20260809 12:09:56.353463 2667 ts_tablet_manager.cc:1397] T 998406d582744483ae82e04cf4be1539 P 36f277a9f15a47bd9843342a5fc170f8: Time spent bootstrapping tablet: real 0.043s user 0.012s sys 0.017s
I20260809 12:09:56.353514 2666 raft_consensus.cc:374] T 998406d582744483ae82e04cf4be1539 P a390a8c24bbe45baa12c712e9cf3c90d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260809 12:09:56.353893 2663 ts_tablet_manager.cc:1939] T 0fe5ad0b3883461695535f6e3320b478 P a390a8c24bbe45baa12c712e9cf3c90d: Deleting consensus metadata
I20260809 12:09:56.353935 2666 raft_consensus.cc:729] T 998406d582744483ae82e04cf4be1539 P a390a8c24bbe45baa12c712e9cf3c90d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a390a8c24bbe45baa12c712e9cf3c90d, State: Initialized, Role: FOLLOWER
I20260809 12:09:56.354338 2666 consensus_queue.cc:260] T 998406d582744483ae82e04cf4be1539 P a390a8c24bbe45baa12c712e9cf3c90d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "36f277a9f15a47bd9843342a5fc170f8" member_type: VOTER last_known_addr { host: "127.0.222.1" port: 41559 } } peers { permanent_uuid: "3842c96efca14e43898885e9664afe98" member_type: VOTER last_known_addr { host: "127.0.222.2" port: 34441 } } peers { permanent_uuid: "a390a8c24bbe45baa12c712e9cf3c90d" member_type: VOTER last_known_addr { host: "127.0.222.3" port: 35491 } }
I20260809 12:09:56.354874 2667 raft_consensus.cc:348] T 998406d582744483ae82e04cf4be1539 P 36f277a9f15a47bd9843342a5fc170f8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "36f277a9f15a47bd9843342a5fc170f8" member_type: VOTER last_known_addr { host: "127.0.222.1" port: 41559 } } peers { permanent_uuid: "3842c96efca14e43898885e9664afe98" member_type: VOTER last_known_addr { host: "127.0.222.2" port: 34441 } } peers { permanent_uuid: "a390a8c24bbe45baa12c712e9cf3c90d" member_type: VOTER last_known_addr { host: "127.0.222.3" port: 35491 } }
I20260809 12:09:56.355353 2667 raft_consensus.cc:374] T 998406d582744483ae82e04cf4be1539 P 36f277a9f15a47bd9843342a5fc170f8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260809 12:09:56.356046 2666 ts_tablet_manager.cc:1428] T 998406d582744483ae82e04cf4be1539 P a390a8c24bbe45baa12c712e9cf3c90d: Time spent starting tablet: real 0.006s user 0.004s sys 0.000s
I20260809 12:09:56.356820 2085 catalog_manager.cc:4841] TS a390a8c24bbe45baa12c712e9cf3c90d (127.0.222.3:35491): tablet 0fe5ad0b3883461695535f6e3320b478 (table test-table [id=257842f46bca4f06927027d120e6d811]) successfully deleted
I20260809 12:09:56.357002 2667 raft_consensus.cc:729] T 998406d582744483ae82e04cf4be1539 P 36f277a9f15a47bd9843342a5fc170f8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 36f277a9f15a47bd9843342a5fc170f8, State: Initialized, Role: FOLLOWER
I20260809 12:09:56.357514 2667 consensus_queue.cc:260] T 998406d582744483ae82e04cf4be1539 P 36f277a9f15a47bd9843342a5fc170f8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "36f277a9f15a47bd9843342a5fc170f8" member_type: VOTER last_known_addr { host: "127.0.222.1" port: 41559 } } peers { permanent_uuid: "3842c96efca14e43898885e9664afe98" member_type: VOTER last_known_addr { host: "127.0.222.2" port: 34441 } } peers { permanent_uuid: "a390a8c24bbe45baa12c712e9cf3c90d" member_type: VOTER last_known_addr { host: "127.0.222.3" port: 35491 } }
I20260809 12:09:56.358490 2667 ts_tablet_manager.cc:1428] T 998406d582744483ae82e04cf4be1539 P 36f277a9f15a47bd9843342a5fc170f8: Time spent starting tablet: real 0.005s user 0.004s sys 0.000s
I20260809 12:09:56.438182 2641 raft_consensus.cc:482] T 998406d582744483ae82e04cf4be1539 P a390a8c24bbe45baa12c712e9cf3c90d [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260809 12:09:56.438766 2641 raft_consensus.cc:504] T 998406d582744483ae82e04cf4be1539 P a390a8c24bbe45baa12c712e9cf3c90d [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "36f277a9f15a47bd9843342a5fc170f8" member_type: VOTER last_known_addr { host: "127.0.222.1" port: 41559 } } peers { permanent_uuid: "3842c96efca14e43898885e9664afe98" member_type: VOTER last_known_addr { host: "127.0.222.2" port: 34441 } } peers { permanent_uuid: "a390a8c24bbe45baa12c712e9cf3c90d" member_type: VOTER last_known_addr { host: "127.0.222.3" port: 35491 } }
I20260809 12:09:56.442284 2641 leader_election.cc:288] T 998406d582744483ae82e04cf4be1539 P a390a8c24bbe45baa12c712e9cf3c90d [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 36f277a9f15a47bd9843342a5fc170f8 (127.0.222.1:41559), 3842c96efca14e43898885e9664afe98 (127.0.222.2:34441)
I20260809 12:09:56.459312 2629 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 12:09:56.442821 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.222.3:46805 (local address 127.0.222.2:34441)
0809 12:09:56.444315 (+ 1494us) server_negotiation.cc:206] Beginning negotiation
0809 12:09:56.444325 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0809 12:09:56.447735 (+ 3410us) server_negotiation.cc:407] Connection header received
0809 12:09:56.448026 (+ 291us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 12:09:56.448036 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 12:09:56.448145 (+ 109us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 12:09:56.448272 (+ 127us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0809 12:09:56.450093 (+ 1821us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 12:09:56.451512 (+ 1419us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 12:09:56.455913 (+ 4401us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 12:09:56.457674 (+ 1761us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 12:09:56.457855 (+ 181us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 12:09:56.457992 (+ 137us) server_negotiation.cc:1035] Waiting for connection context
0809 12:09:56.458545 (+ 553us) server_negotiation.cc:299] Negotiation successful
0809 12:09:56.458706 (+ 161us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":75}
I20260809 12:09:56.459889 2671 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 12:09:56.442514 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.222.2:34441 (local address 127.0.222.3:46805)
0809 12:09:56.447570 (+ 5056us) negotiation.cc:107] Waiting for socket to connect
0809 12:09:56.447592 (+ 22us) client_negotiation.cc:174] Beginning negotiation
0809 12:09:56.447847 (+ 255us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 12:09:56.448573 (+ 726us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 12:09:56.448579 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 12:09:56.448610 (+ 31us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0809 12:09:56.449769 (+ 1159us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 12:09:56.449785 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 12:09:56.451658 (+ 1873us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 12:09:56.451664 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 12:09:56.455689 (+ 4025us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 12:09:56.455708 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 12:09:56.458020 (+ 2312us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 12:09:56.458029 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 12:09:56.458308 (+ 279us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 12:09:56.458359 (+ 51us) client_negotiation.cc:769] Sending connection context
0809 12:09:56.458488 (+ 129us) client_negotiation.cc:240] Negotiation successful
0809 12:09:56.458661 (+ 173us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":4782,"mutex_wait_us":1,"thread_start_us":4459,"threads_started":1}
I20260809 12:09:56.461014 2413 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "998406d582744483ae82e04cf4be1539" candidate_uuid: "a390a8c24bbe45baa12c712e9cf3c90d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3842c96efca14e43898885e9664afe98" is_pre_election: true
I20260809 12:09:56.461560 2672 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 12:09:56.443721 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.222.1:41559 (local address 127.0.222.3:51949)
0809 12:09:56.449188 (+ 5467us) negotiation.cc:107] Waiting for socket to connect
0809 12:09:56.449211 (+ 23us) client_negotiation.cc:174] Beginning negotiation
0809 12:09:56.449553 (+ 342us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 12:09:56.450097 (+ 544us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 12:09:56.450103 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 12:09:56.450149 (+ 46us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0809 12:09:56.451626 (+ 1477us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 12:09:56.451642 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 12:09:56.454014 (+ 2372us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 12:09:56.454020 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 12:09:56.457747 (+ 3727us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 12:09:56.457766 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 12:09:56.460350 (+ 2584us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 12:09:56.460357 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 12:09:56.460536 (+ 179us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 12:09:56.460559 (+ 23us) client_negotiation.cc:769] Sending connection context
0809 12:09:56.460658 (+ 99us) client_negotiation.cc:240] Negotiation successful
0809 12:09:56.460810 (+ 152us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":5238,"mutex_wait_us":55,"thread_start_us":4813,"threads_started":1}
I20260809 12:09:56.461512 2413 raft_consensus.cc:2454] T 998406d582744483ae82e04cf4be1539 P 3842c96efca14e43898885e9664afe98 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a390a8c24bbe45baa12c712e9cf3c90d in term 0.
I20260809 12:09:56.462342 2628 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 12:09:56.443063 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.222.3:51949 (local address 127.0.222.1:41559)
0809 12:09:56.443552 (+ 489us) server_negotiation.cc:206] Beginning negotiation
0809 12:09:56.443570 (+ 18us) server_negotiation.cc:399] Waiting for connection header
0809 12:09:56.449391 (+ 5821us) server_negotiation.cc:407] Connection header received
0809 12:09:56.449794 (+ 403us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 12:09:56.449800 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 12:09:56.449928 (+ 128us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 12:09:56.450036 (+ 108us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0809 12:09:56.451882 (+ 1846us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 12:09:56.453867 (+ 1985us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 12:09:56.457986 (+ 4119us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 12:09:56.459769 (+ 1783us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 12:09:56.459855 (+ 86us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 12:09:56.459925 (+ 70us) server_negotiation.cc:1035] Waiting for connection context
0809 12:09:56.461509 (+ 1584us) server_negotiation.cc:299] Negotiation successful
0809 12:09:56.461819 (+ 310us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":255}
I20260809 12:09:56.462275 2486 leader_election.cc:302] T 998406d582744483ae82e04cf4be1539 P a390a8c24bbe45baa12c712e9cf3c90d [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3842c96efca14e43898885e9664afe98, a390a8c24bbe45baa12c712e9cf3c90d; no voters:
I20260809 12:09:56.462951 2641 raft_consensus.cc:2781] T 998406d582744483ae82e04cf4be1539 P a390a8c24bbe45baa12c712e9cf3c90d [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260809 12:09:56.463279 2641 raft_consensus.cc:482] T 998406d582744483ae82e04cf4be1539 P a390a8c24bbe45baa12c712e9cf3c90d [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260809 12:09:56.463507 2641 raft_consensus.cc:3037] T 998406d582744483ae82e04cf4be1539 P a390a8c24bbe45baa12c712e9cf3c90d [term 0 FOLLOWER]: Advancing to term 1
I20260809 12:09:56.463615 2270 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "998406d582744483ae82e04cf4be1539" candidate_uuid: "a390a8c24bbe45baa12c712e9cf3c90d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "36f277a9f15a47bd9843342a5fc170f8" is_pre_election: true
I20260809 12:09:56.464067 2270 raft_consensus.cc:2454] T 998406d582744483ae82e04cf4be1539 P 36f277a9f15a47bd9843342a5fc170f8 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a390a8c24bbe45baa12c712e9cf3c90d in term 0.
I20260809 12:09:56.467653 2641 raft_consensus.cc:504] T 998406d582744483ae82e04cf4be1539 P a390a8c24bbe45baa12c712e9cf3c90d [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "36f277a9f15a47bd9843342a5fc170f8" member_type: VOTER last_known_addr { host: "127.0.222.1" port: 41559 } } peers { permanent_uuid: "3842c96efca14e43898885e9664afe98" member_type: VOTER last_known_addr { host: "127.0.222.2" port: 34441 } } peers { permanent_uuid: "a390a8c24bbe45baa12c712e9cf3c90d" member_type: VOTER last_known_addr { host: "127.0.222.3" port: 35491 } }
I20260809 12:09:56.468801 2641 leader_election.cc:288] T 998406d582744483ae82e04cf4be1539 P a390a8c24bbe45baa12c712e9cf3c90d [CANDIDATE]: Term 1 election: Requested vote from peers 36f277a9f15a47bd9843342a5fc170f8 (127.0.222.1:41559), 3842c96efca14e43898885e9664afe98 (127.0.222.2:34441)
I20260809 12:09:56.469151 2270 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "998406d582744483ae82e04cf4be1539" candidate_uuid: "a390a8c24bbe45baa12c712e9cf3c90d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "36f277a9f15a47bd9843342a5fc170f8"
I20260809 12:09:56.469245 2413 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "998406d582744483ae82e04cf4be1539" candidate_uuid: "a390a8c24bbe45baa12c712e9cf3c90d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3842c96efca14e43898885e9664afe98"
I20260809 12:09:56.469496 2270 raft_consensus.cc:3037] T 998406d582744483ae82e04cf4be1539 P 36f277a9f15a47bd9843342a5fc170f8 [term 0 FOLLOWER]: Advancing to term 1
I20260809 12:09:56.469661 2413 raft_consensus.cc:3037] T 998406d582744483ae82e04cf4be1539 P 3842c96efca14e43898885e9664afe98 [term 0 FOLLOWER]: Advancing to term 1
I20260809 12:09:56.472370 2413 raft_consensus.cc:2454] T 998406d582744483ae82e04cf4be1539 P 3842c96efca14e43898885e9664afe98 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a390a8c24bbe45baa12c712e9cf3c90d in term 1.
I20260809 12:09:56.472890 2486 leader_election.cc:302] T 998406d582744483ae82e04cf4be1539 P a390a8c24bbe45baa12c712e9cf3c90d [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3842c96efca14e43898885e9664afe98, a390a8c24bbe45baa12c712e9cf3c90d; no voters:
I20260809 12:09:56.473318 2641 raft_consensus.cc:2781] T 998406d582744483ae82e04cf4be1539 P a390a8c24bbe45baa12c712e9cf3c90d [term 1 FOLLOWER]: Leader election won for term 1
I20260809 12:09:56.474572 2641 raft_consensus.cc:686] T 998406d582744483ae82e04cf4be1539 P a390a8c24bbe45baa12c712e9cf3c90d [term 1 LEADER]: Becoming Leader. State: Replica: a390a8c24bbe45baa12c712e9cf3c90d, State: Running, Role: LEADER
I20260809 12:09:56.475229 2270 raft_consensus.cc:2454] T 998406d582744483ae82e04cf4be1539 P 36f277a9f15a47bd9843342a5fc170f8 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a390a8c24bbe45baa12c712e9cf3c90d in term 1.
I20260809 12:09:56.475548 2641 consensus_queue.cc:237] T 998406d582744483ae82e04cf4be1539 P a390a8c24bbe45baa12c712e9cf3c90d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "36f277a9f15a47bd9843342a5fc170f8" member_type: VOTER last_known_addr { host: "127.0.222.1" port: 41559 } } peers { permanent_uuid: "3842c96efca14e43898885e9664afe98" member_type: VOTER last_known_addr { host: "127.0.222.2" port: 34441 } } peers { permanent_uuid: "a390a8c24bbe45baa12c712e9cf3c90d" member_type: VOTER last_known_addr { host: "127.0.222.3" port: 35491 } }
I20260809 12:09:56.481215 2098 catalog_manager.cc:5495] T 998406d582744483ae82e04cf4be1539 P a390a8c24bbe45baa12c712e9cf3c90d reported cstate change: term changed from 0 to 1, leader changed from <none> to a390a8c24bbe45baa12c712e9cf3c90d (127.0.222.3). New cstate: current_term: 1 leader_uuid: "a390a8c24bbe45baa12c712e9cf3c90d" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "36f277a9f15a47bd9843342a5fc170f8" member_type: VOTER last_known_addr { host: "127.0.222.1" port: 41559 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "3842c96efca14e43898885e9664afe98" member_type: VOTER last_known_addr { host: "127.0.222.2" port: 34441 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "a390a8c24bbe45baa12c712e9cf3c90d" member_type: VOTER last_known_addr { host: "127.0.222.3" port: 35491 } health_report { overall_health: HEALTHY } } }
Aug 09 12:09:56 dist-test-slave-2w3w krb5kdc[2051](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786277393, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.222.62@KRBTEST.COM for kudu/127.0.222.62@KRBTEST.COM
I20260809 12:09:56.554392 2680 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 12:09:56.522809 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49870 (local address 127.0.222.62:42617)
0809 12:09:56.524761 (+ 1952us) server_negotiation.cc:206] Beginning negotiation
0809 12:09:56.524770 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0809 12:09:56.525664 (+ 894us) server_negotiation.cc:407] Connection header received
0809 12:09:56.525922 (+ 258us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 12:09:56.525926 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 12:09:56.526027 (+ 101us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 12:09:56.526138 (+ 111us) server_negotiation.cc:226] Negotiated authn=SASL
0809 12:09:56.528912 (+ 2774us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 12:09:56.531085 (+ 2173us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 12:09:56.534249 (+ 3164us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 12:09:56.535506 (+ 1257us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 12:09:56.535628 (+ 122us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 12:09:56.542510 (+ 6882us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 12:09:56.542520 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 12:09:56.542526 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0809 12:09:56.542567 (+ 41us) server_negotiation.cc:922] calling sasl_server_start()
0809 12:09:56.549100 (+ 6533us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 12:09:56.550661 (+ 1561us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 12:09:56.550667 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 12:09:56.550671 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0809 12:09:56.551113 (+ 442us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 12:09:56.551745 (+ 632us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 12:09:56.551749 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 12:09:56.551752 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0809 12:09:56.552233 (+ 481us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 12:09:56.552411 (+ 178us) server_negotiation.cc:1035] Waiting for connection context
0809 12:09:56.553251 (+ 840us) server_negotiation.cc:299] Negotiation successful
0809 12:09:56.553895 (+ 644us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":342,"server-negotiator.queue_time_us":1682,"thread_start_us":1337,"threads_started":1}
I20260809 12:09:56.555233 2681 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 12:09:56.523142 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.222.62:42617 (local address 127.0.0.1:49870)
0809 12:09:56.525527 (+ 2385us) negotiation.cc:107] Waiting for socket to connect
0809 12:09:56.525550 (+ 23us) client_negotiation.cc:174] Beginning negotiation
0809 12:09:56.525778 (+ 228us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 12:09:56.526214 (+ 436us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 12:09:56.526219 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 12:09:56.527843 (+ 1624us) client_negotiation.cc:189] Negotiated authn=SASL
0809 12:09:56.528651 (+ 808us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 12:09:56.528667 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 12:09:56.531294 (+ 2627us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 12:09:56.531300 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 12:09:56.534049 (+ 2749us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 12:09:56.534067 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 12:09:56.535842 (+ 1775us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 12:09:56.535852 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 12:09:56.536161 (+ 309us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 12:09:56.537293 (+ 1132us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0809 12:09:56.537318 (+ 25us) client_negotiation.cc:656] calling sasl_client_start()
0809 12:09:56.541986 (+ 4668us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0809 12:09:56.549348 (+ 7362us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0809 12:09:56.549365 (+ 17us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0809 12:09:56.549373 (+ 8us) client_negotiation.cc:760] calling sasl_client_step()
0809 12:09:56.550488 (+ 1115us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0809 12:09:56.551282 (+ 794us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0809 12:09:56.551288 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0809 12:09:56.551291 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0809 12:09:56.551569 (+ 278us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0809 12:09:56.552487 (+ 918us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0809 12:09:56.552495 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0809 12:09:56.552695 (+ 200us) client_negotiation.cc:769] Sending connection context
0809 12:09:56.553117 (+ 422us) client_negotiation.cc:240] Negotiation successful
0809 12:09:56.553694 (+ 577us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":2146,"thread_start_us":1839,"threads_started":1}
I20260809 12:09:56.564890 2098 catalog_manager.cc:1908] Servicing CreateTable request from {username='kudu', principal='kudu/127.0.222.62@KRBTEST.COM'} at 127.0.0.1:49870:
name: "kudu_system.kudu_transactions"
schema {
columns {
name: "txn_id"
type: INT64
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "entry_type"
type: INT8
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "identifier"
type: STRING
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "metadata"
type: STRING
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
rows: "<redacted>""\006\001\000\000\000\000\000\000\000\000\007\001@B\017\000\000\000\000\000"
indirect_data: "<redacted>"""
}
partition_schema {
range_schema {
columns {
name: "txn_id"
}
}
}
table_type: TXN_STATUS_TABLE
W20260809 12:09:56.566360 2098 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table kudu_system.kudu_transactions in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260809 12:09:56.574131 2244 tablet_service.cc:1467] Processing CreateTablet for tablet 21da537165d245519766d5b22458a75b (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=d2a4e5904f884d549a7c991147c9af75]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260809 12:09:56.574005 2391 tablet_service.cc:1467] Processing CreateTablet for tablet 21da537165d245519766d5b22458a75b (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=d2a4e5904f884d549a7c991147c9af75]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260809 12:09:56.574167 2538 tablet_service.cc:1467] Processing CreateTablet for tablet 21da537165d245519766d5b22458a75b (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=d2a4e5904f884d549a7c991147c9af75]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260809 12:09:56.574730 2538 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 21da537165d245519766d5b22458a75b. 1 dirs total, 0 dirs full, 0 dirs failed
I20260809 12:09:56.574779 2391 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 21da537165d245519766d5b22458a75b. 1 dirs total, 0 dirs full, 0 dirs failed
I20260809 12:09:56.574898 2244 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 21da537165d245519766d5b22458a75b. 1 dirs total, 0 dirs full, 0 dirs failed
I20260809 12:09:56.585587 2666 tablet_bootstrap.cc:492] T 21da537165d245519766d5b22458a75b P a390a8c24bbe45baa12c712e9cf3c90d: Bootstrap starting.
I20260809 12:09:56.587314 2665 tablet_bootstrap.cc:492] T 21da537165d245519766d5b22458a75b P 3842c96efca14e43898885e9664afe98: Bootstrap starting.
I20260809 12:09:56.589012 2666 tablet_bootstrap.cc:654] T 21da537165d245519766d5b22458a75b P a390a8c24bbe45baa12c712e9cf3c90d: Neither blocks nor log segments found. Creating new log.
I20260809 12:09:56.590314 2665 tablet_bootstrap.cc:654] T 21da537165d245519766d5b22458a75b P 3842c96efca14e43898885e9664afe98: Neither blocks nor log segments found. Creating new log.
I20260809 12:09:56.592504 2667 tablet_bootstrap.cc:492] T 21da537165d245519766d5b22458a75b P 36f277a9f15a47bd9843342a5fc170f8: Bootstrap starting.
I20260809 12:09:56.593354 2666 tablet_bootstrap.cc:492] T 21da537165d245519766d5b22458a75b P a390a8c24bbe45baa12c712e9cf3c90d: No bootstrap required, opened a new log
I20260809 12:09:56.594056 2666 ts_tablet_manager.cc:1397] T 21da537165d245519766d5b22458a75b P a390a8c24bbe45baa12c712e9cf3c90d: Time spent bootstrapping tablet: real 0.009s user 0.001s sys 0.006s
I20260809 12:09:56.595685 2666 raft_consensus.cc:348] T 21da537165d245519766d5b22458a75b P a390a8c24bbe45baa12c712e9cf3c90d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "36f277a9f15a47bd9843342a5fc170f8" member_type: VOTER last_known_addr { host: "127.0.222.1" port: 41559 } } peers { permanent_uuid: "3842c96efca14e43898885e9664afe98" member_type: VOTER last_known_addr { host: "127.0.222.2" port: 34441 } } peers { permanent_uuid: "a390a8c24bbe45baa12c712e9cf3c90d" member_type: VOTER last_known_addr { host: "127.0.222.3" port: 35491 } }
I20260809 12:09:56.596247 2666 raft_consensus.cc:374] T 21da537165d245519766d5b22458a75b P a390a8c24bbe45baa12c712e9cf3c90d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260809 12:09:56.596524 2667 tablet_bootstrap.cc:654] T 21da537165d245519766d5b22458a75b P 36f277a9f15a47bd9843342a5fc170f8: Neither blocks nor log segments found. Creating new log.
I20260809 12:09:56.596454 2666 raft_consensus.cc:729] T 21da537165d245519766d5b22458a75b P a390a8c24bbe45baa12c712e9cf3c90d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a390a8c24bbe45baa12c712e9cf3c90d, State: Initialized, Role: FOLLOWER
I20260809 12:09:56.597051 2666 consensus_queue.cc:260] T 21da537165d245519766d5b22458a75b P a390a8c24bbe45baa12c712e9cf3c90d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "36f277a9f15a47bd9843342a5fc170f8" member_type: VOTER last_known_addr { host: "127.0.222.1" port: 41559 } } peers { permanent_uuid: "3842c96efca14e43898885e9664afe98" member_type: VOTER last_known_addr { host: "127.0.222.2" port: 34441 } } peers { permanent_uuid: "a390a8c24bbe45baa12c712e9cf3c90d" member_type: VOTER last_known_addr { host: "127.0.222.3" port: 35491 } }
I20260809 12:09:56.599277 2641 tablet_replica.cc:439] T 21da537165d245519766d5b22458a75b P a390a8c24bbe45baa12c712e9cf3c90d: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "36f277a9f15a47bd9843342a5fc170f8" member_type: VOTER last_known_addr { host: "127.0.222.1" port: 41559 } } peers { permanent_uuid: "3842c96efca14e43898885e9664afe98" member_type: VOTER last_known_addr { host: "127.0.222.2" port: 34441 } } peers { permanent_uuid: "a390a8c24bbe45baa12c712e9cf3c90d" member_type: VOTER last_known_addr { host: "127.0.222.3" port: 35491 } } }
I20260809 12:09:56.599629 2666 ts_tablet_manager.cc:1428] T 21da537165d245519766d5b22458a75b P a390a8c24bbe45baa12c712e9cf3c90d: Time spent starting tablet: real 0.005s user 0.004s sys 0.000s
I20260809 12:09:56.601683 2641 tablet_replica.cc:442] T 21da537165d245519766d5b22458a75b P a390a8c24bbe45baa12c712e9cf3c90d: This TxnStatusTablet replica's current role is: FOLLOWER
I20260809 12:09:56.602468 2665 tablet_bootstrap.cc:492] T 21da537165d245519766d5b22458a75b P 3842c96efca14e43898885e9664afe98: No bootstrap required, opened a new log
I20260809 12:09:56.602664 2665 ts_tablet_manager.cc:1397] T 21da537165d245519766d5b22458a75b P 3842c96efca14e43898885e9664afe98: Time spent bootstrapping tablet: real 0.016s user 0.002s sys 0.008s
I20260809 12:09:56.603904 2665 raft_consensus.cc:348] T 21da537165d245519766d5b22458a75b P 3842c96efca14e43898885e9664afe98 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "36f277a9f15a47bd9843342a5fc170f8" member_type: VOTER last_known_addr { host: "127.0.222.1" port: 41559 } } peers { permanent_uuid: "3842c96efca14e43898885e9664afe98" member_type: VOTER last_known_addr { host: "127.0.222.2" port: 34441 } } peers { permanent_uuid: "a390a8c24bbe45baa12c712e9cf3c90d" member_type: VOTER last_known_addr { host: "127.0.222.3" port: 35491 } }
I20260809 12:09:56.604391 2665 raft_consensus.cc:374] T 21da537165d245519766d5b22458a75b P 3842c96efca14e43898885e9664afe98 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260809 12:09:56.604581 2665 raft_consensus.cc:729] T 21da537165d245519766d5b22458a75b P 3842c96efca14e43898885e9664afe98 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3842c96efca14e43898885e9664afe98, State: Initialized, Role: FOLLOWER
I20260809 12:09:56.604583 2667 tablet_bootstrap.cc:492] T 21da537165d245519766d5b22458a75b P 36f277a9f15a47bd9843342a5fc170f8: No bootstrap required, opened a new log
I20260809 12:09:56.605192 2667 ts_tablet_manager.cc:1397] T 21da537165d245519766d5b22458a75b P 36f277a9f15a47bd9843342a5fc170f8: Time spent bootstrapping tablet: real 0.013s user 0.001s sys 0.006s
I20260809 12:09:56.605192 2665 consensus_queue.cc:260] T 21da537165d245519766d5b22458a75b P 3842c96efca14e43898885e9664afe98 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "36f277a9f15a47bd9843342a5fc170f8" member_type: VOTER last_known_addr { host: "127.0.222.1" port: 41559 } } peers { permanent_uuid: "3842c96efca14e43898885e9664afe98" member_type: VOTER last_known_addr { host: "127.0.222.2" port: 34441 } } peers { permanent_uuid: "a390a8c24bbe45baa12c712e9cf3c90d" member_type: VOTER last_known_addr { host: "127.0.222.3" port: 35491 } }
I20260809 12:09:56.606040 2640 tablet_replica.cc:439] T 21da537165d245519766d5b22458a75b P 3842c96efca14e43898885e9664afe98: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "36f277a9f15a47bd9843342a5fc170f8" member_type: VOTER last_known_addr { host: "127.0.222.1" port: 41559 } } peers { permanent_uuid: "3842c96efca14e43898885e9664afe98" member_type: VOTER last_known_addr { host: "127.0.222.2" port: 34441 } } peers { permanent_uuid: "a390a8c24bbe45baa12c712e9cf3c90d" member_type: VOTER last_known_addr { host: "127.0.222.3" port: 35491 } } }
I20260809 12:09:56.606643 2640 tablet_replica.cc:442] T 21da537165d245519766d5b22458a75b P 3842c96efca14e43898885e9664afe98: This TxnStatusTablet replica's current role is: FOLLOWER
I20260809 12:09:56.606585 2667 raft_consensus.cc:348] T 21da537165d245519766d5b22458a75b P 36f277a9f15a47bd9843342a5fc170f8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "36f277a9f15a47bd9843342a5fc170f8" member_type: VOTER last_known_addr { host: "127.0.222.1" port: 41559 } } peers { permanent_uuid: "3842c96efca14e43898885e9664afe98" member_type: VOTER last_known_addr { host: "127.0.222.2" port: 34441 } } peers { permanent_uuid: "a390a8c24bbe45baa12c712e9cf3c90d" member_type: VOTER last_known_addr { host: "127.0.222.3" port: 35491 } }
I20260809 12:09:56.607071 2667 raft_consensus.cc:374] T 21da537165d245519766d5b22458a75b P 36f277a9f15a47bd9843342a5fc170f8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260809 12:09:56.607244 2667 raft_consensus.cc:729] T 21da537165d245519766d5b22458a75b P 36f277a9f15a47bd9843342a5fc170f8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 36f277a9f15a47bd9843342a5fc170f8, State: Initialized, Role: FOLLOWER
I20260809 12:09:56.607728 2667 consensus_queue.cc:260] T 21da537165d245519766d5b22458a75b P 36f277a9f15a47bd9843342a5fc170f8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "36f277a9f15a47bd9843342a5fc170f8" member_type: VOTER last_known_addr { host: "127.0.222.1" port: 41559 } } peers { permanent_uuid: "3842c96efca14e43898885e9664afe98" member_type: VOTER last_known_addr { host: "127.0.222.2" port: 34441 } } peers { permanent_uuid: "a390a8c24bbe45baa12c712e9cf3c90d" member_type: VOTER last_known_addr { host: "127.0.222.3" port: 35491 } }
I20260809 12:09:56.608482 2659 tablet_replica.cc:439] T 21da537165d245519766d5b22458a75b P 36f277a9f15a47bd9843342a5fc170f8: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "36f277a9f15a47bd9843342a5fc170f8" member_type: VOTER last_known_addr { host: "127.0.222.1" port: 41559 } } peers { permanent_uuid: "3842c96efca14e43898885e9664afe98" member_type: VOTER last_known_addr { host: "127.0.222.2" port: 34441 } } peers { permanent_uuid: "a390a8c24bbe45baa12c712e9cf3c90d" member_type: VOTER last_known_addr { host: "127.0.222.3" port: 35491 } } }
I20260809 12:09:56.609166 2659 tablet_replica.cc:442] T 21da537165d245519766d5b22458a75b P 36f277a9f15a47bd9843342a5fc170f8: This TxnStatusTablet replica's current role is: FOLLOWER
I20260809 12:09:56.610498 2667 ts_tablet_manager.cc:1428] T 21da537165d245519766d5b22458a75b P 36f277a9f15a47bd9843342a5fc170f8: Time spent starting tablet: real 0.005s user 0.000s sys 0.003s
I20260809 12:09:56.611083 2665 ts_tablet_manager.cc:1428] T 21da537165d245519766d5b22458a75b P 3842c96efca14e43898885e9664afe98: Time spent starting tablet: real 0.008s user 0.003s sys 0.000s
I20260809 12:09:56.864693 2659 raft_consensus.cc:482] T 21da537165d245519766d5b22458a75b P 36f277a9f15a47bd9843342a5fc170f8 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260809 12:09:56.865109 2659 raft_consensus.cc:504] T 21da537165d245519766d5b22458a75b P 36f277a9f15a47bd9843342a5fc170f8 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "36f277a9f15a47bd9843342a5fc170f8" member_type: VOTER last_known_addr { host: "127.0.222.1" port: 41559 } } peers { permanent_uuid: "3842c96efca14e43898885e9664afe98" member_type: VOTER last_known_addr { host: "127.0.222.2" port: 34441 } } peers { permanent_uuid: "a390a8c24bbe45baa12c712e9cf3c90d" member_type: VOTER last_known_addr { host: "127.0.222.3" port: 35491 } }
I20260809 12:09:56.866029 2659 leader_election.cc:288] T 21da537165d245519766d5b22458a75b P 36f277a9f15a47bd9843342a5fc170f8 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 3842c96efca14e43898885e9664afe98 (127.0.222.2:34441), a390a8c24bbe45baa12c712e9cf3c90d (127.0.222.3:35491)
I20260809 12:09:56.866714 2413 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "21da537165d245519766d5b22458a75b" candidate_uuid: "36f277a9f15a47bd9843342a5fc170f8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3842c96efca14e43898885e9664afe98" is_pre_election: true
I20260809 12:09:56.866719 2566 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "21da537165d245519766d5b22458a75b" candidate_uuid: "36f277a9f15a47bd9843342a5fc170f8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a390a8c24bbe45baa12c712e9cf3c90d" is_pre_election: true
I20260809 12:09:56.867102 2413 raft_consensus.cc:2454] T 21da537165d245519766d5b22458a75b P 3842c96efca14e43898885e9664afe98 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 36f277a9f15a47bd9843342a5fc170f8 in term 0.
I20260809 12:09:56.867120 2566 raft_consensus.cc:2454] T 21da537165d245519766d5b22458a75b P a390a8c24bbe45baa12c712e9cf3c90d [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 36f277a9f15a47bd9843342a5fc170f8 in term 0.
I20260809 12:09:56.867638 2189 leader_election.cc:302] T 21da537165d245519766d5b22458a75b P 36f277a9f15a47bd9843342a5fc170f8 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 3 yes votes; 0 no votes. yes voters: 36f277a9f15a47bd9843342a5fc170f8, 3842c96efca14e43898885e9664afe98, a390a8c24bbe45baa12c712e9cf3c90d; no voters:
I20260809 12:09:56.867987 2659 raft_consensus.cc:2781] T 21da537165d245519766d5b22458a75b P 36f277a9f15a47bd9843342a5fc170f8 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260809 12:09:56.868129 2659 raft_consensus.cc:482] T 21da537165d245519766d5b22458a75b P 36f277a9f15a47bd9843342a5fc170f8 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260809 12:09:56.868376 2659 raft_consensus.cc:3037] T 21da537165d245519766d5b22458a75b P 36f277a9f15a47bd9843342a5fc170f8 [term 0 FOLLOWER]: Advancing to term 1
I20260809 12:09:56.872613 2659 raft_consensus.cc:504] T 21da537165d245519766d5b22458a75b P 36f277a9f15a47bd9843342a5fc170f8 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "36f277a9f15a47bd9843342a5fc170f8" member_type: VOTER last_known_addr { host: "127.0.222.1" port: 41559 } } peers { permanent_uuid: "3842c96efca14e43898885e9664afe98" member_type: VOTER last_known_addr { host: "127.0.222.2" port: 34441 } } peers { permanent_uuid: "a390a8c24bbe45baa12c712e9cf3c90d" member_type: VOTER last_known_addr { host: "127.0.222.3" port: 35491 } }
I20260809 12:09:56.873500 2659 leader_election.cc:288] T 21da537165d245519766d5b22458a75b P 36f277a9f15a47bd9843342a5fc170f8 [CANDIDATE]: Term 1 election: Requested vote from peers 3842c96efca14e43898885e9664afe98 (127.0.222.2:34441), a390a8c24bbe45baa12c712e9cf3c90d (127.0.222.3:35491)
I20260809 12:09:56.874047 2413 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "21da537165d245519766d5b22458a75b" candidate_uuid: "36f277a9f15a47bd9843342a5fc170f8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3842c96efca14e43898885e9664afe98"
I20260809 12:09:56.874353 2413 raft_consensus.cc:3037] T 21da537165d245519766d5b22458a75b P 3842c96efca14e43898885e9664afe98 [term 0 FOLLOWER]: Advancing to term 1
I20260809 12:09:56.874286 2566 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "21da537165d245519766d5b22458a75b" candidate_uuid: "36f277a9f15a47bd9843342a5fc170f8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a390a8c24bbe45baa12c712e9cf3c90d"
I20260809 12:09:56.874619 2566 raft_consensus.cc:3037] T 21da537165d245519766d5b22458a75b P a390a8c24bbe45baa12c712e9cf3c90d [term 0 FOLLOWER]: Advancing to term 1
I20260809 12:09:56.877660 2413 raft_consensus.cc:2454] T 21da537165d245519766d5b22458a75b P 3842c96efca14e43898885e9664afe98 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 36f277a9f15a47bd9843342a5fc170f8 in term 1.
I20260809 12:09:56.878144 2189 leader_election.cc:302] T 21da537165d245519766d5b22458a75b P 36f277a9f15a47bd9843342a5fc170f8 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 36f277a9f15a47bd9843342a5fc170f8, 3842c96efca14e43898885e9664afe98; no voters:
I20260809 12:09:56.878288 2566 raft_consensus.cc:2454] T 21da537165d245519766d5b22458a75b P a390a8c24bbe45baa12c712e9cf3c90d [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 36f277a9f15a47bd9843342a5fc170f8 in term 1.
I20260809 12:09:56.878552 2659 raft_consensus.cc:2781] T 21da537165d245519766d5b22458a75b P 36f277a9f15a47bd9843342a5fc170f8 [term 1 FOLLOWER]: Leader election won for term 1
I20260809 12:09:56.879987 2659 raft_consensus.cc:686] T 21da537165d245519766d5b22458a75b P 36f277a9f15a47bd9843342a5fc170f8 [term 1 LEADER]: Becoming Leader. State: Replica: 36f277a9f15a47bd9843342a5fc170f8, State: Running, Role: LEADER
I20260809 12:09:56.880342 2659 consensus_queue.cc:237] T 21da537165d245519766d5b22458a75b P 36f277a9f15a47bd9843342a5fc170f8 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "36f277a9f15a47bd9843342a5fc170f8" member_type: VOTER last_known_addr { host: "127.0.222.1" port: 41559 } } peers { permanent_uuid: "3842c96efca14e43898885e9664afe98" member_type: VOTER last_known_addr { host: "127.0.222.2" port: 34441 } } peers { permanent_uuid: "a390a8c24bbe45baa12c712e9cf3c90d" member_type: VOTER last_known_addr { host: "127.0.222.3" port: 35491 } }
I20260809 12:09:56.883912 2685 tablet_replica.cc:439] T 21da537165d245519766d5b22458a75b P 36f277a9f15a47bd9843342a5fc170f8: TxnStatusTablet state changed. Reason: New leader 36f277a9f15a47bd9843342a5fc170f8. Latest consensus state: current_term: 1 leader_uuid: "36f277a9f15a47bd9843342a5fc170f8" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "36f277a9f15a47bd9843342a5fc170f8" member_type: VOTER last_known_addr { host: "127.0.222.1" port: 41559 } } peers { permanent_uuid: "3842c96efca14e43898885e9664afe98" member_type: VOTER last_known_addr { host: "127.0.222.2" port: 34441 } } peers { permanent_uuid: "a390a8c24bbe45baa12c712e9cf3c90d" member_type: VOTER last_known_addr { host: "127.0.222.3" port: 35491 } } }
I20260809 12:09:56.884387 2685 tablet_replica.cc:442] T 21da537165d245519766d5b22458a75b P 36f277a9f15a47bd9843342a5fc170f8: This TxnStatusTablet replica's current role is: LEADER
I20260809 12:09:56.886183 2687 txn_status_manager.cc:873] Waiting until node catch up with all replicated operations in previous term...
I20260809 12:09:56.886138 2099 catalog_manager.cc:5495] T 21da537165d245519766d5b22458a75b P 36f277a9f15a47bd9843342a5fc170f8 reported cstate change: term changed from 0 to 1, leader changed from <none> to 36f277a9f15a47bd9843342a5fc170f8 (127.0.222.1). New cstate: current_term: 1 leader_uuid: "36f277a9f15a47bd9843342a5fc170f8" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "36f277a9f15a47bd9843342a5fc170f8" member_type: VOTER last_known_addr { host: "127.0.222.1" port: 41559 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "3842c96efca14e43898885e9664afe98" member_type: VOTER last_known_addr { host: "127.0.222.2" port: 34441 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "a390a8c24bbe45baa12c712e9cf3c90d" member_type: VOTER last_known_addr { host: "127.0.222.3" port: 35491 } health_report { overall_health: UNKNOWN } } }
I20260809 12:09:56.886608 2687 txn_status_manager.cc:929] Loading transaction status metadata into memory...
I20260809 12:09:56.974859 2681 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 12:09:56.967503 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.222.1:41559 (local address 127.0.0.1:51684)
0809 12:09:56.967863 (+ 360us) negotiation.cc:107] Waiting for socket to connect
0809 12:09:56.967887 (+ 24us) client_negotiation.cc:174] Beginning negotiation
0809 12:09:56.968101 (+ 214us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 12:09:56.968775 (+ 674us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 12:09:56.968779 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 12:09:56.968792 (+ 13us) client_negotiation.cc:189] Negotiated authn=TOKEN
0809 12:09:56.969143 (+ 351us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 12:09:56.969153 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 12:09:56.970429 (+ 1276us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 12:09:56.970435 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 12:09:56.971994 (+ 1559us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 12:09:56.972008 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 12:09:56.973190 (+ 1182us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 12:09:56.973199 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 12:09:56.973437 (+ 238us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 12:09:56.973500 (+ 63us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0809 12:09:56.974070 (+ 570us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0809 12:09:56.974142 (+ 72us) client_negotiation.cc:769] Sending connection context
0809 12:09:56.974236 (+ 94us) client_negotiation.cc:240] Negotiation successful
0809 12:09:56.974409 (+ 173us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":49}
I20260809 12:09:56.975006 2689 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 12:09:56.967521 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51684 (local address 127.0.222.1:41559)
0809 12:09:56.968461 (+ 940us) server_negotiation.cc:206] Beginning negotiation
0809 12:09:56.968467 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0809 12:09:56.968487 (+ 20us) server_negotiation.cc:407] Connection header received
0809 12:09:56.968551 (+ 64us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 12:09:56.968555 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 12:09:56.968616 (+ 61us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 12:09:56.968739 (+ 123us) server_negotiation.cc:226] Negotiated authn=TOKEN
0809 12:09:56.969292 (+ 553us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 12:09:56.970290 (+ 998us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 12:09:56.972148 (+ 1858us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 12:09:56.972962 (+ 814us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 12:09:56.973131 (+ 169us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 12:09:56.973815 (+ 684us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0809 12:09:56.973951 (+ 136us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0809 12:09:56.974071 (+ 120us) server_negotiation.cc:1035] Waiting for connection context
0809 12:09:56.974266 (+ 195us) server_negotiation.cc:299] Negotiation successful
0809 12:09:56.974352 (+ 86us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":806,"thread_start_us":635,"threads_started":1}
I20260809 12:09:56.981287 2566 raft_consensus.cc:1261] T 21da537165d245519766d5b22458a75b P a390a8c24bbe45baa12c712e9cf3c90d [term 1 FOLLOWER]: Refusing update from remote peer 36f277a9f15a47bd9843342a5fc170f8: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260809 12:09:56.981649 2413 raft_consensus.cc:1261] T 21da537165d245519766d5b22458a75b P 3842c96efca14e43898885e9664afe98 [term 1 FOLLOWER]: Refusing update from remote peer 36f277a9f15a47bd9843342a5fc170f8: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260809 12:09:56.982403 2659 consensus_queue.cc:1035] T 21da537165d245519766d5b22458a75b P 36f277a9f15a47bd9843342a5fc170f8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a390a8c24bbe45baa12c712e9cf3c90d" member_type: VOTER last_known_addr { host: "127.0.222.3" port: 35491 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260809 12:09:56.982875 2685 consensus_queue.cc:1035] T 21da537165d245519766d5b22458a75b P 36f277a9f15a47bd9843342a5fc170f8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3842c96efca14e43898885e9664afe98" member_type: VOTER last_known_addr { host: "127.0.222.2" port: 34441 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260809 12:09:56.988430 2641 tablet_replica.cc:439] T 21da537165d245519766d5b22458a75b P a390a8c24bbe45baa12c712e9cf3c90d: TxnStatusTablet state changed. Reason: New leader 36f277a9f15a47bd9843342a5fc170f8. Latest consensus state: current_term: 1 leader_uuid: "36f277a9f15a47bd9843342a5fc170f8" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "36f277a9f15a47bd9843342a5fc170f8" member_type: VOTER last_known_addr { host: "127.0.222.1" port: 41559 } } peers { permanent_uuid: "3842c96efca14e43898885e9664afe98" member_type: VOTER last_known_addr { host: "127.0.222.2" port: 34441 } } peers { permanent_uuid: "a390a8c24bbe45baa12c712e9cf3c90d" member_type: VOTER last_known_addr { host: "127.0.222.3" port: 35491 } } }
I20260809 12:09:56.988812 2641 tablet_replica.cc:442] T 21da537165d245519766d5b22458a75b P a390a8c24bbe45baa12c712e9cf3c90d: This TxnStatusTablet replica's current role is: FOLLOWER
I20260809 12:09:56.993494 2641 tablet_replica.cc:439] T 21da537165d245519766d5b22458a75b P a390a8c24bbe45baa12c712e9cf3c90d: TxnStatusTablet state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "36f277a9f15a47bd9843342a5fc170f8" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "36f277a9f15a47bd9843342a5fc170f8" member_type: VOTER last_known_addr { host: "127.0.222.1" port: 41559 } } peers { permanent_uuid: "3842c96efca14e43898885e9664afe98" member_type: VOTER last_known_addr { host: "127.0.222.2" port: 34441 } } peers { permanent_uuid: "a390a8c24bbe45baa12c712e9cf3c90d" member_type: VOTER last_known_addr { host: "127.0.222.3" port: 35491 } } }
I20260809 12:09:56.993894 2641 tablet_replica.cc:442] T 21da537165d245519766d5b22458a75b P a390a8c24bbe45baa12c712e9cf3c90d: This TxnStatusTablet replica's current role is: FOLLOWER
I20260809 12:09:57.003087 2685 tablet_replica.cc:439] T 21da537165d245519766d5b22458a75b P 36f277a9f15a47bd9843342a5fc170f8: TxnStatusTablet state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "36f277a9f15a47bd9843342a5fc170f8" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "36f277a9f15a47bd9843342a5fc170f8" member_type: VOTER last_known_addr { host: "127.0.222.1" port: 41559 } } peers { permanent_uuid: "3842c96efca14e43898885e9664afe98" member_type: VOTER last_known_addr { host: "127.0.222.2" port: 34441 } } peers { permanent_uuid: "a390a8c24bbe45baa12c712e9cf3c90d" member_type: VOTER last_known_addr { host: "127.0.222.3" port: 35491 } } }
I20260809 12:09:57.003968 2685 tablet_replica.cc:442] T 21da537165d245519766d5b22458a75b P 36f277a9f15a47bd9843342a5fc170f8: This TxnStatusTablet replica's current role is: LEADER
I20260809 12:09:57.005805 2640 tablet_replica.cc:439] T 21da537165d245519766d5b22458a75b P 3842c96efca14e43898885e9664afe98: TxnStatusTablet state changed. Reason: New leader 36f277a9f15a47bd9843342a5fc170f8. Latest consensus state: current_term: 1 leader_uuid: "36f277a9f15a47bd9843342a5fc170f8" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "36f277a9f15a47bd9843342a5fc170f8" member_type: VOTER last_known_addr { host: "127.0.222.1" port: 41559 } } peers { permanent_uuid: "3842c96efca14e43898885e9664afe98" member_type: VOTER last_known_addr { host: "127.0.222.2" port: 34441 } } peers { permanent_uuid: "a390a8c24bbe45baa12c712e9cf3c90d" member_type: VOTER last_known_addr { host: "127.0.222.3" port: 35491 } } }
I20260809 12:09:57.006229 2640 tablet_replica.cc:442] T 21da537165d245519766d5b22458a75b P 3842c96efca14e43898885e9664afe98: This TxnStatusTablet replica's current role is: FOLLOWER
I20260809 12:09:57.021597 2697 tablet_replica.cc:439] T 21da537165d245519766d5b22458a75b P 36f277a9f15a47bd9843342a5fc170f8: TxnStatusTablet state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "36f277a9f15a47bd9843342a5fc170f8" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "36f277a9f15a47bd9843342a5fc170f8" member_type: VOTER last_known_addr { host: "127.0.222.1" port: 41559 } } peers { permanent_uuid: "3842c96efca14e43898885e9664afe98" member_type: VOTER last_known_addr { host: "127.0.222.2" port: 34441 } } peers { permanent_uuid: "a390a8c24bbe45baa12c712e9cf3c90d" member_type: VOTER last_known_addr { host: "127.0.222.3" port: 35491 } } }
I20260809 12:09:57.022759 2697 tablet_replica.cc:442] T 21da537165d245519766d5b22458a75b P 36f277a9f15a47bd9843342a5fc170f8: This TxnStatusTablet replica's current role is: LEADER
I20260809 12:09:57.024085 2640 tablet_replica.cc:439] T 21da537165d245519766d5b22458a75b P 3842c96efca14e43898885e9664afe98: TxnStatusTablet state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "36f277a9f15a47bd9843342a5fc170f8" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "36f277a9f15a47bd9843342a5fc170f8" member_type: VOTER last_known_addr { host: "127.0.222.1" port: 41559 } } peers { permanent_uuid: "3842c96efca14e43898885e9664afe98" member_type: VOTER last_known_addr { host: "127.0.222.2" port: 34441 } } peers { permanent_uuid: "a390a8c24bbe45baa12c712e9cf3c90d" member_type: VOTER last_known_addr { host: "127.0.222.3" port: 35491 } } }
I20260809 12:09:57.024598 2640 tablet_replica.cc:442] T 21da537165d245519766d5b22458a75b P 3842c96efca14e43898885e9664afe98: This TxnStatusTablet replica's current role is: FOLLOWER
I20260809 12:09:57.030634 2700 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 12:09:57.012462 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33056 (local address 127.0.222.3:35491)
0809 12:09:57.019965 (+ 7503us) server_negotiation.cc:206] Beginning negotiation
0809 12:09:57.019975 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0809 12:09:57.020892 (+ 917us) server_negotiation.cc:407] Connection header received
0809 12:09:57.021151 (+ 259us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 12:09:57.021160 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 12:09:57.021357 (+ 197us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 12:09:57.021690 (+ 333us) server_negotiation.cc:226] Negotiated authn=TOKEN
0809 12:09:57.023227 (+ 1537us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 12:09:57.024789 (+ 1562us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 12:09:57.027350 (+ 2561us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 12:09:57.028292 (+ 942us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 12:09:57.028418 (+ 126us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 12:09:57.029059 (+ 641us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0809 12:09:57.029837 (+ 778us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0809 12:09:57.029981 (+ 144us) server_negotiation.cc:1035] Waiting for connection context
0809 12:09:57.030077 (+ 96us) server_negotiation.cc:299] Negotiation successful
0809 12:09:57.030166 (+ 89us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":7122,"thread_start_us":6878,"threads_started":1}
I20260809 12:09:57.056855 2641 consensus_queue.cc:1035] T 998406d582744483ae82e04cf4be1539 P a390a8c24bbe45baa12c712e9cf3c90d [LEADER]: Connected to new peer: Peer: permanent_uuid: "36f277a9f15a47bd9843342a5fc170f8" member_type: VOTER last_known_addr { host: "127.0.222.1" port: 41559 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260809 12:09:57.059965 2689 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 12:09:57.047328 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.222.3:47701 (local address 127.0.222.1:41559)
0809 12:09:57.047670 (+ 342us) server_negotiation.cc:206] Beginning negotiation
0809 12:09:57.047677 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0809 12:09:57.049512 (+ 1835us) server_negotiation.cc:407] Connection header received
0809 12:09:57.050034 (+ 522us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 12:09:57.050041 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 12:09:57.050165 (+ 124us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 12:09:57.050304 (+ 139us) server_negotiation.cc:226] Negotiated authn=TOKEN
0809 12:09:57.051562 (+ 1258us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 12:09:57.053357 (+ 1795us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 12:09:57.056431 (+ 3074us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 12:09:57.057537 (+ 1106us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 12:09:57.057686 (+ 149us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 12:09:57.058801 (+ 1115us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0809 12:09:57.058934 (+ 133us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0809 12:09:57.059089 (+ 155us) server_negotiation.cc:1035] Waiting for connection context
0809 12:09:57.059351 (+ 262us) server_negotiation.cc:299] Negotiation successful
0809 12:09:57.059455 (+ 104us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":134}
I20260809 12:09:57.061376 2705 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 12:09:57.046470 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.222.1:41559 (local address 127.0.222.3:47701)
0809 12:09:57.049238 (+ 2768us) negotiation.cc:107] Waiting for socket to connect
0809 12:09:57.049298 (+ 60us) client_negotiation.cc:174] Beginning negotiation
0809 12:09:57.049623 (+ 325us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 12:09:57.050338 (+ 715us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 12:09:57.050343 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 12:09:57.050356 (+ 13us) client_negotiation.cc:189] Negotiated authn=TOKEN
0809 12:09:57.051164 (+ 808us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 12:09:57.051181 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 12:09:57.053546 (+ 2365us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 12:09:57.053552 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 12:09:57.056183 (+ 2631us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 12:09:57.056200 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 12:09:57.058003 (+ 1803us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 12:09:57.058049 (+ 46us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 12:09:57.058427 (+ 378us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 12:09:57.058499 (+ 72us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0809 12:09:57.059176 (+ 677us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0809 12:09:57.059200 (+ 24us) client_negotiation.cc:769] Sending connection context
0809 12:09:57.060278 (+ 1078us) client_negotiation.cc:240] Negotiation successful
0809 12:09:57.060491 (+ 213us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":2521,"thread_start_us":1913,"threads_started":1}
I20260809 12:09:57.075948 2413 raft_consensus.cc:1261] T 998406d582744483ae82e04cf4be1539 P 3842c96efca14e43898885e9664afe98 [term 1 FOLLOWER]: Refusing update from remote peer a390a8c24bbe45baa12c712e9cf3c90d: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260809 12:09:57.076805 2641 consensus_queue.cc:1035] T 998406d582744483ae82e04cf4be1539 P a390a8c24bbe45baa12c712e9cf3c90d [LEADER]: Connected to new peer: Peer: permanent_uuid: "3842c96efca14e43898885e9664afe98" member_type: VOTER last_known_addr { host: "127.0.222.2" port: 34441 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260809 12:09:57.142948 2700 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 12:09:57.132088 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.222.1:59741 (local address 127.0.222.3:35491)
0809 12:09:57.132315 (+ 227us) server_negotiation.cc:206] Beginning negotiation
0809 12:09:57.132322 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0809 12:09:57.133521 (+ 1199us) server_negotiation.cc:407] Connection header received
0809 12:09:57.133725 (+ 204us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 12:09:57.133731 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 12:09:57.133806 (+ 75us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 12:09:57.133897 (+ 91us) server_negotiation.cc:226] Negotiated authn=TOKEN
0809 12:09:57.135707 (+ 1810us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 12:09:57.137161 (+ 1454us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 12:09:57.140239 (+ 3078us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 12:09:57.141115 (+ 876us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 12:09:57.141263 (+ 148us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 12:09:57.141860 (+ 597us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0809 12:09:57.142021 (+ 161us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0809 12:09:57.142144 (+ 123us) server_negotiation.cc:1035] Waiting for connection context
0809 12:09:57.142341 (+ 197us) server_negotiation.cc:299] Negotiation successful
0809 12:09:57.142451 (+ 110us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":61}
I20260809 12:09:57.143285 2713 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 12:09:57.131987 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.222.3:35491 (local address 127.0.222.1:59741)
0809 12:09:57.133408 (+ 1421us) negotiation.cc:107] Waiting for socket to connect
0809 12:09:57.133425 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0809 12:09:57.133596 (+ 171us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 12:09:57.133914 (+ 318us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 12:09:57.133918 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 12:09:57.133930 (+ 12us) client_negotiation.cc:189] Negotiated authn=TOKEN
0809 12:09:57.135373 (+ 1443us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 12:09:57.135386 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 12:09:57.137425 (+ 2039us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 12:09:57.137430 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 12:09:57.140043 (+ 2613us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 12:09:57.140056 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 12:09:57.141346 (+ 1290us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 12:09:57.141353 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 12:09:57.141535 (+ 182us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 12:09:57.141607 (+ 72us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0809 12:09:57.142172 (+ 565us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0809 12:09:57.142186 (+ 14us) client_negotiation.cc:769] Sending connection context
0809 12:09:57.142272 (+ 86us) client_negotiation.cc:240] Negotiation successful
0809 12:09:57.142390 (+ 118us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":973,"thread_start_us":663,"threads_started":1}
I20260809 12:09:57.196188 2098 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:49866:
table { table_name: "test-table" } modify_external_catalogs: true
I20260809 12:09:57.196596 2098 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:49866:
table { table_name: "test-table" } modify_external_catalogs: true
I20260809 12:09:57.201292 2098 catalog_manager.cc:5782] T 00000000000000000000000000000000 P eb0ae6c68a9b460ebbcc6098cfb950cb: Sending DeleteTablet for 3 replicas of tablet 998406d582744483ae82e04cf4be1539
I20260809 12:09:57.202884 2391 tablet_service.cc:1514] Processing DeleteTablet for tablet 998406d582744483ae82e04cf4be1539 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-09 12:09:57 UTC) from {username='kudu', principal='kudu/127.0.222.62@KRBTEST.COM'} at 127.0.0.1:59500
I20260809 12:09:57.203673 2538 tablet_service.cc:1514] Processing DeleteTablet for tablet 998406d582744483ae82e04cf4be1539 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-09 12:09:57 UTC) from {username='kudu', principal='kudu/127.0.222.62@KRBTEST.COM'} at 127.0.0.1:33046
I20260809 12:09:57.204252 2244 tablet_service.cc:1514] Processing DeleteTablet for tablet 998406d582744483ae82e04cf4be1539 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-09 12:09:57 UTC) from {username='kudu', principal='kudu/127.0.222.62@KRBTEST.COM'} at 127.0.0.1:51668
I20260809 12:09:57.212010 2719 tablet_replica.cc:330] T 998406d582744483ae82e04cf4be1539 P 36f277a9f15a47bd9843342a5fc170f8: stopping tablet replica
I20260809 12:09:57.212450 2719 raft_consensus.cc:2229] T 998406d582744483ae82e04cf4be1539 P 36f277a9f15a47bd9843342a5fc170f8 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260809 12:09:57.213039 2719 raft_consensus.cc:2258] T 998406d582744483ae82e04cf4be1539 P 36f277a9f15a47bd9843342a5fc170f8 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260809 12:09:57.215495 2719 ts_tablet_manager.cc:1905] T 998406d582744483ae82e04cf4be1539 P 36f277a9f15a47bd9843342a5fc170f8: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260809 12:09:57.218475 2716 tablet_replica.cc:330] T 998406d582744483ae82e04cf4be1539 P 3842c96efca14e43898885e9664afe98: stopping tablet replica
I20260809 12:09:57.218806 2716 raft_consensus.cc:2229] T 998406d582744483ae82e04cf4be1539 P 3842c96efca14e43898885e9664afe98 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260809 12:09:57.219264 2716 raft_consensus.cc:2258] T 998406d582744483ae82e04cf4be1539 P 3842c96efca14e43898885e9664afe98 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260809 12:09:57.220643 2716 ts_tablet_manager.cc:1905] T 998406d582744483ae82e04cf4be1539 P 3842c96efca14e43898885e9664afe98: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260809 12:09:57.221606 2717 tablet_replica.cc:330] T 998406d582744483ae82e04cf4be1539 P a390a8c24bbe45baa12c712e9cf3c90d: stopping tablet replica
I20260809 12:09:57.223268 2717 raft_consensus.cc:2229] T 998406d582744483ae82e04cf4be1539 P a390a8c24bbe45baa12c712e9cf3c90d [term 1 LEADER]: Raft consensus shutting down.
I20260809 12:09:57.224257 2717 raft_consensus.cc:2258] T 998406d582744483ae82e04cf4be1539 P a390a8c24bbe45baa12c712e9cf3c90d [term 1 FOLLOWER]: Raft consensus is shut down!
I20260809 12:09:57.232127 2717 ts_tablet_manager.cc:1905] T 998406d582744483ae82e04cf4be1539 P a390a8c24bbe45baa12c712e9cf3c90d: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260809 12:09:57.235325 2716 ts_tablet_manager.cc:1918] T 998406d582744483ae82e04cf4be1539 P 3842c96efca14e43898885e9664afe98: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260809 12:09:57.235622 2716 log.cc:1196] T 998406d582744483ae82e04cf4be1539 P 3842c96efca14e43898885e9664afe98: Deleting WAL directory at /tmp/dist-test-taskgH5Pjm/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786277382073938-888-0/minicluster-data/ts-1/wal/wals/998406d582744483ae82e04cf4be1539
I20260809 12:09:57.236464 2716 ts_tablet_manager.cc:1939] T 998406d582744483ae82e04cf4be1539 P 3842c96efca14e43898885e9664afe98: Deleting consensus metadata
I20260809 12:09:57.237941 2081 catalog_manager.cc:4841] TS 3842c96efca14e43898885e9664afe98 (127.0.222.2:34441): tablet 998406d582744483ae82e04cf4be1539 (table test-table [id=ddca40f91a774027929c394b80bc3930]) successfully deleted
I20260809 12:09:57.238410 2719 ts_tablet_manager.cc:1918] T 998406d582744483ae82e04cf4be1539 P 36f277a9f15a47bd9843342a5fc170f8: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260809 12:09:57.238761 2719 log.cc:1196] T 998406d582744483ae82e04cf4be1539 P 36f277a9f15a47bd9843342a5fc170f8: Deleting WAL directory at /tmp/dist-test-taskgH5Pjm/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786277382073938-888-0/minicluster-data/ts-0/wal/wals/998406d582744483ae82e04cf4be1539
I20260809 12:09:57.241074 2719 ts_tablet_manager.cc:1939] T 998406d582744483ae82e04cf4be1539 P 36f277a9f15a47bd9843342a5fc170f8: Deleting consensus metadata
I20260809 12:09:57.242449 2083 catalog_manager.cc:4841] TS 36f277a9f15a47bd9843342a5fc170f8 (127.0.222.1:41559): tablet 998406d582744483ae82e04cf4be1539 (table test-table [id=ddca40f91a774027929c394b80bc3930]) successfully deleted
I20260809 12:09:57.247704 2717 ts_tablet_manager.cc:1918] T 998406d582744483ae82e04cf4be1539 P a390a8c24bbe45baa12c712e9cf3c90d: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260809 12:09:57.248005 2717 log.cc:1196] T 998406d582744483ae82e04cf4be1539 P a390a8c24bbe45baa12c712e9cf3c90d: Deleting WAL directory at /tmp/dist-test-taskgH5Pjm/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786277382073938-888-0/minicluster-data/ts-2/wal/wals/998406d582744483ae82e04cf4be1539
I20260809 12:09:57.248771 2717 ts_tablet_manager.cc:1939] T 998406d582744483ae82e04cf4be1539 P a390a8c24bbe45baa12c712e9cf3c90d: Deleting consensus metadata
I20260809 12:09:57.250164 2085 catalog_manager.cc:4841] TS a390a8c24bbe45baa12c712e9cf3c90d (127.0.222.3:35491): tablet 998406d582744483ae82e04cf4be1539 (table test-table [id=ddca40f91a774027929c394b80bc3930]) successfully deleted
Aug 09 12:09:57 dist-test-slave-2w3w krb5kdc[2051](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786277395, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.0.222.1@KRBTEST.COM
I20260809 12:09:57.270130 2689 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 12:09:57.218773 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51688 (local address 127.0.222.1:41559)
0809 12:09:57.219052 (+ 279us) server_negotiation.cc:206] Beginning negotiation
0809 12:09:57.219058 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0809 12:09:57.240890 (+ 21832us) server_negotiation.cc:407] Connection header received
0809 12:09:57.241274 (+ 384us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 12:09:57.241291 (+ 17us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 12:09:57.241453 (+ 162us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 12:09:57.241606 (+ 153us) server_negotiation.cc:226] Negotiated authn=SASL
0809 12:09:57.245773 (+ 4167us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 12:09:57.247553 (+ 1780us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 12:09:57.250288 (+ 2735us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 12:09:57.251392 (+ 1104us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 12:09:57.251534 (+ 142us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 12:09:57.258529 (+ 6995us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 12:09:57.258553 (+ 24us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 12:09:57.258575 (+ 22us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0809 12:09:57.258660 (+ 85us) server_negotiation.cc:922] calling sasl_server_start()
0809 12:09:57.265123 (+ 6463us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 12:09:57.266354 (+ 1231us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 12:09:57.266365 (+ 11us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 12:09:57.266383 (+ 18us) server_negotiation.cc:967] calling sasl_server_step()
0809 12:09:57.266643 (+ 260us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 12:09:57.267274 (+ 631us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 12:09:57.267280 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 12:09:57.267291 (+ 11us) server_negotiation.cc:967] calling sasl_server_step()
0809 12:09:57.268049 (+ 758us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 12:09:57.268509 (+ 460us) server_negotiation.cc:1035] Waiting for connection context
0809 12:09:57.269258 (+ 749us) server_negotiation.cc:299] Negotiation successful
0809 12:09:57.269553 (+ 295us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":101}
W20260809 12:09:57.271591 2311 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.SetFlag from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:51688
I20260809 12:09:57.304323 2723 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 12:09:57.276108 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49872 (local address 127.0.222.62:42617)
0809 12:09:57.288255 (+ 12147us) server_negotiation.cc:206] Beginning negotiation
0809 12:09:57.288262 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0809 12:09:57.288286 (+ 24us) server_negotiation.cc:407] Connection header received
0809 12:09:57.288355 (+ 69us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 12:09:57.288361 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 12:09:57.288468 (+ 107us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 12:09:57.288614 (+ 146us) server_negotiation.cc:226] Negotiated authn=SASL
0809 12:09:57.290557 (+ 1943us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 12:09:57.291554 (+ 997us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 12:09:57.293141 (+ 1587us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 12:09:57.293761 (+ 620us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 12:09:57.293898 (+ 137us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 12:09:57.297122 (+ 3224us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 12:09:57.297127 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 12:09:57.297131 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0809 12:09:57.297164 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0809 12:09:57.301119 (+ 3955us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 12:09:57.301910 (+ 791us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 12:09:57.301914 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 12:09:57.301917 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0809 12:09:57.302157 (+ 240us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 12:09:57.302629 (+ 472us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 12:09:57.302634 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 12:09:57.302637 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0809 12:09:57.303020 (+ 383us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 12:09:57.303332 (+ 312us) server_negotiation.cc:1035] Waiting for connection context
0809 12:09:57.303568 (+ 236us) server_negotiation.cc:299] Negotiation successful
0809 12:09:57.303827 (+ 259us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":11971,"thread_start_us":11706,"threads_started":1}
W20260809 12:09:57.305091 2098 server_base.cc:1116] Unauthorized access attempt to method kudu.master.MasterService.TSHeartbeat from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:49872
I20260809 12:09:57.308367 888 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskgH5Pjm/build/asan/bin/kudu pid 2172
I20260809 12:09:57.324349 2689 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 12:09:57.310169 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51692 (local address 127.0.222.1:41559)
0809 12:09:57.310372 (+ 203us) server_negotiation.cc:206] Beginning negotiation
0809 12:09:57.310377 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0809 12:09:57.310391 (+ 14us) server_negotiation.cc:407] Connection header received
0809 12:09:57.310437 (+ 46us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 12:09:57.310440 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 12:09:57.310496 (+ 56us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 12:09:57.310607 (+ 111us) server_negotiation.cc:226] Negotiated authn=SASL
0809 12:09:57.311856 (+ 1249us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 12:09:57.312564 (+ 708us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 12:09:57.313883 (+ 1319us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 12:09:57.314326 (+ 443us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 12:09:57.314465 (+ 139us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 12:09:57.317340 (+ 2875us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 12:09:57.317347 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 12:09:57.317352 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0809 12:09:57.317390 (+ 38us) server_negotiation.cc:922] calling sasl_server_start()
0809 12:09:57.321074 (+ 3684us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 12:09:57.321730 (+ 656us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 12:09:57.321736 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 12:09:57.321739 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0809 12:09:57.321977 (+ 238us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 12:09:57.322360 (+ 383us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 12:09:57.322365 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 12:09:57.322369 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0809 12:09:57.322702 (+ 333us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 12:09:57.323175 (+ 473us) server_negotiation.cc:1035] Waiting for connection context
0809 12:09:57.323347 (+ 172us) server_negotiation.cc:299] Negotiation successful
0809 12:09:57.323802 (+ 455us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":69}
W20260809 12:09:57.325066 2311 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:51692
W20260809 12:09:57.325771 888 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskgH5Pjm/build/asan/bin/kudu pid 2172: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260809 12:09:57.326066 888 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskgH5Pjm/build/asan/bin/kudu with pid 2172
I20260809 12:09:57.356546 888 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskgH5Pjm/build/asan/bin/kudu pid 2322
Aug 09 12:09:57 dist-test-slave-2w3w krb5kdc[2051](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786277395, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.0.222.2@KRBTEST.COM
I20260809 12:09:57.372520 2726 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 12:09:57.357356 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59506 (local address 127.0.222.2:34441)
0809 12:09:57.358312 (+ 956us) server_negotiation.cc:206] Beginning negotiation
0809 12:09:57.358316 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0809 12:09:57.358332 (+ 16us) server_negotiation.cc:407] Connection header received
0809 12:09:57.358384 (+ 52us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 12:09:57.358388 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 12:09:57.358445 (+ 57us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 12:09:57.358582 (+ 137us) server_negotiation.cc:226] Negotiated authn=SASL
0809 12:09:57.359845 (+ 1263us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 12:09:57.360598 (+ 753us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 12:09:57.361886 (+ 1288us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 12:09:57.362312 (+ 426us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 12:09:57.362442 (+ 130us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 12:09:57.366562 (+ 4120us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 12:09:57.366578 (+ 16us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 12:09:57.366597 (+ 19us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0809 12:09:57.366637 (+ 40us) server_negotiation.cc:922] calling sasl_server_start()
0809 12:09:57.369602 (+ 2965us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 12:09:57.370312 (+ 710us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 12:09:57.370325 (+ 13us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 12:09:57.370327 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0809 12:09:57.370466 (+ 139us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 12:09:57.370862 (+ 396us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 12:09:57.370867 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 12:09:57.370871 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0809 12:09:57.371317 (+ 446us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 12:09:57.371624 (+ 307us) server_negotiation.cc:1035] Waiting for connection context
0809 12:09:57.371934 (+ 310us) server_negotiation.cc:299] Negotiation successful
0809 12:09:57.372165 (+ 231us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":855,"thread_start_us":707,"threads_started":1}
W20260809 12:09:57.373207 2459 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:59506
W20260809 12:09:57.373869 888 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskgH5Pjm/build/asan/bin/kudu pid 2322: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260809 12:09:57.374044 888 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskgH5Pjm/build/asan/bin/kudu with pid 2322
I20260809 12:09:57.401603 888 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskgH5Pjm/build/asan/bin/kudu pid 2468
Aug 09 12:09:57 dist-test-slave-2w3w krb5kdc[2051](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786277395, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.0.222.3@KRBTEST.COM
I20260809 12:09:57.419178 2700 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 12:09:57.402386 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33060 (local address 127.0.222.3:35491)
0809 12:09:57.402584 (+ 198us) server_negotiation.cc:206] Beginning negotiation
0809 12:09:57.402589 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0809 12:09:57.402659 (+ 70us) server_negotiation.cc:407] Connection header received
0809 12:09:57.402844 (+ 185us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 12:09:57.402847 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 12:09:57.402932 (+ 85us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 12:09:57.403092 (+ 160us) server_negotiation.cc:226] Negotiated authn=SASL
0809 12:09:57.404809 (+ 1717us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 12:09:57.405619 (+ 810us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 12:09:57.407512 (+ 1893us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 12:09:57.407956 (+ 444us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 12:09:57.408088 (+ 132us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 12:09:57.413085 (+ 4997us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 12:09:57.413098 (+ 13us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 12:09:57.413114 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0809 12:09:57.413151 (+ 37us) server_negotiation.cc:922] calling sasl_server_start()
0809 12:09:57.416304 (+ 3153us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 12:09:57.416994 (+ 690us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 12:09:57.417012 (+ 18us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 12:09:57.417014 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0809 12:09:57.417141 (+ 127us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 12:09:57.417534 (+ 393us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 12:09:57.417539 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 12:09:57.417541 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0809 12:09:57.417897 (+ 356us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 12:09:57.418277 (+ 380us) server_negotiation.cc:1035] Waiting for connection context
0809 12:09:57.418525 (+ 248us) server_negotiation.cc:299] Negotiation successful
0809 12:09:57.418757 (+ 232us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":71}
W20260809 12:09:57.419943 2608 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:33060
W20260809 12:09:57.420627 888 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskgH5Pjm/build/asan/bin/kudu pid 2468: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260809 12:09:57.420792 888 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskgH5Pjm/build/asan/bin/kudu with pid 2468
I20260809 12:09:57.440059 888 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskgH5Pjm/build/asan/bin/kudu pid 2065
I20260809 12:09:57.441234 2139 generic_service.cc:196] Checking for leaks (request via RPC)
I20260809 12:09:57.559461 888 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskgH5Pjm/build/asan/bin/kudu with pid 2065
2026-08-09T12:09:57Z chronyd exiting
[ OK ] SecurityITest.SmokeTestAsAuthorizedUser (5472 ms)
[ RUN ] SecurityITest.TxnSmokeWithDifferentUserTypes
Loading random data
Initializing database '/tmp/dist-test-taskgH5Pjm/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786277382073938-888-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 09 12:09:57 dist-test-slave-2w3w krb5kdc[2731](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 09 12:09:57 dist-test-slave-2w3w krb5kdc[2731](info): set up 2 sockets
Aug 09 12:09:57 dist-test-slave-2w3w krb5kdc[2731](info): commencing operation
krb5kdc: starting...
W20260809 12:09:58.706050 888 mini_kdc.cc:121] Time spent starting KDC: real 1.098s user 0.001s sys 0.021s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskgH5Pjm/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786277382073938-888-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskgH5Pjm/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786277382073938-888-0/krb5kdc/test-user.keytab.
Aug 09 12:09:58 dist-test-slave-2w3w krb5kdc[2731](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786277398, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-09T12:09:58Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-09T12:09:58Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.222.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.222.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.222.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskgH5Pjm/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786277382073938-888-0/krb5kdc/kudu_127.0.222.62.keytab.
Entry for principal kudu/127.0.222.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskgH5Pjm/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786277382073938-888-0/krb5kdc/kudu_127.0.222.62.keytab.
I20260809 12:09:58.914175 888 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskgH5Pjm/build/asan/bin/kudu
/tmp/dist-test-taskgH5Pjm/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskgH5Pjm/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786277382073938-888-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskgH5Pjm/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786277382073938-888-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskgH5Pjm/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786277382073938-888-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskgH5Pjm/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786277382073938-888-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.222.62:42305
--webserver_interface=127.0.222.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskgH5Pjm/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786277382073938-888-0/krb5kdc/kudu_127.0.222.62.keytab
--principal=kudu/127.0.222.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.222.20:43453
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.222.62:42305
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--txn_manager_enabled=true with env {KRB5CCNAME=/tmp/dist-test-taskgH5Pjm/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786277382073938-888-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskgH5Pjm/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786277382073938-888-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskgH5Pjm/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786277382073938-888-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260809 12:09:59.248657 2745 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260809 12:09:59.249017 2745 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260809 12:09:59.249173 2745 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260809 12:09:59.256629 2745 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260809 12:09:59.256770 2745 flags.cc:425] Enabled experimental flag: --txn_manager_enabled=true
W20260809 12:09:59.256863 2745 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260809 12:09:59.256979 2745 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260809 12:09:59.257062 2745 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260809 12:09:59.257153 2745 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260809 12:09:59.266759 2745 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.222.20:43453
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskgH5Pjm/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786277382073938-888-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskgH5Pjm/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786277382073938-888-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.222.62:42305
--txn_manager_enabled=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskgH5Pjm/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786277382073938-888-0/krb5kdc/kudu_127.0.222.62.keytab
--principal=kudu/127.0.222.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.222.62:42305
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskgH5Pjm/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786277382073938-888-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.222.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskgH5Pjm/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786277382073938-888-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 09 Aug 2026 11:06:41 UTC on ccycloud.quasar-bwoogy.root.comops.site
build id 4575991
ASAN enabled
I20260809 12:09:59.268121 2745 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260809 12:09:59.269498 2745 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260809 12:09:59.281365 2750 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 12:09:59.281327 2751 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 12:09:59.281265 2753 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260809 12:09:59.282244 2745 server_base.cc:1034] running on GCE node
I20260809 12:09:59.283008 2745 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260809 12:09:59.284343 2745 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260809 12:09:59.285568 2745 hybrid_clock.cc:648] HybridClock initialized: now 1786277399285512 us; error 30 us; skew 500 ppm
I20260809 12:09:59.285846 2745 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 09 12:09:59 dist-test-slave-2w3w krb5kdc[2731](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786277399, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.222.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260809 12:09:59.293393 2745 init.cc:377] Logged in from keytab as kudu/127.0.222.62@KRBTEST.COM (short username kudu)
I20260809 12:09:59.295127 2745 webserver.cc:460] Webserver started at http://127.0.222.62:35587/ using document root <none> and password file <none>
I20260809 12:09:59.295907 2745 fs_manager.cc:359] Metadata directory not provided
I20260809 12:09:59.296025 2745 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260809 12:09:59.296387 2745 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260809 12:09:59.299609 2745 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskgH5Pjm/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786277382073938-888-0/minicluster-data/master-0/data/instance:
uuid: "846fd7fb008b4e5ea8bd71f4646a10c4"
format_stamp: "Formatted at 2026-08-09 12:09:59 on dist-test-slave-2w3w"
server_key: "fa9aed2b012f4f2dd80a098423e74dbb"
server_key_iv: "fb5c7a85799b0f4a06a0113dbe5a4f05"
server_key_version: "encryptionkey@0"
I20260809 12:09:59.300439 2745 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskgH5Pjm/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786277382073938-888-0/minicluster-data/master-0/wal/instance:
uuid: "846fd7fb008b4e5ea8bd71f4646a10c4"
format_stamp: "Formatted at 2026-08-09 12:09:59 on dist-test-slave-2w3w"
server_key: "fa9aed2b012f4f2dd80a098423e74dbb"
server_key_iv: "fb5c7a85799b0f4a06a0113dbe5a4f05"
server_key_version: "encryptionkey@0"
I20260809 12:09:59.305823 2745 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.000s
I20260809 12:09:59.309360 2760 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260809 12:09:59.310747 2745 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.000s sys 0.002s
I20260809 12:09:59.310992 2745 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskgH5Pjm/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786277382073938-888-0/minicluster-data/master-0/data,/tmp/dist-test-taskgH5Pjm/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786277382073938-888-0/minicluster-data/master-0/wal
uuid: "846fd7fb008b4e5ea8bd71f4646a10c4"
format_stamp: "Formatted at 2026-08-09 12:09:59 on dist-test-slave-2w3w"
server_key: "fa9aed2b012f4f2dd80a098423e74dbb"
server_key_iv: "fb5c7a85799b0f4a06a0113dbe5a4f05"
server_key_version: "encryptionkey@0"
I20260809 12:09:59.311242 2745 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskgH5Pjm/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786277382073938-888-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskgH5Pjm/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786277382073938-888-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskgH5Pjm/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786277382073938-888-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260809 12:09:59.333412 2745 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260809 12:09:59.337630 2745 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260809 12:09:59.337932 2745 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260809 12:09:59.360209 2745 rpc_server.cc:307] RPC server started. Bound to: 127.0.222.62:42305
I20260809 12:09:59.360229 2828 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.222.62:42305 every 8 connection(s)
I20260809 12:09:59.361580 2745 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskgH5Pjm/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786277382073938-888-0/minicluster-data/master-0/data/info.pb
I20260809 12:09:59.364687 888 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskgH5Pjm/build/asan/bin/kudu as pid 2745
I20260809 12:09:59.364897 888 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskgH5Pjm/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786277382073938-888-0/minicluster-data/master-0/wal/instance
I20260809 12:09:59.365399 2829 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260809 12:09:59.365624 888 external_mini_cluster.cc:1428] Setting key d0b0c7012b056507f22023ae09cd6791
I20260809 12:09:59.380106 2829 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 846fd7fb008b4e5ea8bd71f4646a10c4: Bootstrap starting.
Aug 09 12:09:59 dist-test-slave-2w3w krb5kdc[2731](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786277398, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.222.62@KRBTEST.COM
I20260809 12:09:59.384136 2829 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 846fd7fb008b4e5ea8bd71f4646a10c4: Neither blocks nor log segments found. Creating new log.
I20260809 12:09:59.385524 2829 log.cc:824] T 00000000000000000000000000000000 P 846fd7fb008b4e5ea8bd71f4646a10c4: Log is configured to *not* fsync() on all Append() calls
I20260809 12:09:59.388422 2829 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 846fd7fb008b4e5ea8bd71f4646a10c4: No bootstrap required, opened a new log
I20260809 12:09:59.389642 2832 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 12:09:59.367624 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51974 (local address 127.0.222.62:42305)
0809 12:09:59.368513 (+ 889us) server_negotiation.cc:206] Beginning negotiation
0809 12:09:59.368535 (+ 22us) server_negotiation.cc:399] Waiting for connection header
0809 12:09:59.368585 (+ 50us) server_negotiation.cc:407] Connection header received
0809 12:09:59.370501 (+ 1916us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 12:09:59.370529 (+ 28us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 12:09:59.371059 (+ 530us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 12:09:59.371495 (+ 436us) server_negotiation.cc:226] Negotiated authn=SASL
0809 12:09:59.372765 (+ 1270us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 12:09:59.374811 (+ 2046us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 12:09:59.376116 (+ 1305us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 12:09:59.377233 (+ 1117us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 12:09:59.377436 (+ 203us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 12:09:59.381398 (+ 3962us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 12:09:59.381409 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 12:09:59.381428 (+ 19us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0809 12:09:59.381486 (+ 58us) server_negotiation.cc:922] calling sasl_server_start()
0809 12:09:59.385587 (+ 4101us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 12:09:59.386295 (+ 708us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 12:09:59.386302 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 12:09:59.386308 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0809 12:09:59.386507 (+ 199us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 12:09:59.386887 (+ 380us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 12:09:59.386892 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 12:09:59.386895 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0809 12:09:59.387534 (+ 639us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 12:09:59.387828 (+ 294us) server_negotiation.cc:1035] Waiting for connection context
0809 12:09:59.388280 (+ 452us) server_negotiation.cc:299] Negotiation successful
0809 12:09:59.388706 (+ 426us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":639,"thread_start_us":398,"threads_started":1}
I20260809 12:09:59.396674 2829 raft_consensus.cc:348] T 00000000000000000000000000000000 P 846fd7fb008b4e5ea8bd71f4646a10c4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "846fd7fb008b4e5ea8bd71f4646a10c4" member_type: VOTER last_known_addr { host: "127.0.222.62" port: 42305 } }
I20260809 12:09:59.397092 2829 raft_consensus.cc:374] T 00000000000000000000000000000000 P 846fd7fb008b4e5ea8bd71f4646a10c4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260809 12:09:59.397197 2829 raft_consensus.cc:729] T 00000000000000000000000000000000 P 846fd7fb008b4e5ea8bd71f4646a10c4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 846fd7fb008b4e5ea8bd71f4646a10c4, State: Initialized, Role: FOLLOWER
I20260809 12:09:59.397737 2829 consensus_queue.cc:260] T 00000000000000000000000000000000 P 846fd7fb008b4e5ea8bd71f4646a10c4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "846fd7fb008b4e5ea8bd71f4646a10c4" member_type: VOTER last_known_addr { host: "127.0.222.62" port: 42305 } }
I20260809 12:09:59.397990 2829 raft_consensus.cc:388] T 00000000000000000000000000000000 P 846fd7fb008b4e5ea8bd71f4646a10c4 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260809 12:09:59.398140 2829 raft_consensus.cc:482] T 00000000000000000000000000000000 P 846fd7fb008b4e5ea8bd71f4646a10c4 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260809 12:09:59.398306 2829 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 846fd7fb008b4e5ea8bd71f4646a10c4 [term 0 FOLLOWER]: Advancing to term 1
I20260809 12:09:59.400836 2829 raft_consensus.cc:504] T 00000000000000000000000000000000 P 846fd7fb008b4e5ea8bd71f4646a10c4 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "846fd7fb008b4e5ea8bd71f4646a10c4" member_type: VOTER last_known_addr { host: "127.0.222.62" port: 42305 } }
I20260809 12:09:59.401343 2829 leader_election.cc:302] T 00000000000000000000000000000000 P 846fd7fb008b4e5ea8bd71f4646a10c4 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 846fd7fb008b4e5ea8bd71f4646a10c4; no voters:
I20260809 12:09:59.401914 2829 leader_election.cc:288] T 00000000000000000000000000000000 P 846fd7fb008b4e5ea8bd71f4646a10c4 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260809 12:09:59.402043 2834 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 846fd7fb008b4e5ea8bd71f4646a10c4 [term 1 FOLLOWER]: Leader election won for term 1
I20260809 12:09:59.402770 2834 raft_consensus.cc:686] T 00000000000000000000000000000000 P 846fd7fb008b4e5ea8bd71f4646a10c4 [term 1 LEADER]: Becoming Leader. State: Replica: 846fd7fb008b4e5ea8bd71f4646a10c4, State: Running, Role: LEADER
I20260809 12:09:59.403455 2834 consensus_queue.cc:237] T 00000000000000000000000000000000 P 846fd7fb008b4e5ea8bd71f4646a10c4 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "846fd7fb008b4e5ea8bd71f4646a10c4" member_type: VOTER last_known_addr { host: "127.0.222.62" port: 42305 } }
I20260809 12:09:59.404065 2829 sys_catalog.cc:564] T 00000000000000000000000000000000 P 846fd7fb008b4e5ea8bd71f4646a10c4 [sys.catalog]: configured and running, proceeding with master startup.
I20260809 12:09:59.406177 2835 sys_catalog.cc:455] T 00000000000000000000000000000000 P 846fd7fb008b4e5ea8bd71f4646a10c4 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "846fd7fb008b4e5ea8bd71f4646a10c4" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "846fd7fb008b4e5ea8bd71f4646a10c4" member_type: VOTER last_known_addr { host: "127.0.222.62" port: 42305 } } }
I20260809 12:09:59.406595 2835 sys_catalog.cc:458] T 00000000000000000000000000000000 P 846fd7fb008b4e5ea8bd71f4646a10c4 [sys.catalog]: This master's current role is: LEADER
I20260809 12:09:59.406483 2836 sys_catalog.cc:455] T 00000000000000000000000000000000 P 846fd7fb008b4e5ea8bd71f4646a10c4 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 846fd7fb008b4e5ea8bd71f4646a10c4. Latest consensus state: current_term: 1 leader_uuid: "846fd7fb008b4e5ea8bd71f4646a10c4" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "846fd7fb008b4e5ea8bd71f4646a10c4" member_type: VOTER last_known_addr { host: "127.0.222.62" port: 42305 } } }
I20260809 12:09:59.406756 2836 sys_catalog.cc:458] T 00000000000000000000000000000000 P 846fd7fb008b4e5ea8bd71f4646a10c4 [sys.catalog]: This master's current role is: LEADER
I20260809 12:09:59.409972 2842 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260809 12:09:59.415222 2842 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260809 12:09:59.423949 2842 catalog_manager.cc:1347] Generated new cluster ID: 9f605783cc1b4cba8aa07eb1f0f8cf76
I20260809 12:09:59.424073 2842 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260809 12:09:59.448978 2842 catalog_manager.cc:1370] Generated new certificate authority record
I20260809 12:09:59.450361 2842 catalog_manager.cc:1504] Loading token signing keys...
I20260809 12:09:59.459997 2842 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 846fd7fb008b4e5ea8bd71f4646a10c4: Generated new TSK 0
I20260809 12:09:59.460860 2842 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.0.222.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.222.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.222.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskgH5Pjm/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786277382073938-888-0/krb5kdc/kudu_127.0.222.1.keytab.
Entry for principal kudu/127.0.222.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskgH5Pjm/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786277382073938-888-0/krb5kdc/kudu_127.0.222.1.keytab.
I20260809 12:09:59.520252 888 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskgH5Pjm/build/asan/bin/kudu
/tmp/dist-test-taskgH5Pjm/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskgH5Pjm/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786277382073938-888-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskgH5Pjm/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786277382073938-888-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskgH5Pjm/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786277382073938-888-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskgH5Pjm/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786277382073938-888-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.222.1:0
--local_ip_for_outbound_sockets=127.0.222.1
--webserver_interface=127.0.222.1
--webserver_port=0
--tserver_master_addrs=127.0.222.62:42305
--keytab_file=/tmp/dist-test-taskgH5Pjm/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786277382073938-888-0/krb5kdc/kudu_127.0.222.1.keytab
--principal=kudu/127.0.222.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.222.20:43453
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true
--txn_keepalive_interval_ms=500 with env {KRB5CCNAME=/tmp/dist-test-taskgH5Pjm/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786277382073938-888-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskgH5Pjm/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786277382073938-888-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskgH5Pjm/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786277382073938-888-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260809 12:09:59.868090 2856 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260809 12:09:59.868369 2856 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260809 12:09:59.868475 2856 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260809 12:09:59.875782 2856 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260809 12:09:59.875931 2856 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260809 12:09:59.876106 2856 flags.cc:425] Enabled experimental flag: --txn_keepalive_interval_ms=500
W20260809 12:09:59.876190 2856 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260809 12:09:59.876298 2856 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.222.1
I20260809 12:09:59.886098 2856 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.222.20:43453
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskgH5Pjm/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786277382073938-888-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskgH5Pjm/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786277382073938-888-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskgH5Pjm/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786277382073938-888-0/krb5kdc/kudu_127.0.222.1.keytab
--principal=kudu/127.0.222.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.222.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskgH5Pjm/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786277382073938-888-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.222.1
--webserver_port=0
--txn_keepalive_interval_ms=500
--enable_txn_system_client_init=true
--tserver_master_addrs=127.0.222.62:42305
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.222.1
--log_dir=/tmp/dist-test-taskgH5Pjm/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786277382073938-888-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 09 Aug 2026 11:06:41 UTC on ccycloud.quasar-bwoogy.root.comops.site
build id 4575991
ASAN enabled
I20260809 12:09:59.887454 2856 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260809 12:09:59.888695 2856 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260809 12:09:59.901785 2861 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 12:09:59.901882 2862 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 12:09:59.901888 2864 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260809 12:09:59.902909 2856 server_base.cc:1034] running on GCE node
I20260809 12:09:59.903558 2856 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260809 12:09:59.904582 2856 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260809 12:09:59.905833 2856 hybrid_clock.cc:648] HybridClock initialized: now 1786277399905788 us; error 64 us; skew 500 ppm
I20260809 12:09:59.906150 2856 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 09 12:09:59 dist-test-slave-2w3w krb5kdc[2731](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786277399, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.222.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260809 12:09:59.909621 2856 init.cc:377] Logged in from keytab as kudu/127.0.222.1@KRBTEST.COM (short username kudu)
I20260809 12:09:59.911237 2856 webserver.cc:460] Webserver started at http://127.0.222.1:43321/ using document root <none> and password file <none>
I20260809 12:09:59.911885 2856 fs_manager.cc:359] Metadata directory not provided
I20260809 12:09:59.911991 2856 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260809 12:09:59.912262 2856 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260809 12:09:59.915329 2856 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskgH5Pjm/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786277382073938-888-0/minicluster-data/ts-0/data/instance:
uuid: "9cea8553e4754cd89ec242d7c7a78577"
format_stamp: "Formatted at 2026-08-09 12:09:59 on dist-test-slave-2w3w"
server_key: "778e6b6978a9772bd395f34fd44f6815"
server_key_iv: "3ee22d78184ee6ac588a1742ccceee96"
server_key_version: "encryptionkey@0"
I20260809 12:09:59.916128 2856 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskgH5Pjm/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786277382073938-888-0/minicluster-data/ts-0/wal/instance:
uuid: "9cea8553e4754cd89ec242d7c7a78577"
format_stamp: "Formatted at 2026-08-09 12:09:59 on dist-test-slave-2w3w"
server_key: "778e6b6978a9772bd395f34fd44f6815"
server_key_iv: "3ee22d78184ee6ac588a1742ccceee96"
server_key_version: "encryptionkey@0"
I20260809 12:09:59.921535 2856 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.000s
I20260809 12:09:59.925009 2871 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260809 12:09:59.926277 2856 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260809 12:09:59.926451 2856 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskgH5Pjm/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786277382073938-888-0/minicluster-data/ts-0/data,/tmp/dist-test-taskgH5Pjm/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786277382073938-888-0/minicluster-data/ts-0/wal
uuid: "9cea8553e4754cd89ec242d7c7a78577"
format_stamp: "Formatted at 2026-08-09 12:09:59 on dist-test-slave-2w3w"
server_key: "778e6b6978a9772bd395f34fd44f6815"
server_key_iv: "3ee22d78184ee6ac588a1742ccceee96"
server_key_version: "encryptionkey@0"
I20260809 12:09:59.926653 2856 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskgH5Pjm/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786277382073938-888-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskgH5Pjm/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786277382073938-888-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskgH5Pjm/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786277382073938-888-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260809 12:09:59.938764 2856 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260809 12:09:59.942680 2856 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260809 12:09:59.943012 2856 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260809 12:09:59.945683 2856 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260809 12:09:59.945822 2856 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260809 12:09:59.945964 2856 ts_tablet_manager.cc:610] Registered 0 tablets
I20260809 12:09:59.946056 2856 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Aug 09 12:09:59 dist-test-slave-2w3w krb5kdc[2731](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786277399, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.222.1@KRBTEST.COM for kudu/127.0.222.62@KRBTEST.COM
I20260809 12:09:59.971129 2882 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 12:09:59.946357 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.222.1:44107 (local address 127.0.222.62:42305)
0809 12:09:59.947071 (+ 714us) server_negotiation.cc:206] Beginning negotiation
0809 12:09:59.947077 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0809 12:09:59.948109 (+ 1032us) server_negotiation.cc:407] Connection header received
0809 12:09:59.950154 (+ 2045us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 12:09:59.950163 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 12:09:59.950271 (+ 108us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 12:09:59.950443 (+ 172us) server_negotiation.cc:226] Negotiated authn=SASL
0809 12:09:59.952770 (+ 2327us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 12:09:59.953836 (+ 1066us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 12:09:59.956634 (+ 2798us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 12:09:59.957053 (+ 419us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 12:09:59.957193 (+ 140us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 12:09:59.962848 (+ 5655us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 12:09:59.962854 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 12:09:59.962859 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0809 12:09:59.962927 (+ 68us) server_negotiation.cc:922] calling sasl_server_start()
0809 12:09:59.966042 (+ 3115us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 12:09:59.967484 (+ 1442us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 12:09:59.967491 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 12:09:59.967496 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0809 12:09:59.967712 (+ 216us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 12:09:59.968393 (+ 681us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 12:09:59.968397 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 12:09:59.968399 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0809 12:09:59.968821 (+ 422us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 12:09:59.969006 (+ 185us) server_negotiation.cc:1035] Waiting for connection context
0809 12:09:59.970156 (+ 1150us) server_negotiation.cc:299] Negotiation successful
0809 12:09:59.970546 (+ 390us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":568,"thread_start_us":447,"threads_started":1}
I20260809 12:09:59.971781 2883 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 12:09:59.947019 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.222.62:42305 (local address 127.0.222.1:44107)
0809 12:09:59.947924 (+ 905us) negotiation.cc:107] Waiting for socket to connect
0809 12:09:59.947974 (+ 50us) client_negotiation.cc:174] Beginning negotiation
0809 12:09:59.949879 (+ 1905us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 12:09:59.950733 (+ 854us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 12:09:59.950744 (+ 11us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 12:09:59.951590 (+ 846us) client_negotiation.cc:189] Negotiated authn=SASL
0809 12:09:59.952563 (+ 973us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 12:09:59.952593 (+ 30us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 12:09:59.954005 (+ 1412us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 12:09:59.954014 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 12:09:59.956456 (+ 2442us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 12:09:59.956478 (+ 22us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 12:09:59.957325 (+ 847us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 12:09:59.957333 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 12:09:59.957595 (+ 262us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 12:09:59.958552 (+ 957us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0809 12:09:59.958592 (+ 40us) client_negotiation.cc:656] calling sasl_client_start()
0809 12:09:59.962477 (+ 3885us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0809 12:09:59.966247 (+ 3770us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0809 12:09:59.966262 (+ 15us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0809 12:09:59.966283 (+ 21us) client_negotiation.cc:760] calling sasl_client_step()
0809 12:09:59.967247 (+ 964us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0809 12:09:59.967862 (+ 615us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0809 12:09:59.967869 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0809 12:09:59.967872 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0809 12:09:59.968238 (+ 366us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0809 12:09:59.968977 (+ 739us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0809 12:09:59.968986 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0809 12:09:59.969593 (+ 607us) client_negotiation.cc:769] Sending connection context
0809 12:09:59.969982 (+ 389us) client_negotiation.cc:240] Negotiation successful
0809 12:09:59.970463 (+ 481us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":616,"thread_start_us":427,"threads_started":1}
I20260809 12:09:59.996026 2882 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 12:09:59.978145 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.222.1:50141 (local address 127.0.222.62:42305)
0809 12:09:59.978327 (+ 182us) server_negotiation.cc:206] Beginning negotiation
0809 12:09:59.978331 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0809 12:09:59.980028 (+ 1697us) server_negotiation.cc:407] Connection header received
0809 12:09:59.980266 (+ 238us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 12:09:59.980270 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 12:09:59.980337 (+ 67us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 12:09:59.980441 (+ 104us) server_negotiation.cc:226] Negotiated authn=SASL
0809 12:09:59.981467 (+ 1026us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 12:09:59.982948 (+ 1481us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 12:09:59.985128 (+ 2180us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 12:09:59.985719 (+ 591us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 12:09:59.985877 (+ 158us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 12:09:59.988863 (+ 2986us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 12:09:59.988868 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 12:09:59.988871 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0809 12:09:59.988905 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0809 12:09:59.992269 (+ 3364us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 12:09:59.993442 (+ 1173us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 12:09:59.993450 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 12:09:59.993455 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0809 12:09:59.993617 (+ 162us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 12:09:59.994111 (+ 494us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 12:09:59.994118 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 12:09:59.994124 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0809 12:09:59.994604 (+ 480us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 12:09:59.994802 (+ 198us) server_negotiation.cc:1035] Waiting for connection context
0809 12:09:59.995571 (+ 769us) server_negotiation.cc:299] Negotiation successful
0809 12:09:59.995780 (+ 209us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":71}
I20260809 12:09:59.996654 2957 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 12:09:59.977880 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.222.62:42305 (local address 127.0.222.1:50141)
0809 12:09:59.979891 (+ 2011us) negotiation.cc:107] Waiting for socket to connect
0809 12:09:59.979917 (+ 26us) client_negotiation.cc:174] Beginning negotiation
0809 12:09:59.980116 (+ 199us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 12:09:59.980473 (+ 357us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 12:09:59.980477 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 12:09:59.980970 (+ 493us) client_negotiation.cc:189] Negotiated authn=SASL
0809 12:09:59.981261 (+ 291us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 12:09:59.981271 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 12:09:59.983147 (+ 1876us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 12:09:59.983156 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 12:09:59.984951 (+ 1795us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 12:09:59.984964 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 12:09:59.985946 (+ 982us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 12:09:59.985955 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 12:09:59.986223 (+ 268us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 12:09:59.987021 (+ 798us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0809 12:09:59.987055 (+ 34us) client_negotiation.cc:656] calling sasl_client_start()
0809 12:09:59.988672 (+ 1617us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0809 12:09:59.992479 (+ 3807us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0809 12:09:59.992484 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0809 12:09:59.992489 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0809 12:09:59.993274 (+ 785us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0809 12:09:59.993744 (+ 470us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0809 12:09:59.993753 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0809 12:09:59.993757 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0809 12:09:59.993974 (+ 217us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0809 12:09:59.994883 (+ 909us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0809 12:09:59.994890 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0809 12:09:59.995196 (+ 306us) client_negotiation.cc:769] Sending connection context
0809 12:09:59.995428 (+ 232us) client_negotiation.cc:240] Negotiation successful
0809 12:09:59.995948 (+ 520us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1815,"thread_start_us":1630,"threads_started":1}
I20260809 12:10:00.001611 2856 rpc_server.cc:307] RPC server started. Bound to: 127.0.222.1:35257
I20260809 12:10:00.001658 3002 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.222.1:35257 every 8 connection(s)
I20260809 12:10:00.002835 2856 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskgH5Pjm/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786277382073938-888-0/minicluster-data/ts-0/data/info.pb
I20260809 12:10:00.005059 3003 heartbeater.cc:347] Connected to a master server at 127.0.222.62:42305
I20260809 12:10:00.005304 3003 heartbeater.cc:464] Registering TS with master...
I20260809 12:10:00.006003 3003 heartbeater.cc:511] Master 127.0.222.62:42305 requested a full tablet report, sending...
I20260809 12:10:00.008132 2780 ts_manager.cc:194] Registered new tserver with Master: 9cea8553e4754cd89ec242d7c7a78577 (127.0.222.1:35257)
I20260809 12:10:00.009766 2780 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.222.1@KRBTEST.COM'} at 127.0.222.1:44107
I20260809 12:10:00.012403 888 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskgH5Pjm/build/asan/bin/kudu as pid 2856
I20260809 12:10:00.012610 888 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskgH5Pjm/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786277382073938-888-0/minicluster-data/ts-0/wal/instance
I20260809 12:10:00.013360 888 external_mini_cluster.cc:1428] Setting key 5da4414352835d01f9bfd965fe65423f
WARNING: no policy specified for kudu/127.0.222.2@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.222.2@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.222.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskgH5Pjm/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786277382073938-888-0/krb5kdc/kudu_127.0.222.2.keytab.
Entry for principal kudu/127.0.222.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskgH5Pjm/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786277382073938-888-0/krb5kdc/kudu_127.0.222.2.keytab.
I20260809 12:10:00.058257 888 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskgH5Pjm/build/asan/bin/kudu
/tmp/dist-test-taskgH5Pjm/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskgH5Pjm/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786277382073938-888-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskgH5Pjm/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786277382073938-888-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskgH5Pjm/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786277382073938-888-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskgH5Pjm/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786277382073938-888-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.222.2:0
--local_ip_for_outbound_sockets=127.0.222.2
--webserver_interface=127.0.222.2
--webserver_port=0
--tserver_master_addrs=127.0.222.62:42305
--keytab_file=/tmp/dist-test-taskgH5Pjm/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786277382073938-888-0/krb5kdc/kudu_127.0.222.2.keytab
--principal=kudu/127.0.222.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.222.20:43453
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true
--txn_keepalive_interval_ms=500 with env {KRB5CCNAME=/tmp/dist-test-taskgH5Pjm/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786277382073938-888-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskgH5Pjm/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786277382073938-888-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskgH5Pjm/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786277382073938-888-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260809 12:10:00.388790 3007 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260809 12:10:00.389077 3007 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260809 12:10:00.389221 3007 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260809 12:10:00.396575 3007 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260809 12:10:00.396723 3007 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260809 12:10:00.396881 3007 flags.cc:425] Enabled experimental flag: --txn_keepalive_interval_ms=500
W20260809 12:10:00.396962 3007 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260809 12:10:00.397082 3007 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.222.2
I20260809 12:10:00.407214 3007 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.222.20:43453
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskgH5Pjm/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786277382073938-888-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskgH5Pjm/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786277382073938-888-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskgH5Pjm/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786277382073938-888-0/krb5kdc/kudu_127.0.222.2.keytab
--principal=kudu/127.0.222.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.222.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskgH5Pjm/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786277382073938-888-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.222.2
--webserver_port=0
--txn_keepalive_interval_ms=500
--enable_txn_system_client_init=true
--tserver_master_addrs=127.0.222.62:42305
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.222.2
--log_dir=/tmp/dist-test-taskgH5Pjm/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786277382073938-888-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 09 Aug 2026 11:06:41 UTC on ccycloud.quasar-bwoogy.root.comops.site
build id 4575991
ASAN enabled
I20260809 12:10:00.408564 3007 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260809 12:10:00.409926 3007 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260809 12:10:00.423501 3012 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 12:10:00.423424 3015 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 12:10:00.423401 3013 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260809 12:10:00.424546 3007 server_base.cc:1034] running on GCE node
I20260809 12:10:00.425283 3007 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260809 12:10:00.426333 3007 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260809 12:10:00.427560 3007 hybrid_clock.cc:648] HybridClock initialized: now 1786277400427511 us; error 45 us; skew 500 ppm
I20260809 12:10:00.427934 3007 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 09 12:10:00 dist-test-slave-2w3w krb5kdc[2731](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786277400, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.222.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260809 12:10:00.431340 3007 init.cc:377] Logged in from keytab as kudu/127.0.222.2@KRBTEST.COM (short username kudu)
I20260809 12:10:00.433025 3007 webserver.cc:460] Webserver started at http://127.0.222.2:33277/ using document root <none> and password file <none>
I20260809 12:10:00.433748 3007 fs_manager.cc:359] Metadata directory not provided
I20260809 12:10:00.433907 3007 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260809 12:10:00.434235 3007 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260809 12:10:00.437387 3007 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskgH5Pjm/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786277382073938-888-0/minicluster-data/ts-1/data/instance:
uuid: "26d0a424d542436a8830dbe1f8d5142e"
format_stamp: "Formatted at 2026-08-09 12:10:00 on dist-test-slave-2w3w"
server_key: "a2a0b2da7460c95ecb00d945fdcd96f4"
server_key_iv: "b8e44196061739f4c76f5a295ecce1b9"
server_key_version: "encryptionkey@0"
I20260809 12:10:00.438220 3007 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskgH5Pjm/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786277382073938-888-0/minicluster-data/ts-1/wal/instance:
uuid: "26d0a424d542436a8830dbe1f8d5142e"
format_stamp: "Formatted at 2026-08-09 12:10:00 on dist-test-slave-2w3w"
server_key: "a2a0b2da7460c95ecb00d945fdcd96f4"
server_key_iv: "b8e44196061739f4c76f5a295ecce1b9"
server_key_version: "encryptionkey@0"
I20260809 12:10:00.443565 3007 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.007s sys 0.001s
I20260809 12:10:00.447213 3022 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260809 12:10:00.448573 3007 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260809 12:10:00.448761 3007 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskgH5Pjm/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786277382073938-888-0/minicluster-data/ts-1/data,/tmp/dist-test-taskgH5Pjm/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786277382073938-888-0/minicluster-data/ts-1/wal
uuid: "26d0a424d542436a8830dbe1f8d5142e"
format_stamp: "Formatted at 2026-08-09 12:10:00 on dist-test-slave-2w3w"
server_key: "a2a0b2da7460c95ecb00d945fdcd96f4"
server_key_iv: "b8e44196061739f4c76f5a295ecce1b9"
server_key_version: "encryptionkey@0"
I20260809 12:10:00.448982 3007 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskgH5Pjm/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786277382073938-888-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskgH5Pjm/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786277382073938-888-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskgH5Pjm/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786277382073938-888-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260809 12:10:00.465936 3007 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260809 12:10:00.470235 3007 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260809 12:10:00.470521 3007 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260809 12:10:00.473558 3007 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260809 12:10:00.473694 3007 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260809 12:10:00.473831 3007 ts_tablet_manager.cc:610] Registered 0 tablets
I20260809 12:10:00.473922 3007 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Aug 09 12:10:00 dist-test-slave-2w3w krb5kdc[2731](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786277400, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.222.2@KRBTEST.COM for kudu/127.0.222.62@KRBTEST.COM
I20260809 12:10:00.495174 2882 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 12:10:00.473895 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.222.2:34117 (local address 127.0.222.62:42305)
0809 12:10:00.474095 (+ 200us) server_negotiation.cc:206] Beginning negotiation
0809 12:10:00.474101 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0809 12:10:00.475513 (+ 1412us) server_negotiation.cc:407] Connection header received
0809 12:10:00.477766 (+ 2253us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 12:10:00.477772 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 12:10:00.477875 (+ 103us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 12:10:00.477994 (+ 119us) server_negotiation.cc:226] Negotiated authn=SASL
0809 12:10:00.480303 (+ 2309us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 12:10:00.481345 (+ 1042us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 12:10:00.483030 (+ 1685us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 12:10:00.483652 (+ 622us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 12:10:00.483791 (+ 139us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 12:10:00.488761 (+ 4970us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 12:10:00.488766 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 12:10:00.488769 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0809 12:10:00.488806 (+ 37us) server_negotiation.cc:922] calling sasl_server_start()
0809 12:10:00.491730 (+ 2924us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 12:10:00.492673 (+ 943us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 12:10:00.492678 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 12:10:00.492682 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0809 12:10:00.492809 (+ 127us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 12:10:00.493265 (+ 456us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 12:10:00.493273 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 12:10:00.493278 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0809 12:10:00.493676 (+ 398us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 12:10:00.493850 (+ 174us) server_negotiation.cc:1035] Waiting for connection context
0809 12:10:00.494537 (+ 687us) server_negotiation.cc:299] Negotiation successful
0809 12:10:00.494779 (+ 242us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":52}
I20260809 12:10:00.496312 3031 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 12:10:00.474383 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.222.62:42305 (local address 127.0.222.2:34117)
0809 12:10:00.475329 (+ 946us) negotiation.cc:107] Waiting for socket to connect
0809 12:10:00.475387 (+ 58us) client_negotiation.cc:174] Beginning negotiation
0809 12:10:00.477459 (+ 2072us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 12:10:00.478272 (+ 813us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 12:10:00.478287 (+ 15us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 12:10:00.479094 (+ 807us) client_negotiation.cc:189] Negotiated authn=SASL
0809 12:10:00.480096 (+ 1002us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 12:10:00.480130 (+ 34us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 12:10:00.481528 (+ 1398us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 12:10:00.481534 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 12:10:00.482848 (+ 1314us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 12:10:00.482860 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 12:10:00.483922 (+ 1062us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 12:10:00.483930 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 12:10:00.484186 (+ 256us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 12:10:00.485146 (+ 960us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0809 12:10:00.485172 (+ 26us) client_negotiation.cc:656] calling sasl_client_start()
0809 12:10:00.488522 (+ 3350us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0809 12:10:00.491890 (+ 3368us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0809 12:10:00.491901 (+ 11us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0809 12:10:00.491930 (+ 29us) client_negotiation.cc:760] calling sasl_client_step()
0809 12:10:00.492539 (+ 609us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0809 12:10:00.492927 (+ 388us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0809 12:10:00.492932 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0809 12:10:00.492935 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0809 12:10:00.493137 (+ 202us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0809 12:10:00.493813 (+ 676us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0809 12:10:00.493821 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0809 12:10:00.494184 (+ 363us) client_negotiation.cc:769] Sending connection context
0809 12:10:00.494438 (+ 254us) client_negotiation.cc:240] Negotiation successful
0809 12:10:00.495007 (+ 569us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":680,"thread_start_us":405,"threads_started":1}
I20260809 12:10:00.518476 3078 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 12:10:00.502152 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.222.62:42305 (local address 127.0.222.2:50517)
0809 12:10:00.502703 (+ 551us) negotiation.cc:107] Waiting for socket to connect
0809 12:10:00.502728 (+ 25us) client_negotiation.cc:174] Beginning negotiation
0809 12:10:00.502908 (+ 180us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 12:10:00.503391 (+ 483us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 12:10:00.503395 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 12:10:00.503962 (+ 567us) client_negotiation.cc:189] Negotiated authn=SASL
0809 12:10:00.504463 (+ 501us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 12:10:00.504477 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 12:10:00.506029 (+ 1552us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 12:10:00.506033 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 12:10:00.507882 (+ 1849us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 12:10:00.507901 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 12:10:00.508929 (+ 1028us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 12:10:00.508935 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 12:10:00.509213 (+ 278us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 12:10:00.509999 (+ 786us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0809 12:10:00.510018 (+ 19us) client_negotiation.cc:656] calling sasl_client_start()
0809 12:10:00.510827 (+ 809us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0809 12:10:00.515583 (+ 4756us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0809 12:10:00.515590 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0809 12:10:00.515594 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0809 12:10:00.516046 (+ 452us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0809 12:10:00.516554 (+ 508us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0809 12:10:00.516557 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0809 12:10:00.516559 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0809 12:10:00.516686 (+ 127us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0809 12:10:00.517522 (+ 836us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0809 12:10:00.517526 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0809 12:10:00.517740 (+ 214us) client_negotiation.cc:769] Sending connection context
0809 12:10:00.517871 (+ 131us) client_negotiation.cc:240] Negotiation successful
0809 12:10:00.518147 (+ 276us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":424,"thread_start_us":279,"threads_started":1}
I20260809 12:10:00.518777 2882 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 12:10:00.502186 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.222.2:50517 (local address 127.0.222.62:42305)
0809 12:10:00.502372 (+ 186us) server_negotiation.cc:206] Beginning negotiation
0809 12:10:00.502380 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0809 12:10:00.502856 (+ 476us) server_negotiation.cc:407] Connection header received
0809 12:10:00.503110 (+ 254us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 12:10:00.503118 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 12:10:00.503224 (+ 106us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 12:10:00.503374 (+ 150us) server_negotiation.cc:226] Negotiated authn=SASL
0809 12:10:00.504686 (+ 1312us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 12:10:00.505833 (+ 1147us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 12:10:00.508090 (+ 2257us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 12:10:00.508732 (+ 642us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 12:10:00.508876 (+ 144us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 12:10:00.511076 (+ 2200us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 12:10:00.511086 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 12:10:00.511093 (+ 7us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0809 12:10:00.511148 (+ 55us) server_negotiation.cc:922] calling sasl_server_start()
0809 12:10:00.515412 (+ 4264us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 12:10:00.516189 (+ 777us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 12:10:00.516197 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 12:10:00.516204 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0809 12:10:00.516436 (+ 232us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 12:10:00.516792 (+ 356us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 12:10:00.516800 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 12:10:00.516806 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0809 12:10:00.517398 (+ 592us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 12:10:00.517590 (+ 192us) server_negotiation.cc:1035] Waiting for connection context
0809 12:10:00.518025 (+ 435us) server_negotiation.cc:299] Negotiation successful
0809 12:10:00.518344 (+ 319us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":51}
I20260809 12:10:00.533316 3007 rpc_server.cc:307] RPC server started. Bound to: 127.0.222.2:37581
I20260809 12:10:00.533372 3153 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.222.2:37581 every 8 connection(s)
I20260809 12:10:00.534518 3007 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskgH5Pjm/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786277382073938-888-0/minicluster-data/ts-1/data/info.pb
I20260809 12:10:00.536607 3154 heartbeater.cc:347] Connected to a master server at 127.0.222.62:42305
I20260809 12:10:00.536828 3154 heartbeater.cc:464] Registering TS with master...
I20260809 12:10:00.537474 3154 heartbeater.cc:511] Master 127.0.222.62:42305 requested a full tablet report, sending...
I20260809 12:10:00.538400 888 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskgH5Pjm/build/asan/bin/kudu as pid 3007
I20260809 12:10:00.538595 888 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskgH5Pjm/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786277382073938-888-0/minicluster-data/ts-1/wal/instance
I20260809 12:10:00.539039 2780 ts_manager.cc:194] Registered new tserver with Master: 26d0a424d542436a8830dbe1f8d5142e (127.0.222.2:37581)
I20260809 12:10:00.539129 888 external_mini_cluster.cc:1428] Setting key 888a98f05e4ae374e12af36fd7e7bcde
I20260809 12:10:00.539942 2780 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.222.2@KRBTEST.COM'} at 127.0.222.2:34117
WARNING: no policy specified for kudu/127.0.222.3@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.222.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.222.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskgH5Pjm/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786277382073938-888-0/krb5kdc/kudu_127.0.222.3.keytab.
Entry for principal kudu/127.0.222.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskgH5Pjm/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786277382073938-888-0/krb5kdc/kudu_127.0.222.3.keytab.
I20260809 12:10:00.585515 888 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskgH5Pjm/build/asan/bin/kudu
/tmp/dist-test-taskgH5Pjm/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskgH5Pjm/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786277382073938-888-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskgH5Pjm/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786277382073938-888-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskgH5Pjm/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786277382073938-888-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskgH5Pjm/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786277382073938-888-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.222.3:0
--local_ip_for_outbound_sockets=127.0.222.3
--webserver_interface=127.0.222.3
--webserver_port=0
--tserver_master_addrs=127.0.222.62:42305
--keytab_file=/tmp/dist-test-taskgH5Pjm/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786277382073938-888-0/krb5kdc/kudu_127.0.222.3.keytab
--principal=kudu/127.0.222.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.222.20:43453
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true
--txn_keepalive_interval_ms=500 with env {KRB5CCNAME=/tmp/dist-test-taskgH5Pjm/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786277382073938-888-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskgH5Pjm/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786277382073938-888-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskgH5Pjm/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786277382073938-888-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260809 12:10:00.916944 3158 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260809 12:10:00.917253 3158 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260809 12:10:00.917431 3158 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260809 12:10:00.924644 3158 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260809 12:10:00.924790 3158 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260809 12:10:00.924947 3158 flags.cc:425] Enabled experimental flag: --txn_keepalive_interval_ms=500
W20260809 12:10:00.925032 3158 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260809 12:10:00.925169 3158 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.222.3
I20260809 12:10:00.935379 3158 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.222.20:43453
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskgH5Pjm/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786277382073938-888-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskgH5Pjm/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786277382073938-888-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskgH5Pjm/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786277382073938-888-0/krb5kdc/kudu_127.0.222.3.keytab
--principal=kudu/127.0.222.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.222.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskgH5Pjm/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786277382073938-888-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.222.3
--webserver_port=0
--txn_keepalive_interval_ms=500
--enable_txn_system_client_init=true
--tserver_master_addrs=127.0.222.62:42305
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.222.3
--log_dir=/tmp/dist-test-taskgH5Pjm/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786277382073938-888-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 09 Aug 2026 11:06:41 UTC on ccycloud.quasar-bwoogy.root.comops.site
build id 4575991
ASAN enabled
I20260809 12:10:00.936691 3158 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260809 12:10:00.938041 3158 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260809 12:10:00.950923 3164 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 12:10:00.951283 3163 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 12:10:00.951349 3166 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260809 12:10:00.951691 3158 server_base.cc:1034] running on GCE node
I20260809 12:10:00.952342 3158 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260809 12:10:00.953256 3158 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260809 12:10:00.954486 3158 hybrid_clock.cc:648] HybridClock initialized: now 1786277400954440 us; error 53 us; skew 500 ppm
I20260809 12:10:00.954826 3158 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 09 12:10:00 dist-test-slave-2w3w krb5kdc[2731](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786277400, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.222.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260809 12:10:00.958094 3158 init.cc:377] Logged in from keytab as kudu/127.0.222.3@KRBTEST.COM (short username kudu)
I20260809 12:10:00.959569 3158 webserver.cc:460] Webserver started at http://127.0.222.3:42653/ using document root <none> and password file <none>
I20260809 12:10:00.960189 3158 fs_manager.cc:359] Metadata directory not provided
I20260809 12:10:00.960289 3158 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260809 12:10:00.960547 3158 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260809 12:10:00.963404 3158 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskgH5Pjm/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786277382073938-888-0/minicluster-data/ts-2/data/instance:
uuid: "823d98af597d4559bc2e475e2d08a5df"
format_stamp: "Formatted at 2026-08-09 12:10:00 on dist-test-slave-2w3w"
server_key: "ebfda9589cb485d88cc83be7b3a1fa9e"
server_key_iv: "84b8b23e46d494a356536e38252dff60"
server_key_version: "encryptionkey@0"
I20260809 12:10:00.964164 3158 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskgH5Pjm/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786277382073938-888-0/minicluster-data/ts-2/wal/instance:
uuid: "823d98af597d4559bc2e475e2d08a5df"
format_stamp: "Formatted at 2026-08-09 12:10:00 on dist-test-slave-2w3w"
server_key: "ebfda9589cb485d88cc83be7b3a1fa9e"
server_key_iv: "84b8b23e46d494a356536e38252dff60"
server_key_version: "encryptionkey@0"
I20260809 12:10:00.969313 3158 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.001s sys 0.004s
I20260809 12:10:00.972747 3173 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260809 12:10:00.974148 3158 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.000s
I20260809 12:10:00.974314 3158 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskgH5Pjm/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786277382073938-888-0/minicluster-data/ts-2/data,/tmp/dist-test-taskgH5Pjm/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786277382073938-888-0/minicluster-data/ts-2/wal
uuid: "823d98af597d4559bc2e475e2d08a5df"
format_stamp: "Formatted at 2026-08-09 12:10:00 on dist-test-slave-2w3w"
server_key: "ebfda9589cb485d88cc83be7b3a1fa9e"
server_key_iv: "84b8b23e46d494a356536e38252dff60"
server_key_version: "encryptionkey@0"
I20260809 12:10:00.974562 3158 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskgH5Pjm/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786277382073938-888-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskgH5Pjm/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786277382073938-888-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskgH5Pjm/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786277382073938-888-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260809 12:10:00.992772 3158 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260809 12:10:00.996894 3158 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260809 12:10:00.997213 3158 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260809 12:10:01.000105 3158 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260809 12:10:01.000203 3158 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260809 12:10:01.000360 3158 ts_tablet_manager.cc:610] Registered 0 tablets
I20260809 12:10:01.000495 3158 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260809 12:10:01.013222 3003 heartbeater.cc:503] Master 127.0.222.62:42305 was elected leader, sending a full tablet report...
Aug 09 12:10:01 dist-test-slave-2w3w krb5kdc[2731](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786277400, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.222.3@KRBTEST.COM for kudu/127.0.222.62@KRBTEST.COM
I20260809 12:10:01.029402 2882 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 12:10:01.002146 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.222.3:57319 (local address 127.0.222.62:42305)
0809 12:10:01.002344 (+ 198us) server_negotiation.cc:206] Beginning negotiation
0809 12:10:01.002349 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0809 12:10:01.003160 (+ 811us) server_negotiation.cc:407] Connection header received
0809 12:10:01.006456 (+ 3296us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 12:10:01.006467 (+ 11us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 12:10:01.006567 (+ 100us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 12:10:01.006657 (+ 90us) server_negotiation.cc:226] Negotiated authn=SASL
0809 12:10:01.009799 (+ 3142us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 12:10:01.011075 (+ 1276us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 12:10:01.014390 (+ 3315us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 12:10:01.015190 (+ 800us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 12:10:01.015345 (+ 155us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 12:10:01.020675 (+ 5330us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 12:10:01.020680 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 12:10:01.020683 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0809 12:10:01.020716 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0809 12:10:01.024838 (+ 4122us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 12:10:01.026044 (+ 1206us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 12:10:01.026052 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 12:10:01.026059 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0809 12:10:01.026247 (+ 188us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 12:10:01.027006 (+ 759us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 12:10:01.027015 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 12:10:01.027021 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0809 12:10:01.027461 (+ 440us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 12:10:01.027638 (+ 177us) server_negotiation.cc:1035] Waiting for connection context
0809 12:10:01.028751 (+ 1113us) server_negotiation.cc:299] Negotiation successful
0809 12:10:01.029012 (+ 261us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":60}
I20260809 12:10:01.031091 3183 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 12:10:01.001402 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.222.62:42305 (local address 127.0.222.3:57319)
0809 12:10:01.002911 (+ 1509us) negotiation.cc:107] Waiting for socket to connect
0809 12:10:01.003019 (+ 108us) client_negotiation.cc:174] Beginning negotiation
0809 12:10:01.006015 (+ 2996us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 12:10:01.007337 (+ 1322us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 12:10:01.007351 (+ 14us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 12:10:01.008422 (+ 1071us) client_negotiation.cc:189] Negotiated authn=SASL
0809 12:10:01.009557 (+ 1135us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 12:10:01.009589 (+ 32us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 12:10:01.011269 (+ 1680us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 12:10:01.011278 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 12:10:01.014135 (+ 2857us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 12:10:01.014156 (+ 21us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 12:10:01.015552 (+ 1396us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 12:10:01.015564 (+ 12us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 12:10:01.016007 (+ 443us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 12:10:01.016894 (+ 887us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0809 12:10:01.016937 (+ 43us) client_negotiation.cc:656] calling sasl_client_start()
0809 12:10:01.020403 (+ 3466us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0809 12:10:01.024997 (+ 4594us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0809 12:10:01.025007 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0809 12:10:01.025034 (+ 27us) client_negotiation.cc:760] calling sasl_client_step()
0809 12:10:01.025875 (+ 841us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0809 12:10:01.026448 (+ 573us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0809 12:10:01.026452 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0809 12:10:01.026456 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0809 12:10:01.026803 (+ 347us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0809 12:10:01.027649 (+ 846us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0809 12:10:01.027659 (+ 10us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0809 12:10:01.028208 (+ 549us) client_negotiation.cc:769] Sending connection context
0809 12:10:01.028612 (+ 404us) client_negotiation.cc:240] Negotiation successful
0809 12:10:01.029202 (+ 590us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1147,"thread_start_us":857,"threads_started":1}
I20260809 12:10:01.053679 2882 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 12:10:01.037867 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.222.3:35231 (local address 127.0.222.62:42305)
0809 12:10:01.038093 (+ 226us) server_negotiation.cc:206] Beginning negotiation
0809 12:10:01.038101 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0809 12:10:01.038751 (+ 650us) server_negotiation.cc:407] Connection header received
0809 12:10:01.038947 (+ 196us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 12:10:01.038950 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 12:10:01.039062 (+ 112us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 12:10:01.039168 (+ 106us) server_negotiation.cc:226] Negotiated authn=SASL
0809 12:10:01.040182 (+ 1014us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 12:10:01.041052 (+ 870us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 12:10:01.042391 (+ 1339us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 12:10:01.043037 (+ 646us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 12:10:01.043174 (+ 137us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 12:10:01.045954 (+ 2780us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 12:10:01.045963 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 12:10:01.045968 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0809 12:10:01.046026 (+ 58us) server_negotiation.cc:922] calling sasl_server_start()
0809 12:10:01.050064 (+ 4038us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 12:10:01.051112 (+ 1048us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 12:10:01.051117 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 12:10:01.051121 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0809 12:10:01.051284 (+ 163us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 12:10:01.051778 (+ 494us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 12:10:01.051782 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 12:10:01.051785 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0809 12:10:01.052186 (+ 401us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 12:10:01.052407 (+ 221us) server_negotiation.cc:1035] Waiting for connection context
0809 12:10:01.052979 (+ 572us) server_negotiation.cc:299] Negotiation successful
0809 12:10:01.053253 (+ 274us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":68}
I20260809 12:10:01.054297 3275 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 12:10:01.037875 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.222.62:42305 (local address 127.0.222.3:35231)
0809 12:10:01.038625 (+ 750us) negotiation.cc:107] Waiting for socket to connect
0809 12:10:01.038650 (+ 25us) client_negotiation.cc:174] Beginning negotiation
0809 12:10:01.038821 (+ 171us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 12:10:01.039203 (+ 382us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 12:10:01.039207 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 12:10:01.039737 (+ 530us) client_negotiation.cc:189] Negotiated authn=SASL
0809 12:10:01.040038 (+ 301us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 12:10:01.040050 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 12:10:01.041226 (+ 1176us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 12:10:01.041231 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 12:10:01.042240 (+ 1009us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 12:10:01.042251 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 12:10:01.043285 (+ 1034us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 12:10:01.043296 (+ 11us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 12:10:01.043576 (+ 280us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 12:10:01.044291 (+ 715us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0809 12:10:01.044330 (+ 39us) client_negotiation.cc:656] calling sasl_client_start()
0809 12:10:01.045684 (+ 1354us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0809 12:10:01.050236 (+ 4552us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0809 12:10:01.050244 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0809 12:10:01.050250 (+ 6us) client_negotiation.cc:760] calling sasl_client_step()
0809 12:10:01.050891 (+ 641us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0809 12:10:01.051425 (+ 534us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0809 12:10:01.051430 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0809 12:10:01.051433 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0809 12:10:01.051607 (+ 174us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0809 12:10:01.052358 (+ 751us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0809 12:10:01.052366 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0809 12:10:01.052636 (+ 270us) client_negotiation.cc:769] Sending connection context
0809 12:10:01.052835 (+ 199us) client_negotiation.cc:240] Negotiation successful
0809 12:10:01.053508 (+ 673us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":560,"thread_start_us":394,"threads_started":1}
I20260809 12:10:01.060173 3158 rpc_server.cc:307] RPC server started. Bound to: 127.0.222.3:36309
I20260809 12:10:01.060249 3299 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.222.3:36309 every 8 connection(s)
I20260809 12:10:01.061385 3158 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskgH5Pjm/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786277382073938-888-0/minicluster-data/ts-2/data/info.pb
I20260809 12:10:01.063598 3300 heartbeater.cc:347] Connected to a master server at 127.0.222.62:42305
I20260809 12:10:01.063783 3300 heartbeater.cc:464] Registering TS with master...
I20260809 12:10:01.064363 3300 heartbeater.cc:511] Master 127.0.222.62:42305 requested a full tablet report, sending...
I20260809 12:10:01.065652 2778 ts_manager.cc:194] Registered new tserver with Master: 823d98af597d4559bc2e475e2d08a5df (127.0.222.3:36309)
I20260809 12:10:01.066530 2778 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.222.3@KRBTEST.COM'} at 127.0.222.3:57319
I20260809 12:10:01.068624 888 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskgH5Pjm/build/asan/bin/kudu as pid 3158
I20260809 12:10:01.068779 888 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskgH5Pjm/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786277382073938-888-0/minicluster-data/ts-2/wal/instance
I20260809 12:10:01.069199 888 external_mini_cluster.cc:1428] Setting key c1d78372b69eaff2a6e211cd998bd0b4
I20260809 12:10:01.071853 888 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
Aug 09 12:10:01 dist-test-slave-2w3w krb5kdc[2731](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786277401, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
Aug 09 12:10:01 dist-test-slave-2w3w krb5kdc[2731](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786277401, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.222.62@KRBTEST.COM
I20260809 12:10:01.125820 2882 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 12:10:01.112195 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51984 (local address 127.0.222.62:42305)
0809 12:10:01.112418 (+ 223us) server_negotiation.cc:206] Beginning negotiation
0809 12:10:01.112426 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0809 12:10:01.112608 (+ 182us) server_negotiation.cc:407] Connection header received
0809 12:10:01.112813 (+ 205us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 12:10:01.112823 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 12:10:01.112892 (+ 69us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 12:10:01.112996 (+ 104us) server_negotiation.cc:226] Negotiated authn=SASL
0809 12:10:01.114352 (+ 1356us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 12:10:01.115210 (+ 858us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 12:10:01.116581 (+ 1371us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 12:10:01.116973 (+ 392us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 12:10:01.117092 (+ 119us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 12:10:01.120945 (+ 3853us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 12:10:01.120949 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 12:10:01.120952 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0809 12:10:01.120990 (+ 38us) server_negotiation.cc:922] calling sasl_server_start()
0809 12:10:01.123531 (+ 2541us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 12:10:01.124135 (+ 604us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 12:10:01.124138 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 12:10:01.124141 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0809 12:10:01.124262 (+ 121us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 12:10:01.124561 (+ 299us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 12:10:01.124564 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 12:10:01.124566 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0809 12:10:01.124825 (+ 259us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 12:10:01.125015 (+ 190us) server_negotiation.cc:1035] Waiting for connection context
0809 12:10:01.125253 (+ 238us) server_negotiation.cc:299] Negotiation successful
0809 12:10:01.125447 (+ 194us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":71}
I20260809 12:10:01.129863 2778 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:51984:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260809 12:10:01.137140 2778 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260809 12:10:01.165611 3318 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 12:10:01.149939 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.222.2:37581 (local address 127.0.0.1:48188)
0809 12:10:01.155483 (+ 5544us) negotiation.cc:107] Waiting for socket to connect
0809 12:10:01.155501 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0809 12:10:01.155625 (+ 124us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 12:10:01.157164 (+ 1539us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 12:10:01.157172 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 12:10:01.157207 (+ 35us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0809 12:10:01.157748 (+ 541us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 12:10:01.157764 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 12:10:01.160236 (+ 2472us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 12:10:01.160244 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 12:10:01.163152 (+ 2908us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 12:10:01.163167 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 12:10:01.164591 (+ 1424us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 12:10:01.164598 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 12:10:01.164770 (+ 172us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 12:10:01.164794 (+ 24us) client_negotiation.cc:769] Sending connection context
0809 12:10:01.164916 (+ 122us) client_negotiation.cc:240] Negotiation successful
0809 12:10:01.165148 (+ 232us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":5408,"mutex_wait_us":4,"thread_start_us":5250,"threads_started":1}
I20260809 12:10:01.165896 3319 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 12:10:01.150642 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48188 (local address 127.0.222.2:37581)
0809 12:10:01.156146 (+ 5504us) server_negotiation.cc:206] Beginning negotiation
0809 12:10:01.156156 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0809 12:10:01.156190 (+ 34us) server_negotiation.cc:407] Connection header received
0809 12:10:01.156312 (+ 122us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 12:10:01.156323 (+ 11us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 12:10:01.156590 (+ 267us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 12:10:01.156871 (+ 281us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0809 12:10:01.157994 (+ 1123us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 12:10:01.160002 (+ 2008us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 12:10:01.163347 (+ 3345us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 12:10:01.164353 (+ 1006us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 12:10:01.164503 (+ 150us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 12:10:01.164716 (+ 213us) server_negotiation.cc:1035] Waiting for connection context
0809 12:10:01.165170 (+ 454us) server_negotiation.cc:299] Negotiation successful
0809 12:10:01.165356 (+ 186us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":5343,"thread_start_us":5164,"threads_started":1}
I20260809 12:10:01.166258 3316 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 12:10:01.149640 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.222.1:35257 (local address 127.0.0.1:42438)
0809 12:10:01.152956 (+ 3316us) negotiation.cc:107] Waiting for socket to connect
0809 12:10:01.153002 (+ 46us) client_negotiation.cc:174] Beginning negotiation
0809 12:10:01.153357 (+ 355us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 12:10:01.155710 (+ 2353us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 12:10:01.155721 (+ 11us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 12:10:01.155759 (+ 38us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0809 12:10:01.156182 (+ 423us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 12:10:01.156196 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 12:10:01.158914 (+ 2718us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 12:10:01.158923 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 12:10:01.162108 (+ 3185us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 12:10:01.162126 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 12:10:01.164412 (+ 2286us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 12:10:01.164417 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 12:10:01.164715 (+ 298us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 12:10:01.164778 (+ 63us) client_negotiation.cc:769] Sending connection context
0809 12:10:01.164916 (+ 138us) client_negotiation.cc:240] Negotiation successful
0809 12:10:01.165841 (+ 925us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":2880,"mutex_wait_us":743,"thread_start_us":2748,"threads_started":1}
I20260809 12:10:01.166958 3317 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 12:10:01.149814 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42438 (local address 127.0.222.1:35257)
0809 12:10:01.154616 (+ 4802us) server_negotiation.cc:206] Beginning negotiation
0809 12:10:01.154629 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0809 12:10:01.154665 (+ 36us) server_negotiation.cc:407] Connection header received
0809 12:10:01.154803 (+ 138us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 12:10:01.154814 (+ 11us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 12:10:01.155157 (+ 343us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 12:10:01.155407 (+ 250us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0809 12:10:01.156470 (+ 1063us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 12:10:01.158707 (+ 2237us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 12:10:01.162327 (+ 3620us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 12:10:01.163897 (+ 1570us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 12:10:01.164024 (+ 127us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 12:10:01.164277 (+ 253us) server_negotiation.cc:1035] Waiting for connection context
0809 12:10:01.166420 (+ 2143us) server_negotiation.cc:299] Negotiation successful
0809 12:10:01.166628 (+ 208us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":4584,"thread_start_us":338,"threads_started":1}
I20260809 12:10:01.169731 3082 tablet_service.cc:1467] Processing CreateTablet for tablet 79f986af492747e8911414a8dd4f5d18 (DEFAULT_TABLE table=test-table [id=a8bc98782b0d4ebfaa09425c27627bd3]), partition=RANGE (key) PARTITION UNBOUNDED
I20260809 12:10:01.170926 3321 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 12:10:01.156194 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.222.3:36309 (local address 127.0.0.1:42174)
0809 12:10:01.162647 (+ 6453us) negotiation.cc:107] Waiting for socket to connect
0809 12:10:01.162665 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0809 12:10:01.162796 (+ 131us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 12:10:01.164037 (+ 1241us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 12:10:01.164042 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 12:10:01.164069 (+ 27us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0809 12:10:01.164545 (+ 476us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 12:10:01.164559 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 12:10:01.166857 (+ 2298us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 12:10:01.166863 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 12:10:01.168717 (+ 1854us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 12:10:01.168732 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 12:10:01.170254 (+ 1522us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 12:10:01.170260 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 12:10:01.170441 (+ 181us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 12:10:01.170476 (+ 35us) client_negotiation.cc:769] Sending connection context
0809 12:10:01.170566 (+ 90us) client_negotiation.cc:240] Negotiation successful
0809 12:10:01.170661 (+ 95us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":6327,"mutex_wait_us":1,"thread_start_us":6033,"threads_started":1}
I20260809 12:10:01.170471 2928 tablet_service.cc:1467] Processing CreateTablet for tablet 79f986af492747e8911414a8dd4f5d18 (DEFAULT_TABLE table=test-table [id=a8bc98782b0d4ebfaa09425c27627bd3]), partition=RANGE (key) PARTITION UNBOUNDED
I20260809 12:10:01.171341 3082 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 79f986af492747e8911414a8dd4f5d18. 1 dirs total, 0 dirs full, 0 dirs failed
I20260809 12:10:01.171697 3320 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 12:10:01.156001 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42174 (local address 127.0.222.3:36309)
0809 12:10:01.163423 (+ 7422us) server_negotiation.cc:206] Beginning negotiation
0809 12:10:01.163431 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0809 12:10:01.163465 (+ 34us) server_negotiation.cc:407] Connection header received
0809 12:10:01.163556 (+ 91us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 12:10:01.163563 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 12:10:01.163820 (+ 257us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 12:10:01.164025 (+ 205us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0809 12:10:01.165059 (+ 1034us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 12:10:01.166641 (+ 1582us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 12:10:01.168918 (+ 2277us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 12:10:01.170051 (+ 1133us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 12:10:01.170218 (+ 167us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 12:10:01.170457 (+ 239us) server_negotiation.cc:1035] Waiting for connection context
0809 12:10:01.171294 (+ 837us) server_negotiation.cc:299] Negotiation successful
0809 12:10:01.171452 (+ 158us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":7242,"thread_start_us":7077,"threads_started":1}
I20260809 12:10:01.171828 2928 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 79f986af492747e8911414a8dd4f5d18. 1 dirs total, 0 dirs full, 0 dirs failed
I20260809 12:10:01.174953 3227 tablet_service.cc:1467] Processing CreateTablet for tablet 79f986af492747e8911414a8dd4f5d18 (DEFAULT_TABLE table=test-table [id=a8bc98782b0d4ebfaa09425c27627bd3]), partition=RANGE (key) PARTITION UNBOUNDED
I20260809 12:10:01.176069 3227 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 79f986af492747e8911414a8dd4f5d18. 1 dirs total, 0 dirs full, 0 dirs failed
I20260809 12:10:01.186694 3323 tablet_bootstrap.cc:492] T 79f986af492747e8911414a8dd4f5d18 P 9cea8553e4754cd89ec242d7c7a78577: Bootstrap starting.
I20260809 12:10:01.187621 3322 tablet_bootstrap.cc:492] T 79f986af492747e8911414a8dd4f5d18 P 26d0a424d542436a8830dbe1f8d5142e: Bootstrap starting.
I20260809 12:10:01.188731 3324 tablet_bootstrap.cc:492] T 79f986af492747e8911414a8dd4f5d18 P 823d98af597d4559bc2e475e2d08a5df: Bootstrap starting.
I20260809 12:10:01.190891 3322 tablet_bootstrap.cc:654] T 79f986af492747e8911414a8dd4f5d18 P 26d0a424d542436a8830dbe1f8d5142e: Neither blocks nor log segments found. Creating new log.
I20260809 12:10:01.191731 3324 tablet_bootstrap.cc:654] T 79f986af492747e8911414a8dd4f5d18 P 823d98af597d4559bc2e475e2d08a5df: Neither blocks nor log segments found. Creating new log.
I20260809 12:10:01.192824 3324 log.cc:824] T 79f986af492747e8911414a8dd4f5d18 P 823d98af597d4559bc2e475e2d08a5df: Log is configured to *not* fsync() on all Append() calls
I20260809 12:10:01.193841 3323 tablet_bootstrap.cc:654] T 79f986af492747e8911414a8dd4f5d18 P 9cea8553e4754cd89ec242d7c7a78577: Neither blocks nor log segments found. Creating new log.
I20260809 12:10:01.194624 3322 log.cc:824] T 79f986af492747e8911414a8dd4f5d18 P 26d0a424d542436a8830dbe1f8d5142e: Log is configured to *not* fsync() on all Append() calls
I20260809 12:10:01.195103 3323 log.cc:824] T 79f986af492747e8911414a8dd4f5d18 P 9cea8553e4754cd89ec242d7c7a78577: Log is configured to *not* fsync() on all Append() calls
I20260809 12:10:01.196269 3324 tablet_bootstrap.cc:492] T 79f986af492747e8911414a8dd4f5d18 P 823d98af597d4559bc2e475e2d08a5df: No bootstrap required, opened a new log
I20260809 12:10:01.196594 3324 ts_tablet_manager.cc:1397] T 79f986af492747e8911414a8dd4f5d18 P 823d98af597d4559bc2e475e2d08a5df: Time spent bootstrapping tablet: real 0.008s user 0.007s sys 0.000s
I20260809 12:10:01.197633 3323 tablet_bootstrap.cc:492] T 79f986af492747e8911414a8dd4f5d18 P 9cea8553e4754cd89ec242d7c7a78577: No bootstrap required, opened a new log
I20260809 12:10:01.197799 3322 tablet_bootstrap.cc:492] T 79f986af492747e8911414a8dd4f5d18 P 26d0a424d542436a8830dbe1f8d5142e: No bootstrap required, opened a new log
I20260809 12:10:01.197988 3323 ts_tablet_manager.cc:1397] T 79f986af492747e8911414a8dd4f5d18 P 9cea8553e4754cd89ec242d7c7a78577: Time spent bootstrapping tablet: real 0.012s user 0.005s sys 0.000s
I20260809 12:10:01.198066 3322 ts_tablet_manager.cc:1397] T 79f986af492747e8911414a8dd4f5d18 P 26d0a424d542436a8830dbe1f8d5142e: Time spent bootstrapping tablet: real 0.011s user 0.004s sys 0.002s
I20260809 12:10:01.205840 3323 raft_consensus.cc:348] T 79f986af492747e8911414a8dd4f5d18 P 9cea8553e4754cd89ec242d7c7a78577 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9cea8553e4754cd89ec242d7c7a78577" member_type: VOTER last_known_addr { host: "127.0.222.1" port: 35257 } } peers { permanent_uuid: "26d0a424d542436a8830dbe1f8d5142e" member_type: VOTER last_known_addr { host: "127.0.222.2" port: 37581 } } peers { permanent_uuid: "823d98af597d4559bc2e475e2d08a5df" member_type: VOTER last_known_addr { host: "127.0.222.3" port: 36309 } }
I20260809 12:10:01.206231 3323 raft_consensus.cc:374] T 79f986af492747e8911414a8dd4f5d18 P 9cea8553e4754cd89ec242d7c7a78577 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260809 12:10:01.206390 3323 raft_consensus.cc:729] T 79f986af492747e8911414a8dd4f5d18 P 9cea8553e4754cd89ec242d7c7a78577 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9cea8553e4754cd89ec242d7c7a78577, State: Initialized, Role: FOLLOWER
I20260809 12:10:01.207079 3323 consensus_queue.cc:260] T 79f986af492747e8911414a8dd4f5d18 P 9cea8553e4754cd89ec242d7c7a78577 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9cea8553e4754cd89ec242d7c7a78577" member_type: VOTER last_known_addr { host: "127.0.222.1" port: 35257 } } peers { permanent_uuid: "26d0a424d542436a8830dbe1f8d5142e" member_type: VOTER last_known_addr { host: "127.0.222.2" port: 37581 } } peers { permanent_uuid: "823d98af597d4559bc2e475e2d08a5df" member_type: VOTER last_known_addr { host: "127.0.222.3" port: 36309 } }
I20260809 12:10:01.207736 3324 raft_consensus.cc:348] T 79f986af492747e8911414a8dd4f5d18 P 823d98af597d4559bc2e475e2d08a5df [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9cea8553e4754cd89ec242d7c7a78577" member_type: VOTER last_known_addr { host: "127.0.222.1" port: 35257 } } peers { permanent_uuid: "26d0a424d542436a8830dbe1f8d5142e" member_type: VOTER last_known_addr { host: "127.0.222.2" port: 37581 } } peers { permanent_uuid: "823d98af597d4559bc2e475e2d08a5df" member_type: VOTER last_known_addr { host: "127.0.222.3" port: 36309 } }
I20260809 12:10:01.208220 3324 raft_consensus.cc:374] T 79f986af492747e8911414a8dd4f5d18 P 823d98af597d4559bc2e475e2d08a5df [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260809 12:10:01.208376 3324 raft_consensus.cc:729] T 79f986af492747e8911414a8dd4f5d18 P 823d98af597d4559bc2e475e2d08a5df [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 823d98af597d4559bc2e475e2d08a5df, State: Initialized, Role: FOLLOWER
I20260809 12:10:01.208640 3322 raft_consensus.cc:348] T 79f986af492747e8911414a8dd4f5d18 P 26d0a424d542436a8830dbe1f8d5142e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9cea8553e4754cd89ec242d7c7a78577" member_type: VOTER last_known_addr { host: "127.0.222.1" port: 35257 } } peers { permanent_uuid: "26d0a424d542436a8830dbe1f8d5142e" member_type: VOTER last_known_addr { host: "127.0.222.2" port: 37581 } } peers { permanent_uuid: "823d98af597d4559bc2e475e2d08a5df" member_type: VOTER last_known_addr { host: "127.0.222.3" port: 36309 } }
I20260809 12:10:01.209038 3323 ts_tablet_manager.cc:1428] T 79f986af492747e8911414a8dd4f5d18 P 9cea8553e4754cd89ec242d7c7a78577: Time spent starting tablet: real 0.011s user 0.006s sys 0.005s
I20260809 12:10:01.209082 3322 raft_consensus.cc:374] T 79f986af492747e8911414a8dd4f5d18 P 26d0a424d542436a8830dbe1f8d5142e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260809 12:10:01.209244 3322 raft_consensus.cc:729] T 79f986af492747e8911414a8dd4f5d18 P 26d0a424d542436a8830dbe1f8d5142e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 26d0a424d542436a8830dbe1f8d5142e, State: Initialized, Role: FOLLOWER
I20260809 12:10:01.209233 3324 consensus_queue.cc:260] T 79f986af492747e8911414a8dd4f5d18 P 823d98af597d4559bc2e475e2d08a5df [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9cea8553e4754cd89ec242d7c7a78577" member_type: VOTER last_known_addr { host: "127.0.222.1" port: 35257 } } peers { permanent_uuid: "26d0a424d542436a8830dbe1f8d5142e" member_type: VOTER last_known_addr { host: "127.0.222.2" port: 37581 } } peers { permanent_uuid: "823d98af597d4559bc2e475e2d08a5df" member_type: VOTER last_known_addr { host: "127.0.222.3" port: 36309 } }
I20260809 12:10:01.210047 3322 consensus_queue.cc:260] T 79f986af492747e8911414a8dd4f5d18 P 26d0a424d542436a8830dbe1f8d5142e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9cea8553e4754cd89ec242d7c7a78577" member_type: VOTER last_known_addr { host: "127.0.222.1" port: 35257 } } peers { permanent_uuid: "26d0a424d542436a8830dbe1f8d5142e" member_type: VOTER last_known_addr { host: "127.0.222.2" port: 37581 } } peers { permanent_uuid: "823d98af597d4559bc2e475e2d08a5df" member_type: VOTER last_known_addr { host: "127.0.222.3" port: 36309 } }
I20260809 12:10:01.210510 3300 heartbeater.cc:503] Master 127.0.222.62:42305 was elected leader, sending a full tablet report...
I20260809 12:10:01.211265 3324 ts_tablet_manager.cc:1428] T 79f986af492747e8911414a8dd4f5d18 P 823d98af597d4559bc2e475e2d08a5df: Time spent starting tablet: real 0.015s user 0.013s sys 0.001s
I20260809 12:10:01.217087 3154 heartbeater.cc:503] Master 127.0.222.62:42305 was elected leader, sending a full tablet report...
I20260809 12:10:01.217760 3322 ts_tablet_manager.cc:1428] T 79f986af492747e8911414a8dd4f5d18 P 26d0a424d542436a8830dbe1f8d5142e: Time spent starting tablet: real 0.020s user 0.012s sys 0.008s
W20260809 12:10:01.255396 3004 tablet.cc:2369] T 79f986af492747e8911414a8dd4f5d18 P 9cea8553e4754cd89ec242d7c7a78577: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260809 12:10:01.286563 3155 tablet.cc:2369] T 79f986af492747e8911414a8dd4f5d18 P 26d0a424d542436a8830dbe1f8d5142e: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260809 12:10:01.313148 3301 tablet.cc:2369] T 79f986af492747e8911414a8dd4f5d18 P 823d98af597d4559bc2e475e2d08a5df: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260809 12:10:01.552376 3328 raft_consensus.cc:482] T 79f986af492747e8911414a8dd4f5d18 P 9cea8553e4754cd89ec242d7c7a78577 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260809 12:10:01.552665 3328 raft_consensus.cc:504] T 79f986af492747e8911414a8dd4f5d18 P 9cea8553e4754cd89ec242d7c7a78577 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9cea8553e4754cd89ec242d7c7a78577" member_type: VOTER last_known_addr { host: "127.0.222.1" port: 35257 } } peers { permanent_uuid: "26d0a424d542436a8830dbe1f8d5142e" member_type: VOTER last_known_addr { host: "127.0.222.2" port: 37581 } } peers { permanent_uuid: "823d98af597d4559bc2e475e2d08a5df" member_type: VOTER last_known_addr { host: "127.0.222.3" port: 36309 } }
I20260809 12:10:01.554585 3328 leader_election.cc:288] T 79f986af492747e8911414a8dd4f5d18 P 9cea8553e4754cd89ec242d7c7a78577 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 26d0a424d542436a8830dbe1f8d5142e (127.0.222.2:37581), 823d98af597d4559bc2e475e2d08a5df (127.0.222.3:36309)
I20260809 12:10:01.562495 3319 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 12:10:01.555215 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.222.1:35209 (local address 127.0.222.2:37581)
0809 12:10:01.555466 (+ 251us) server_negotiation.cc:206] Beginning negotiation
0809 12:10:01.555476 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0809 12:10:01.556014 (+ 538us) server_negotiation.cc:407] Connection header received
0809 12:10:01.556335 (+ 321us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 12:10:01.556343 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 12:10:01.556431 (+ 88us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 12:10:01.556489 (+ 58us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0809 12:10:01.557501 (+ 1012us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 12:10:01.558306 (+ 805us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 12:10:01.560602 (+ 2296us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 12:10:01.561407 (+ 805us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 12:10:01.561521 (+ 114us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 12:10:01.561602 (+ 81us) server_negotiation.cc:1035] Waiting for connection context
0809 12:10:01.562073 (+ 471us) server_negotiation.cc:299] Negotiation successful
0809 12:10:01.562210 (+ 137us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":67}
I20260809 12:10:01.562572 3320 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 12:10:01.556124 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.222.1:60919 (local address 127.0.222.3:36309)
0809 12:10:01.556344 (+ 220us) server_negotiation.cc:206] Beginning negotiation
0809 12:10:01.556350 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0809 12:10:01.556701 (+ 351us) server_negotiation.cc:407] Connection header received
0809 12:10:01.556852 (+ 151us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 12:10:01.556858 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 12:10:01.556946 (+ 88us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 12:10:01.557080 (+ 134us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0809 12:10:01.557756 (+ 676us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 12:10:01.558567 (+ 811us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 12:10:01.560700 (+ 2133us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 12:10:01.561596 (+ 896us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 12:10:01.561743 (+ 147us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 12:10:01.561859 (+ 116us) server_negotiation.cc:1035] Waiting for connection context
0809 12:10:01.562113 (+ 254us) server_negotiation.cc:299] Negotiation successful
0809 12:10:01.562245 (+ 132us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":81}
I20260809 12:10:01.562712 3331 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 12:10:01.555136 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.222.2:37581 (local address 127.0.222.1:35209)
0809 12:10:01.555874 (+ 738us) negotiation.cc:107] Waiting for socket to connect
0809 12:10:01.555893 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0809 12:10:01.556100 (+ 207us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 12:10:01.556703 (+ 603us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 12:10:01.556708 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 12:10:01.556743 (+ 35us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0809 12:10:01.557317 (+ 574us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 12:10:01.557334 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 12:10:01.558446 (+ 1112us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 12:10:01.558452 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 12:10:01.560441 (+ 1989us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 12:10:01.560455 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 12:10:01.561645 (+ 1190us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 12:10:01.561655 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 12:10:01.561863 (+ 208us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 12:10:01.561927 (+ 64us) client_negotiation.cc:769] Sending connection context
0809 12:10:01.562009 (+ 82us) client_negotiation.cc:240] Negotiation successful
0809 12:10:01.562185 (+ 176us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":555,"mutex_wait_us":34,"thread_start_us":377,"threads_started":1}
I20260809 12:10:01.562757 3332 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 12:10:01.555964 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.222.3:36309 (local address 127.0.222.1:60919)
0809 12:10:01.556593 (+ 629us) negotiation.cc:107] Waiting for socket to connect
0809 12:10:01.556607 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0809 12:10:01.556721 (+ 114us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 12:10:01.557123 (+ 402us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 12:10:01.557126 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 12:10:01.557150 (+ 24us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0809 12:10:01.557592 (+ 442us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 12:10:01.557606 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 12:10:01.558744 (+ 1138us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 12:10:01.558748 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 12:10:01.560512 (+ 1764us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 12:10:01.560543 (+ 31us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 12:10:01.561814 (+ 1271us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 12:10:01.561818 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 12:10:01.561929 (+ 111us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 12:10:01.561950 (+ 21us) client_negotiation.cc:769] Sending connection context
0809 12:10:01.562062 (+ 112us) client_negotiation.cc:240] Negotiation successful
0809 12:10:01.562219 (+ 157us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":529,"mutex_wait_us":26,"thread_start_us":355,"threads_started":1}
I20260809 12:10:01.563946 3250 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "79f986af492747e8911414a8dd4f5d18" candidate_uuid: "9cea8553e4754cd89ec242d7c7a78577" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "823d98af597d4559bc2e475e2d08a5df" is_pre_election: true
I20260809 12:10:01.563952 3106 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "79f986af492747e8911414a8dd4f5d18" candidate_uuid: "9cea8553e4754cd89ec242d7c7a78577" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "26d0a424d542436a8830dbe1f8d5142e" is_pre_election: true
I20260809 12:10:01.564427 3250 raft_consensus.cc:2454] T 79f986af492747e8911414a8dd4f5d18 P 823d98af597d4559bc2e475e2d08a5df [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9cea8553e4754cd89ec242d7c7a78577 in term 0.
I20260809 12:10:01.564460 3106 raft_consensus.cc:2454] T 79f986af492747e8911414a8dd4f5d18 P 26d0a424d542436a8830dbe1f8d5142e [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9cea8553e4754cd89ec242d7c7a78577 in term 0.
I20260809 12:10:01.565056 2873 leader_election.cc:302] T 79f986af492747e8911414a8dd4f5d18 P 9cea8553e4754cd89ec242d7c7a78577 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 823d98af597d4559bc2e475e2d08a5df, 9cea8553e4754cd89ec242d7c7a78577; no voters:
I20260809 12:10:01.565451 3328 raft_consensus.cc:2781] T 79f986af492747e8911414a8dd4f5d18 P 9cea8553e4754cd89ec242d7c7a78577 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260809 12:10:01.565586 3328 raft_consensus.cc:482] T 79f986af492747e8911414a8dd4f5d18 P 9cea8553e4754cd89ec242d7c7a78577 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260809 12:10:01.565685 3328 raft_consensus.cc:3037] T 79f986af492747e8911414a8dd4f5d18 P 9cea8553e4754cd89ec242d7c7a78577 [term 0 FOLLOWER]: Advancing to term 1
I20260809 12:10:01.568192 3328 raft_consensus.cc:504] T 79f986af492747e8911414a8dd4f5d18 P 9cea8553e4754cd89ec242d7c7a78577 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9cea8553e4754cd89ec242d7c7a78577" member_type: VOTER last_known_addr { host: "127.0.222.1" port: 35257 } } peers { permanent_uuid: "26d0a424d542436a8830dbe1f8d5142e" member_type: VOTER last_known_addr { host: "127.0.222.2" port: 37581 } } peers { permanent_uuid: "823d98af597d4559bc2e475e2d08a5df" member_type: VOTER last_known_addr { host: "127.0.222.3" port: 36309 } }
I20260809 12:10:01.568857 3328 leader_election.cc:288] T 79f986af492747e8911414a8dd4f5d18 P 9cea8553e4754cd89ec242d7c7a78577 [CANDIDATE]: Term 1 election: Requested vote from peers 26d0a424d542436a8830dbe1f8d5142e (127.0.222.2:37581), 823d98af597d4559bc2e475e2d08a5df (127.0.222.3:36309)
I20260809 12:10:01.569234 3106 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "79f986af492747e8911414a8dd4f5d18" candidate_uuid: "9cea8553e4754cd89ec242d7c7a78577" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "26d0a424d542436a8830dbe1f8d5142e"
I20260809 12:10:01.569329 3250 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "79f986af492747e8911414a8dd4f5d18" candidate_uuid: "9cea8553e4754cd89ec242d7c7a78577" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "823d98af597d4559bc2e475e2d08a5df"
I20260809 12:10:01.569460 3106 raft_consensus.cc:3037] T 79f986af492747e8911414a8dd4f5d18 P 26d0a424d542436a8830dbe1f8d5142e [term 0 FOLLOWER]: Advancing to term 1
I20260809 12:10:01.569563 3250 raft_consensus.cc:3037] T 79f986af492747e8911414a8dd4f5d18 P 823d98af597d4559bc2e475e2d08a5df [term 0 FOLLOWER]: Advancing to term 1
I20260809 12:10:01.571714 3250 raft_consensus.cc:2454] T 79f986af492747e8911414a8dd4f5d18 P 823d98af597d4559bc2e475e2d08a5df [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9cea8553e4754cd89ec242d7c7a78577 in term 1.
I20260809 12:10:01.572019 3106 raft_consensus.cc:2454] T 79f986af492747e8911414a8dd4f5d18 P 26d0a424d542436a8830dbe1f8d5142e [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9cea8553e4754cd89ec242d7c7a78577 in term 1.
I20260809 12:10:01.572134 2873 leader_election.cc:302] T 79f986af492747e8911414a8dd4f5d18 P 9cea8553e4754cd89ec242d7c7a78577 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 823d98af597d4559bc2e475e2d08a5df, 9cea8553e4754cd89ec242d7c7a78577; no voters:
I20260809 12:10:01.572520 3328 raft_consensus.cc:2781] T 79f986af492747e8911414a8dd4f5d18 P 9cea8553e4754cd89ec242d7c7a78577 [term 1 FOLLOWER]: Leader election won for term 1
I20260809 12:10:01.572973 3328 raft_consensus.cc:686] T 79f986af492747e8911414a8dd4f5d18 P 9cea8553e4754cd89ec242d7c7a78577 [term 1 LEADER]: Becoming Leader. State: Replica: 9cea8553e4754cd89ec242d7c7a78577, State: Running, Role: LEADER
I20260809 12:10:01.573531 3328 consensus_queue.cc:237] T 79f986af492747e8911414a8dd4f5d18 P 9cea8553e4754cd89ec242d7c7a78577 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9cea8553e4754cd89ec242d7c7a78577" member_type: VOTER last_known_addr { host: "127.0.222.1" port: 35257 } } peers { permanent_uuid: "26d0a424d542436a8830dbe1f8d5142e" member_type: VOTER last_known_addr { host: "127.0.222.2" port: 37581 } } peers { permanent_uuid: "823d98af597d4559bc2e475e2d08a5df" member_type: VOTER last_known_addr { host: "127.0.222.3" port: 36309 } }
I20260809 12:10:01.578753 2778 catalog_manager.cc:5495] T 79f986af492747e8911414a8dd4f5d18 P 9cea8553e4754cd89ec242d7c7a78577 reported cstate change: term changed from 0 to 1, leader changed from <none> to 9cea8553e4754cd89ec242d7c7a78577 (127.0.222.1). New cstate: current_term: 1 leader_uuid: "9cea8553e4754cd89ec242d7c7a78577" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9cea8553e4754cd89ec242d7c7a78577" member_type: VOTER last_known_addr { host: "127.0.222.1" port: 35257 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "26d0a424d542436a8830dbe1f8d5142e" member_type: VOTER last_known_addr { host: "127.0.222.2" port: 37581 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "823d98af597d4559bc2e475e2d08a5df" member_type: VOTER last_known_addr { host: "127.0.222.3" port: 36309 } health_report { overall_health: UNKNOWN } } }
Aug 09 12:10:01 dist-test-slave-2w3w krb5kdc[2731](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786277399, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.222.62@KRBTEST.COM for kudu/127.0.222.62@KRBTEST.COM
I20260809 12:10:01.650839 3340 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 12:10:01.637494 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.222.62:42305 (local address 127.0.0.1:51994)
0809 12:10:01.638263 (+ 769us) negotiation.cc:107] Waiting for socket to connect
0809 12:10:01.638293 (+ 30us) client_negotiation.cc:174] Beginning negotiation
0809 12:10:01.638525 (+ 232us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 12:10:01.638829 (+ 304us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 12:10:01.638832 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 12:10:01.639574 (+ 742us) client_negotiation.cc:189] Negotiated authn=SASL
0809 12:10:01.639828 (+ 254us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 12:10:01.639837 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 12:10:01.640913 (+ 1076us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 12:10:01.640916 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 12:10:01.641847 (+ 931us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 12:10:01.641857 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 12:10:01.642550 (+ 693us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 12:10:01.642555 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 12:10:01.642686 (+ 131us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 12:10:01.643402 (+ 716us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0809 12:10:01.643418 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0809 12:10:01.645627 (+ 2209us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0809 12:10:01.648575 (+ 2948us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0809 12:10:01.648585 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0809 12:10:01.648591 (+ 6us) client_negotiation.cc:760] calling sasl_client_step()
0809 12:10:01.648976 (+ 385us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0809 12:10:01.649304 (+ 328us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0809 12:10:01.649307 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0809 12:10:01.649309 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0809 12:10:01.649464 (+ 155us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0809 12:10:01.649977 (+ 513us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0809 12:10:01.649984 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0809 12:10:01.650129 (+ 145us) client_negotiation.cc:769] Sending connection context
0809 12:10:01.650265 (+ 136us) client_negotiation.cc:240] Negotiation successful
0809 12:10:01.650463 (+ 198us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":550,"thread_start_us":333,"threads_started":1}
I20260809 12:10:01.651072 3341 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 12:10:01.637531 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51994 (local address 127.0.222.62:42305)
0809 12:10:01.638419 (+ 888us) server_negotiation.cc:206] Beginning negotiation
0809 12:10:01.638423 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0809 12:10:01.638449 (+ 26us) server_negotiation.cc:407] Connection header received
0809 12:10:01.638633 (+ 184us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 12:10:01.638636 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 12:10:01.638692 (+ 56us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 12:10:01.638795 (+ 103us) server_negotiation.cc:226] Negotiated authn=SASL
0809 12:10:01.639979 (+ 1184us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 12:10:01.640808 (+ 829us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 12:10:01.641979 (+ 1171us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 12:10:01.642382 (+ 403us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 12:10:01.642474 (+ 92us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 12:10:01.645839 (+ 3365us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 12:10:01.645844 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 12:10:01.645847 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0809 12:10:01.645877 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0809 12:10:01.648458 (+ 2581us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 12:10:01.649079 (+ 621us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 12:10:01.649083 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 12:10:01.649086 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0809 12:10:01.649206 (+ 120us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 12:10:01.649575 (+ 369us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 12:10:01.649578 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 12:10:01.649581 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0809 12:10:01.649859 (+ 278us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 12:10:01.649989 (+ 130us) server_negotiation.cc:1035] Waiting for connection context
0809 12:10:01.650366 (+ 377us) server_negotiation.cc:299] Negotiation successful
0809 12:10:01.650621 (+ 255us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":47,"server-negotiator.queue_time_us":774,"thread_start_us":656,"threads_started":1}
I20260809 12:10:01.656292 2778 catalog_manager.cc:1908] Servicing CreateTable request from {username='kudu', principal='kudu/127.0.222.62@KRBTEST.COM'} at 127.0.0.1:51994:
name: "kudu_system.kudu_transactions"
schema {
columns {
name: "txn_id"
type: INT64
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "entry_type"
type: INT8
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "identifier"
type: STRING
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "metadata"
type: STRING
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
rows: "<redacted>""\006\001\000\000\000\000\000\000\000\000\007\001@B\017\000\000\000\000\000"
indirect_data: "<redacted>"""
}
partition_schema {
range_schema {
columns {
name: "txn_id"
}
}
}
table_type: TXN_STATUS_TABLE
W20260809 12:10:01.657321 2778 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table kudu_system.kudu_transactions in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260809 12:10:01.664228 3082 tablet_service.cc:1467] Processing CreateTablet for tablet 18c21b711bbe46e08a1f1609cf2b8a50 (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=ff2fce38373e422e8ca03a4cb8319603]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260809 12:10:01.664069 2928 tablet_service.cc:1467] Processing CreateTablet for tablet 18c21b711bbe46e08a1f1609cf2b8a50 (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=ff2fce38373e422e8ca03a4cb8319603]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260809 12:10:01.665100 2928 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 18c21b711bbe46e08a1f1609cf2b8a50. 1 dirs total, 0 dirs full, 0 dirs failed
I20260809 12:10:01.665102 3082 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 18c21b711bbe46e08a1f1609cf2b8a50. 1 dirs total, 0 dirs full, 0 dirs failed
I20260809 12:10:01.664572 3227 tablet_service.cc:1467] Processing CreateTablet for tablet 18c21b711bbe46e08a1f1609cf2b8a50 (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=ff2fce38373e422e8ca03a4cb8319603]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260809 12:10:01.665464 3227 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 18c21b711bbe46e08a1f1609cf2b8a50. 1 dirs total, 0 dirs full, 0 dirs failed
I20260809 12:10:01.671988 3324 tablet_bootstrap.cc:492] T 18c21b711bbe46e08a1f1609cf2b8a50 P 823d98af597d4559bc2e475e2d08a5df: Bootstrap starting.
I20260809 12:10:01.672312 3322 tablet_bootstrap.cc:492] T 18c21b711bbe46e08a1f1609cf2b8a50 P 26d0a424d542436a8830dbe1f8d5142e: Bootstrap starting.
I20260809 12:10:01.672312 3323 tablet_bootstrap.cc:492] T 18c21b711bbe46e08a1f1609cf2b8a50 P 9cea8553e4754cd89ec242d7c7a78577: Bootstrap starting.
I20260809 12:10:01.674045 3324 tablet_bootstrap.cc:654] T 18c21b711bbe46e08a1f1609cf2b8a50 P 823d98af597d4559bc2e475e2d08a5df: Neither blocks nor log segments found. Creating new log.
I20260809 12:10:01.674048 3322 tablet_bootstrap.cc:654] T 18c21b711bbe46e08a1f1609cf2b8a50 P 26d0a424d542436a8830dbe1f8d5142e: Neither blocks nor log segments found. Creating new log.
I20260809 12:10:01.674089 3323 tablet_bootstrap.cc:654] T 18c21b711bbe46e08a1f1609cf2b8a50 P 9cea8553e4754cd89ec242d7c7a78577: Neither blocks nor log segments found. Creating new log.
I20260809 12:10:01.680075 3324 tablet_bootstrap.cc:492] T 18c21b711bbe46e08a1f1609cf2b8a50 P 823d98af597d4559bc2e475e2d08a5df: No bootstrap required, opened a new log
I20260809 12:10:01.680266 3324 ts_tablet_manager.cc:1397] T 18c21b711bbe46e08a1f1609cf2b8a50 P 823d98af597d4559bc2e475e2d08a5df: Time spent bootstrapping tablet: real 0.008s user 0.004s sys 0.002s
I20260809 12:10:01.680959 3324 raft_consensus.cc:348] T 18c21b711bbe46e08a1f1609cf2b8a50 P 823d98af597d4559bc2e475e2d08a5df [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9cea8553e4754cd89ec242d7c7a78577" member_type: VOTER last_known_addr { host: "127.0.222.1" port: 35257 } } peers { permanent_uuid: "26d0a424d542436a8830dbe1f8d5142e" member_type: VOTER last_known_addr { host: "127.0.222.2" port: 37581 } } peers { permanent_uuid: "823d98af597d4559bc2e475e2d08a5df" member_type: VOTER last_known_addr { host: "127.0.222.3" port: 36309 } }
I20260809 12:10:01.681198 3324 raft_consensus.cc:374] T 18c21b711bbe46e08a1f1609cf2b8a50 P 823d98af597d4559bc2e475e2d08a5df [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260809 12:10:01.681325 3324 raft_consensus.cc:729] T 18c21b711bbe46e08a1f1609cf2b8a50 P 823d98af597d4559bc2e475e2d08a5df [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 823d98af597d4559bc2e475e2d08a5df, State: Initialized, Role: FOLLOWER
I20260809 12:10:01.681594 3324 consensus_queue.cc:260] T 18c21b711bbe46e08a1f1609cf2b8a50 P 823d98af597d4559bc2e475e2d08a5df [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9cea8553e4754cd89ec242d7c7a78577" member_type: VOTER last_known_addr { host: "127.0.222.1" port: 35257 } } peers { permanent_uuid: "26d0a424d542436a8830dbe1f8d5142e" member_type: VOTER last_known_addr { host: "127.0.222.2" port: 37581 } } peers { permanent_uuid: "823d98af597d4559bc2e475e2d08a5df" member_type: VOTER last_known_addr { host: "127.0.222.3" port: 36309 } }
I20260809 12:10:01.682215 3329 tablet_replica.cc:439] T 18c21b711bbe46e08a1f1609cf2b8a50 P 823d98af597d4559bc2e475e2d08a5df: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9cea8553e4754cd89ec242d7c7a78577" member_type: VOTER last_known_addr { host: "127.0.222.1" port: 35257 } } peers { permanent_uuid: "26d0a424d542436a8830dbe1f8d5142e" member_type: VOTER last_known_addr { host: "127.0.222.2" port: 37581 } } peers { permanent_uuid: "823d98af597d4559bc2e475e2d08a5df" member_type: VOTER last_known_addr { host: "127.0.222.3" port: 36309 } } }
I20260809 12:10:01.682695 3324 ts_tablet_manager.cc:1428] T 18c21b711bbe46e08a1f1609cf2b8a50 P 823d98af597d4559bc2e475e2d08a5df: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20260809 12:10:01.682696 3329 tablet_replica.cc:442] T 18c21b711bbe46e08a1f1609cf2b8a50 P 823d98af597d4559bc2e475e2d08a5df: This TxnStatusTablet replica's current role is: FOLLOWER
I20260809 12:10:01.684654 3323 tablet_bootstrap.cc:492] T 18c21b711bbe46e08a1f1609cf2b8a50 P 9cea8553e4754cd89ec242d7c7a78577: No bootstrap required, opened a new log
I20260809 12:10:01.684655 3322 tablet_bootstrap.cc:492] T 18c21b711bbe46e08a1f1609cf2b8a50 P 26d0a424d542436a8830dbe1f8d5142e: No bootstrap required, opened a new log
I20260809 12:10:01.684827 3322 ts_tablet_manager.cc:1397] T 18c21b711bbe46e08a1f1609cf2b8a50 P 26d0a424d542436a8830dbe1f8d5142e: Time spent bootstrapping tablet: real 0.013s user 0.001s sys 0.010s
I20260809 12:10:01.684906 3323 ts_tablet_manager.cc:1397] T 18c21b711bbe46e08a1f1609cf2b8a50 P 9cea8553e4754cd89ec242d7c7a78577: Time spent bootstrapping tablet: real 0.013s user 0.001s sys 0.009s
I20260809 12:10:01.685685 3322 raft_consensus.cc:348] T 18c21b711bbe46e08a1f1609cf2b8a50 P 26d0a424d542436a8830dbe1f8d5142e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9cea8553e4754cd89ec242d7c7a78577" member_type: VOTER last_known_addr { host: "127.0.222.1" port: 35257 } } peers { permanent_uuid: "26d0a424d542436a8830dbe1f8d5142e" member_type: VOTER last_known_addr { host: "127.0.222.2" port: 37581 } } peers { permanent_uuid: "823d98af597d4559bc2e475e2d08a5df" member_type: VOTER last_known_addr { host: "127.0.222.3" port: 36309 } }
I20260809 12:10:01.685935 3322 raft_consensus.cc:374] T 18c21b711bbe46e08a1f1609cf2b8a50 P 26d0a424d542436a8830dbe1f8d5142e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260809 12:10:01.686040 3322 raft_consensus.cc:729] T 18c21b711bbe46e08a1f1609cf2b8a50 P 26d0a424d542436a8830dbe1f8d5142e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 26d0a424d542436a8830dbe1f8d5142e, State: Initialized, Role: FOLLOWER
I20260809 12:10:01.686280 3323 raft_consensus.cc:348] T 18c21b711bbe46e08a1f1609cf2b8a50 P 9cea8553e4754cd89ec242d7c7a78577 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9cea8553e4754cd89ec242d7c7a78577" member_type: VOTER last_known_addr { host: "127.0.222.1" port: 35257 } } peers { permanent_uuid: "26d0a424d542436a8830dbe1f8d5142e" member_type: VOTER last_known_addr { host: "127.0.222.2" port: 37581 } } peers { permanent_uuid: "823d98af597d4559bc2e475e2d08a5df" member_type: VOTER last_known_addr { host: "127.0.222.3" port: 36309 } }
I20260809 12:10:01.686268 3322 consensus_queue.cc:260] T 18c21b711bbe46e08a1f1609cf2b8a50 P 26d0a424d542436a8830dbe1f8d5142e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9cea8553e4754cd89ec242d7c7a78577" member_type: VOTER last_known_addr { host: "127.0.222.1" port: 35257 } } peers { permanent_uuid: "26d0a424d542436a8830dbe1f8d5142e" member_type: VOTER last_known_addr { host: "127.0.222.2" port: 37581 } } peers { permanent_uuid: "823d98af597d4559bc2e475e2d08a5df" member_type: VOTER last_known_addr { host: "127.0.222.3" port: 36309 } }
I20260809 12:10:01.686523 3323 raft_consensus.cc:374] T 18c21b711bbe46e08a1f1609cf2b8a50 P 9cea8553e4754cd89ec242d7c7a78577 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260809 12:10:01.686615 3323 raft_consensus.cc:729] T 18c21b711bbe46e08a1f1609cf2b8a50 P 9cea8553e4754cd89ec242d7c7a78577 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9cea8553e4754cd89ec242d7c7a78577, State: Initialized, Role: FOLLOWER
I20260809 12:10:01.687122 3322 ts_tablet_manager.cc:1428] T 18c21b711bbe46e08a1f1609cf2b8a50 P 26d0a424d542436a8830dbe1f8d5142e: Time spent starting tablet: real 0.002s user 0.000s sys 0.002s
I20260809 12:10:01.686935 3330 tablet_replica.cc:439] T 18c21b711bbe46e08a1f1609cf2b8a50 P 26d0a424d542436a8830dbe1f8d5142e: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9cea8553e4754cd89ec242d7c7a78577" member_type: VOTER last_known_addr { host: "127.0.222.1" port: 35257 } } peers { permanent_uuid: "26d0a424d542436a8830dbe1f8d5142e" member_type: VOTER last_known_addr { host: "127.0.222.2" port: 37581 } } peers { permanent_uuid: "823d98af597d4559bc2e475e2d08a5df" member_type: VOTER last_known_addr { host: "127.0.222.3" port: 36309 } } }
I20260809 12:10:01.687506 3330 tablet_replica.cc:442] T 18c21b711bbe46e08a1f1609cf2b8a50 P 26d0a424d542436a8830dbe1f8d5142e: This TxnStatusTablet replica's current role is: FOLLOWER
I20260809 12:10:01.687505 3323 consensus_queue.cc:260] T 18c21b711bbe46e08a1f1609cf2b8a50 P 9cea8553e4754cd89ec242d7c7a78577 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9cea8553e4754cd89ec242d7c7a78577" member_type: VOTER last_known_addr { host: "127.0.222.1" port: 35257 } } peers { permanent_uuid: "26d0a424d542436a8830dbe1f8d5142e" member_type: VOTER last_known_addr { host: "127.0.222.2" port: 37581 } } peers { permanent_uuid: "823d98af597d4559bc2e475e2d08a5df" member_type: VOTER last_known_addr { host: "127.0.222.3" port: 36309 } }
I20260809 12:10:01.688015 3328 tablet_replica.cc:439] T 18c21b711bbe46e08a1f1609cf2b8a50 P 9cea8553e4754cd89ec242d7c7a78577: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9cea8553e4754cd89ec242d7c7a78577" member_type: VOTER last_known_addr { host: "127.0.222.1" port: 35257 } } peers { permanent_uuid: "26d0a424d542436a8830dbe1f8d5142e" member_type: VOTER last_known_addr { host: "127.0.222.2" port: 37581 } } peers { permanent_uuid: "823d98af597d4559bc2e475e2d08a5df" member_type: VOTER last_known_addr { host: "127.0.222.3" port: 36309 } } }
I20260809 12:10:01.688390 3328 tablet_replica.cc:442] T 18c21b711bbe46e08a1f1609cf2b8a50 P 9cea8553e4754cd89ec242d7c7a78577: This TxnStatusTablet replica's current role is: FOLLOWER
I20260809 12:10:01.688710 3323 ts_tablet_manager.cc:1428] T 18c21b711bbe46e08a1f1609cf2b8a50 P 9cea8553e4754cd89ec242d7c7a78577: Time spent starting tablet: real 0.003s user 0.000s sys 0.002s
I20260809 12:10:01.764478 3328 raft_consensus.cc:482] T 18c21b711bbe46e08a1f1609cf2b8a50 P 9cea8553e4754cd89ec242d7c7a78577 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260809 12:10:01.764737 3328 raft_consensus.cc:504] T 18c21b711bbe46e08a1f1609cf2b8a50 P 9cea8553e4754cd89ec242d7c7a78577 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9cea8553e4754cd89ec242d7c7a78577" member_type: VOTER last_known_addr { host: "127.0.222.1" port: 35257 } } peers { permanent_uuid: "26d0a424d542436a8830dbe1f8d5142e" member_type: VOTER last_known_addr { host: "127.0.222.2" port: 37581 } } peers { permanent_uuid: "823d98af597d4559bc2e475e2d08a5df" member_type: VOTER last_known_addr { host: "127.0.222.3" port: 36309 } }
I20260809 12:10:01.765560 3328 leader_election.cc:288] T 18c21b711bbe46e08a1f1609cf2b8a50 P 9cea8553e4754cd89ec242d7c7a78577 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 26d0a424d542436a8830dbe1f8d5142e (127.0.222.2:37581), 823d98af597d4559bc2e475e2d08a5df (127.0.222.3:36309)
I20260809 12:10:01.766227 3106 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "18c21b711bbe46e08a1f1609cf2b8a50" candidate_uuid: "9cea8553e4754cd89ec242d7c7a78577" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "26d0a424d542436a8830dbe1f8d5142e" is_pre_election: true
I20260809 12:10:01.766345 3250 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "18c21b711bbe46e08a1f1609cf2b8a50" candidate_uuid: "9cea8553e4754cd89ec242d7c7a78577" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "823d98af597d4559bc2e475e2d08a5df" is_pre_election: true
I20260809 12:10:01.766510 3106 raft_consensus.cc:2454] T 18c21b711bbe46e08a1f1609cf2b8a50 P 26d0a424d542436a8830dbe1f8d5142e [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9cea8553e4754cd89ec242d7c7a78577 in term 0.
I20260809 12:10:01.766660 3250 raft_consensus.cc:2454] T 18c21b711bbe46e08a1f1609cf2b8a50 P 823d98af597d4559bc2e475e2d08a5df [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9cea8553e4754cd89ec242d7c7a78577 in term 0.
I20260809 12:10:01.767059 2873 leader_election.cc:302] T 18c21b711bbe46e08a1f1609cf2b8a50 P 9cea8553e4754cd89ec242d7c7a78577 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 26d0a424d542436a8830dbe1f8d5142e, 9cea8553e4754cd89ec242d7c7a78577; no voters:
I20260809 12:10:01.767398 3328 raft_consensus.cc:2781] T 18c21b711bbe46e08a1f1609cf2b8a50 P 9cea8553e4754cd89ec242d7c7a78577 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260809 12:10:01.767515 3328 raft_consensus.cc:482] T 18c21b711bbe46e08a1f1609cf2b8a50 P 9cea8553e4754cd89ec242d7c7a78577 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260809 12:10:01.767640 3328 raft_consensus.cc:3037] T 18c21b711bbe46e08a1f1609cf2b8a50 P 9cea8553e4754cd89ec242d7c7a78577 [term 0 FOLLOWER]: Advancing to term 1
I20260809 12:10:01.769806 3328 raft_consensus.cc:504] T 18c21b711bbe46e08a1f1609cf2b8a50 P 9cea8553e4754cd89ec242d7c7a78577 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9cea8553e4754cd89ec242d7c7a78577" member_type: VOTER last_known_addr { host: "127.0.222.1" port: 35257 } } peers { permanent_uuid: "26d0a424d542436a8830dbe1f8d5142e" member_type: VOTER last_known_addr { host: "127.0.222.2" port: 37581 } } peers { permanent_uuid: "823d98af597d4559bc2e475e2d08a5df" member_type: VOTER last_known_addr { host: "127.0.222.3" port: 36309 } }
I20260809 12:10:01.770457 3328 leader_election.cc:288] T 18c21b711bbe46e08a1f1609cf2b8a50 P 9cea8553e4754cd89ec242d7c7a78577 [CANDIDATE]: Term 1 election: Requested vote from peers 26d0a424d542436a8830dbe1f8d5142e (127.0.222.2:37581), 823d98af597d4559bc2e475e2d08a5df (127.0.222.3:36309)
I20260809 12:10:01.770798 3106 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "18c21b711bbe46e08a1f1609cf2b8a50" candidate_uuid: "9cea8553e4754cd89ec242d7c7a78577" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "26d0a424d542436a8830dbe1f8d5142e"
I20260809 12:10:01.770850 3250 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "18c21b711bbe46e08a1f1609cf2b8a50" candidate_uuid: "9cea8553e4754cd89ec242d7c7a78577" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "823d98af597d4559bc2e475e2d08a5df"
I20260809 12:10:01.771085 3250 raft_consensus.cc:3037] T 18c21b711bbe46e08a1f1609cf2b8a50 P 823d98af597d4559bc2e475e2d08a5df [term 0 FOLLOWER]: Advancing to term 1
I20260809 12:10:01.771098 3106 raft_consensus.cc:3037] T 18c21b711bbe46e08a1f1609cf2b8a50 P 26d0a424d542436a8830dbe1f8d5142e [term 0 FOLLOWER]: Advancing to term 1
I20260809 12:10:01.773363 3250 raft_consensus.cc:2454] T 18c21b711bbe46e08a1f1609cf2b8a50 P 823d98af597d4559bc2e475e2d08a5df [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9cea8553e4754cd89ec242d7c7a78577 in term 1.
I20260809 12:10:01.773564 3106 raft_consensus.cc:2454] T 18c21b711bbe46e08a1f1609cf2b8a50 P 26d0a424d542436a8830dbe1f8d5142e [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9cea8553e4754cd89ec242d7c7a78577 in term 1.
I20260809 12:10:01.773787 2873 leader_election.cc:302] T 18c21b711bbe46e08a1f1609cf2b8a50 P 9cea8553e4754cd89ec242d7c7a78577 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 823d98af597d4559bc2e475e2d08a5df, 9cea8553e4754cd89ec242d7c7a78577; no voters:
I20260809 12:10:01.774081 3328 raft_consensus.cc:2781] T 18c21b711bbe46e08a1f1609cf2b8a50 P 9cea8553e4754cd89ec242d7c7a78577 [term 1 FOLLOWER]: Leader election won for term 1
I20260809 12:10:01.774264 3328 raft_consensus.cc:686] T 18c21b711bbe46e08a1f1609cf2b8a50 P 9cea8553e4754cd89ec242d7c7a78577 [term 1 LEADER]: Becoming Leader. State: Replica: 9cea8553e4754cd89ec242d7c7a78577, State: Running, Role: LEADER
I20260809 12:10:01.774530 3328 consensus_queue.cc:237] T 18c21b711bbe46e08a1f1609cf2b8a50 P 9cea8553e4754cd89ec242d7c7a78577 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9cea8553e4754cd89ec242d7c7a78577" member_type: VOTER last_known_addr { host: "127.0.222.1" port: 35257 } } peers { permanent_uuid: "26d0a424d542436a8830dbe1f8d5142e" member_type: VOTER last_known_addr { host: "127.0.222.2" port: 37581 } } peers { permanent_uuid: "823d98af597d4559bc2e475e2d08a5df" member_type: VOTER last_known_addr { host: "127.0.222.3" port: 36309 } }
I20260809 12:10:01.776141 3333 tablet_replica.cc:439] T 18c21b711bbe46e08a1f1609cf2b8a50 P 9cea8553e4754cd89ec242d7c7a78577: TxnStatusTablet state changed. Reason: New leader 9cea8553e4754cd89ec242d7c7a78577. Latest consensus state: current_term: 1 leader_uuid: "9cea8553e4754cd89ec242d7c7a78577" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9cea8553e4754cd89ec242d7c7a78577" member_type: VOTER last_known_addr { host: "127.0.222.1" port: 35257 } } peers { permanent_uuid: "26d0a424d542436a8830dbe1f8d5142e" member_type: VOTER last_known_addr { host: "127.0.222.2" port: 37581 } } peers { permanent_uuid: "823d98af597d4559bc2e475e2d08a5df" member_type: VOTER last_known_addr { host: "127.0.222.3" port: 36309 } } }
I20260809 12:10:01.776417 3333 tablet_replica.cc:442] T 18c21b711bbe46e08a1f1609cf2b8a50 P 9cea8553e4754cd89ec242d7c7a78577: This TxnStatusTablet replica's current role is: LEADER
I20260809 12:10:01.777285 3347 txn_status_manager.cc:873] Waiting until node catch up with all replicated operations in previous term...
I20260809 12:10:01.777588 3347 txn_status_manager.cc:929] Loading transaction status metadata into memory...
I20260809 12:10:01.777802 2780 catalog_manager.cc:5495] T 18c21b711bbe46e08a1f1609cf2b8a50 P 9cea8553e4754cd89ec242d7c7a78577 reported cstate change: term changed from 0 to 1, leader changed from <none> to 9cea8553e4754cd89ec242d7c7a78577 (127.0.222.1). New cstate: current_term: 1 leader_uuid: "9cea8553e4754cd89ec242d7c7a78577" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9cea8553e4754cd89ec242d7c7a78577" member_type: VOTER last_known_addr { host: "127.0.222.1" port: 35257 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "26d0a424d542436a8830dbe1f8d5142e" member_type: VOTER last_known_addr { host: "127.0.222.2" port: 37581 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "823d98af597d4559bc2e475e2d08a5df" member_type: VOTER last_known_addr { host: "127.0.222.3" port: 36309 } health_report { overall_health: UNKNOWN } } }
I20260809 12:10:01.813144 3340 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 12:10:01.806312 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.222.1:35257 (local address 127.0.0.1:42442)
0809 12:10:01.806520 (+ 208us) negotiation.cc:107] Waiting for socket to connect
0809 12:10:01.806555 (+ 35us) client_negotiation.cc:174] Beginning negotiation
0809 12:10:01.806718 (+ 163us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 12:10:01.807428 (+ 710us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 12:10:01.807433 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 12:10:01.807445 (+ 12us) client_negotiation.cc:189] Negotiated authn=TOKEN
0809 12:10:01.807664 (+ 219us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 12:10:01.807681 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 12:10:01.808966 (+ 1285us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 12:10:01.808972 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 12:10:01.810165 (+ 1193us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 12:10:01.810177 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 12:10:01.811323 (+ 1146us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 12:10:01.811328 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 12:10:01.811468 (+ 140us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 12:10:01.811523 (+ 55us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0809 12:10:01.812717 (+ 1194us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0809 12:10:01.812735 (+ 18us) client_negotiation.cc:769] Sending connection context
0809 12:10:01.812792 (+ 57us) client_negotiation.cc:240] Negotiation successful
0809 12:10:01.812908 (+ 116us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":49}
I20260809 12:10:01.813364 3350 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 12:10:01.806378 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42442 (local address 127.0.222.1:35257)
0809 12:10:01.807065 (+ 687us) server_negotiation.cc:206] Beginning negotiation
0809 12:10:01.807070 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0809 12:10:01.807089 (+ 19us) server_negotiation.cc:407] Connection header received
0809 12:10:01.807172 (+ 83us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 12:10:01.807175 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 12:10:01.807246 (+ 71us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 12:10:01.807410 (+ 164us) server_negotiation.cc:226] Negotiated authn=TOKEN
0809 12:10:01.807878 (+ 468us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 12:10:01.808836 (+ 958us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 12:10:01.810315 (+ 1479us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 12:10:01.811151 (+ 836us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 12:10:01.811299 (+ 148us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 12:10:01.811758 (+ 459us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0809 12:10:01.812608 (+ 850us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0809 12:10:01.812769 (+ 161us) server_negotiation.cc:1035] Waiting for connection context
0809 12:10:01.812840 (+ 71us) server_negotiation.cc:299] Negotiation successful
0809 12:10:01.812935 (+ 95us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":490,"thread_start_us":306,"threads_started":1}
I20260809 12:10:01.821612 3106 raft_consensus.cc:1261] T 18c21b711bbe46e08a1f1609cf2b8a50 P 26d0a424d542436a8830dbe1f8d5142e [term 1 FOLLOWER]: Refusing update from remote peer 9cea8553e4754cd89ec242d7c7a78577: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260809 12:10:01.821678 3250 raft_consensus.cc:1261] T 18c21b711bbe46e08a1f1609cf2b8a50 P 823d98af597d4559bc2e475e2d08a5df [term 1 FOLLOWER]: Refusing update from remote peer 9cea8553e4754cd89ec242d7c7a78577: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260809 12:10:01.822819 3328 consensus_queue.cc:1035] T 18c21b711bbe46e08a1f1609cf2b8a50 P 9cea8553e4754cd89ec242d7c7a78577 [LEADER]: Connected to new peer: Peer: permanent_uuid: "26d0a424d542436a8830dbe1f8d5142e" member_type: VOTER last_known_addr { host: "127.0.222.2" port: 37581 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260809 12:10:01.823236 3333 consensus_queue.cc:1035] T 18c21b711bbe46e08a1f1609cf2b8a50 P 9cea8553e4754cd89ec242d7c7a78577 [LEADER]: Connected to new peer: Peer: permanent_uuid: "823d98af597d4559bc2e475e2d08a5df" member_type: VOTER last_known_addr { host: "127.0.222.3" port: 36309 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260809 12:10:01.834411 3330 tablet_replica.cc:439] T 18c21b711bbe46e08a1f1609cf2b8a50 P 26d0a424d542436a8830dbe1f8d5142e: TxnStatusTablet state changed. Reason: New leader 9cea8553e4754cd89ec242d7c7a78577. Latest consensus state: current_term: 1 leader_uuid: "9cea8553e4754cd89ec242d7c7a78577" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9cea8553e4754cd89ec242d7c7a78577" member_type: VOTER last_known_addr { host: "127.0.222.1" port: 35257 } } peers { permanent_uuid: "26d0a424d542436a8830dbe1f8d5142e" member_type: VOTER last_known_addr { host: "127.0.222.2" port: 37581 } } peers { permanent_uuid: "823d98af597d4559bc2e475e2d08a5df" member_type: VOTER last_known_addr { host: "127.0.222.3" port: 36309 } } }
I20260809 12:10:01.834723 3330 tablet_replica.cc:442] T 18c21b711bbe46e08a1f1609cf2b8a50 P 26d0a424d542436a8830dbe1f8d5142e: This TxnStatusTablet replica's current role is: FOLLOWER
I20260809 12:10:01.837419 3351 mvcc.cc:205] Tried to move back new op lower bound from 7316592237849862144 to 7316592237671055360. Current Snapshot: MvccSnapshot[applied={T|T < 7316592237849862144}]
I20260809 12:10:01.842800 3330 tablet_replica.cc:439] T 18c21b711bbe46e08a1f1609cf2b8a50 P 26d0a424d542436a8830dbe1f8d5142e: TxnStatusTablet state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "9cea8553e4754cd89ec242d7c7a78577" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9cea8553e4754cd89ec242d7c7a78577" member_type: VOTER last_known_addr { host: "127.0.222.1" port: 35257 } } peers { permanent_uuid: "26d0a424d542436a8830dbe1f8d5142e" member_type: VOTER last_known_addr { host: "127.0.222.2" port: 37581 } } peers { permanent_uuid: "823d98af597d4559bc2e475e2d08a5df" member_type: VOTER last_known_addr { host: "127.0.222.3" port: 36309 } } }
I20260809 12:10:01.843056 3330 tablet_replica.cc:442] T 18c21b711bbe46e08a1f1609cf2b8a50 P 26d0a424d542436a8830dbe1f8d5142e: This TxnStatusTablet replica's current role is: FOLLOWER
I20260809 12:10:01.844892 3328 tablet_replica.cc:439] T 18c21b711bbe46e08a1f1609cf2b8a50 P 9cea8553e4754cd89ec242d7c7a78577: TxnStatusTablet state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "9cea8553e4754cd89ec242d7c7a78577" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9cea8553e4754cd89ec242d7c7a78577" member_type: VOTER last_known_addr { host: "127.0.222.1" port: 35257 } } peers { permanent_uuid: "26d0a424d542436a8830dbe1f8d5142e" member_type: VOTER last_known_addr { host: "127.0.222.2" port: 37581 } } peers { permanent_uuid: "823d98af597d4559bc2e475e2d08a5df" member_type: VOTER last_known_addr { host: "127.0.222.3" port: 36309 } } }
I20260809 12:10:01.845217 3328 tablet_replica.cc:442] T 18c21b711bbe46e08a1f1609cf2b8a50 P 9cea8553e4754cd89ec242d7c7a78577: This TxnStatusTablet replica's current role is: LEADER
I20260809 12:10:01.845176 3329 tablet_replica.cc:439] T 18c21b711bbe46e08a1f1609cf2b8a50 P 823d98af597d4559bc2e475e2d08a5df: TxnStatusTablet state changed. Reason: New leader 9cea8553e4754cd89ec242d7c7a78577. Latest consensus state: current_term: 1 leader_uuid: "9cea8553e4754cd89ec242d7c7a78577" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9cea8553e4754cd89ec242d7c7a78577" member_type: VOTER last_known_addr { host: "127.0.222.1" port: 35257 } } peers { permanent_uuid: "26d0a424d542436a8830dbe1f8d5142e" member_type: VOTER last_known_addr { host: "127.0.222.2" port: 37581 } } peers { permanent_uuid: "823d98af597d4559bc2e475e2d08a5df" member_type: VOTER last_known_addr { host: "127.0.222.3" port: 36309 } } }
I20260809 12:10:01.845429 3329 tablet_replica.cc:442] T 18c21b711bbe46e08a1f1609cf2b8a50 P 823d98af597d4559bc2e475e2d08a5df: This TxnStatusTablet replica's current role is: FOLLOWER
I20260809 12:10:01.851094 3353 mvcc.cc:205] Tried to move back new op lower bound from 7316592237849862144 to 7316592237671055360. Current Snapshot: MvccSnapshot[applied={T|T < 7316592237849862144}]
I20260809 12:10:01.851783 3329 tablet_replica.cc:439] T 18c21b711bbe46e08a1f1609cf2b8a50 P 823d98af597d4559bc2e475e2d08a5df: TxnStatusTablet state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "9cea8553e4754cd89ec242d7c7a78577" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9cea8553e4754cd89ec242d7c7a78577" member_type: VOTER last_known_addr { host: "127.0.222.1" port: 35257 } } peers { permanent_uuid: "26d0a424d542436a8830dbe1f8d5142e" member_type: VOTER last_known_addr { host: "127.0.222.2" port: 37581 } } peers { permanent_uuid: "823d98af597d4559bc2e475e2d08a5df" member_type: VOTER last_known_addr { host: "127.0.222.3" port: 36309 } } }
I20260809 12:10:01.852020 3329 tablet_replica.cc:442] T 18c21b711bbe46e08a1f1609cf2b8a50 P 823d98af597d4559bc2e475e2d08a5df: This TxnStatusTablet replica's current role is: FOLLOWER
I20260809 12:10:01.849850 3358 tablet_replica.cc:439] T 18c21b711bbe46e08a1f1609cf2b8a50 P 9cea8553e4754cd89ec242d7c7a78577: TxnStatusTablet state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "9cea8553e4754cd89ec242d7c7a78577" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9cea8553e4754cd89ec242d7c7a78577" member_type: VOTER last_known_addr { host: "127.0.222.1" port: 35257 } } peers { permanent_uuid: "26d0a424d542436a8830dbe1f8d5142e" member_type: VOTER last_known_addr { host: "127.0.222.2" port: 37581 } } peers { permanent_uuid: "823d98af597d4559bc2e475e2d08a5df" member_type: VOTER last_known_addr { host: "127.0.222.3" port: 36309 } } }
I20260809 12:10:01.852394 3358 tablet_replica.cc:442] T 18c21b711bbe46e08a1f1609cf2b8a50 P 9cea8553e4754cd89ec242d7c7a78577: This TxnStatusTablet replica's current role is: LEADER
I20260809 12:10:01.862388 3350 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 12:10:01.856388 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42458 (local address 127.0.222.1:35257)
0809 12:10:01.856576 (+ 188us) server_negotiation.cc:206] Beginning negotiation
0809 12:10:01.856583 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0809 12:10:01.857139 (+ 556us) server_negotiation.cc:407] Connection header received
0809 12:10:01.857293 (+ 154us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 12:10:01.857299 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 12:10:01.857407 (+ 108us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 12:10:01.857513 (+ 106us) server_negotiation.cc:226] Negotiated authn=TOKEN
0809 12:10:01.857849 (+ 336us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 12:10:01.859085 (+ 1236us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 12:10:01.860509 (+ 1424us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 12:10:01.861178 (+ 669us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 12:10:01.861323 (+ 145us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 12:10:01.861653 (+ 330us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0809 12:10:01.861785 (+ 132us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0809 12:10:01.861870 (+ 85us) server_negotiation.cc:1035] Waiting for connection context
0809 12:10:01.861988 (+ 118us) server_negotiation.cc:299] Negotiation successful
0809 12:10:01.862073 (+ 85us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":54}
I20260809 12:10:01.879393 3350 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 12:10:01.873562 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.222.1:51205 (local address 127.0.222.1:35257)
0809 12:10:01.873756 (+ 194us) server_negotiation.cc:206] Beginning negotiation
0809 12:10:01.873763 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0809 12:10:01.874448 (+ 685us) server_negotiation.cc:407] Connection header received
0809 12:10:01.874635 (+ 187us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 12:10:01.874640 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 12:10:01.874710 (+ 70us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 12:10:01.874814 (+ 104us) server_negotiation.cc:226] Negotiated authn=TOKEN
0809 12:10:01.875352 (+ 538us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 12:10:01.876084 (+ 732us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 12:10:01.877518 (+ 1434us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 12:10:01.878072 (+ 554us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 12:10:01.878193 (+ 121us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 12:10:01.878655 (+ 462us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0809 12:10:01.878756 (+ 101us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0809 12:10:01.878848 (+ 92us) server_negotiation.cc:1035] Waiting for connection context
0809 12:10:01.879059 (+ 211us) server_negotiation.cc:299] Negotiation successful
0809 12:10:01.879138 (+ 79us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":58}
I20260809 12:10:01.880023 3364 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 12:10:01.873577 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.222.1:35257 (local address 127.0.222.1:51205)
0809 12:10:01.874348 (+ 771us) negotiation.cc:107] Waiting for socket to connect
0809 12:10:01.874370 (+ 22us) client_negotiation.cc:174] Beginning negotiation
0809 12:10:01.874489 (+ 119us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 12:10:01.874837 (+ 348us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 12:10:01.874841 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 12:10:01.874852 (+ 11us) client_negotiation.cc:189] Negotiated authn=TOKEN
0809 12:10:01.875212 (+ 360us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 12:10:01.875220 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 12:10:01.876214 (+ 994us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 12:10:01.876219 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 12:10:01.877367 (+ 1148us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 12:10:01.877379 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 12:10:01.878261 (+ 882us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 12:10:01.878267 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 12:10:01.878391 (+ 124us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 12:10:01.878441 (+ 50us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0809 12:10:01.878846 (+ 405us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0809 12:10:01.878900 (+ 54us) client_negotiation.cc:769] Sending connection context
0809 12:10:01.879595 (+ 695us) client_negotiation.cc:240] Negotiation successful
0809 12:10:01.879726 (+ 131us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":553,"thread_start_us":388,"threads_started":1}
I20260809 12:10:01.890113 3106 raft_consensus.cc:1261] T 79f986af492747e8911414a8dd4f5d18 P 26d0a424d542436a8830dbe1f8d5142e [term 1 FOLLOWER]: Refusing update from remote peer 9cea8553e4754cd89ec242d7c7a78577: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260809 12:10:01.890220 3250 raft_consensus.cc:1261] T 79f986af492747e8911414a8dd4f5d18 P 823d98af597d4559bc2e475e2d08a5df [term 1 FOLLOWER]: Refusing update from remote peer 9cea8553e4754cd89ec242d7c7a78577: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260809 12:10:01.890802 3328 consensus_queue.cc:1035] T 79f986af492747e8911414a8dd4f5d18 P 9cea8553e4754cd89ec242d7c7a78577 [LEADER]: Connected to new peer: Peer: permanent_uuid: "26d0a424d542436a8830dbe1f8d5142e" member_type: VOTER last_known_addr { host: "127.0.222.2" port: 37581 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260809 12:10:01.891347 3358 consensus_queue.cc:1035] T 79f986af492747e8911414a8dd4f5d18 P 9cea8553e4754cd89ec242d7c7a78577 [LEADER]: Connected to new peer: Peer: permanent_uuid: "823d98af597d4559bc2e475e2d08a5df" member_type: VOTER last_known_addr { host: "127.0.222.3" port: 36309 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260809 12:10:01.900966 3352 mvcc.cc:205] Tried to move back new op lower bound from 7316592238137090048 to 7316592236849459200. Current Snapshot: MvccSnapshot[applied={T|T < 7316592238137090048}]
I20260809 12:10:01.959967 2778 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:51984:
table { table_name: "test-table" } modify_external_catalogs: true
I20260809 12:10:01.960280 2778 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:51984:
table { table_name: "test-table" } modify_external_catalogs: true
I20260809 12:10:01.964322 2778 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 846fd7fb008b4e5ea8bd71f4646a10c4: Sending DeleteTablet for 3 replicas of tablet 79f986af492747e8911414a8dd4f5d18
I20260809 12:10:01.965749 3227 tablet_service.cc:1514] Processing DeleteTablet for tablet 79f986af492747e8911414a8dd4f5d18 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-09 12:10:01 UTC) from {username='kudu', principal='kudu/127.0.222.62@KRBTEST.COM'} at 127.0.0.1:42174
I20260809 12:10:01.965721 2927 tablet_service.cc:1514] Processing DeleteTablet for tablet 79f986af492747e8911414a8dd4f5d18 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-09 12:10:01 UTC) from {username='kudu', principal='kudu/127.0.222.62@KRBTEST.COM'} at 127.0.0.1:42438
I20260809 12:10:01.965803 3082 tablet_service.cc:1514] Processing DeleteTablet for tablet 79f986af492747e8911414a8dd4f5d18 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-09 12:10:01 UTC) from {username='kudu', principal='kudu/127.0.222.62@KRBTEST.COM'} at 127.0.0.1:48188
I20260809 12:10:01.970943 2778 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:51984:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260809 12:10:01.972697 2778 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260809 12:10:01.975970 3372 tablet_replica.cc:330] T 79f986af492747e8911414a8dd4f5d18 P 823d98af597d4559bc2e475e2d08a5df: stopping tablet replica
I20260809 12:10:01.976644 3370 tablet_replica.cc:330] T 79f986af492747e8911414a8dd4f5d18 P 9cea8553e4754cd89ec242d7c7a78577: stopping tablet replica
I20260809 12:10:01.976722 3372 raft_consensus.cc:2229] T 79f986af492747e8911414a8dd4f5d18 P 823d98af597d4559bc2e475e2d08a5df [term 1 FOLLOWER]: Raft consensus shutting down.
I20260809 12:10:01.977169 3372 raft_consensus.cc:2258] T 79f986af492747e8911414a8dd4f5d18 P 823d98af597d4559bc2e475e2d08a5df [term 1 FOLLOWER]: Raft consensus is shut down!
I20260809 12:10:01.977409 3370 raft_consensus.cc:2229] T 79f986af492747e8911414a8dd4f5d18 P 9cea8553e4754cd89ec242d7c7a78577 [term 1 LEADER]: Raft consensus shutting down.
I20260809 12:10:01.979862 3372 ts_tablet_manager.cc:1905] T 79f986af492747e8911414a8dd4f5d18 P 823d98af597d4559bc2e475e2d08a5df: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260809 12:10:01.979888 3370 raft_consensus.cc:2258] T 79f986af492747e8911414a8dd4f5d18 P 9cea8553e4754cd89ec242d7c7a78577 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260809 12:10:01.982081 3370 ts_tablet_manager.cc:1905] T 79f986af492747e8911414a8dd4f5d18 P 9cea8553e4754cd89ec242d7c7a78577: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260809 12:10:01.983955 3371 tablet_replica.cc:330] T 79f986af492747e8911414a8dd4f5d18 P 26d0a424d542436a8830dbe1f8d5142e: stopping tablet replica
I20260809 12:10:01.984753 3227 tablet_service.cc:1467] Processing CreateTablet for tablet 4f441cf08efc474da096b2ebb6008ad0 (DEFAULT_TABLE table=test-table [id=654250b2eb3e43408280fab55a9c48e0]), partition=RANGE (key) PARTITION UNBOUNDED
I20260809 12:10:01.984827 2927 tablet_service.cc:1467] Processing CreateTablet for tablet 4f441cf08efc474da096b2ebb6008ad0 (DEFAULT_TABLE table=test-table [id=654250b2eb3e43408280fab55a9c48e0]), partition=RANGE (key) PARTITION UNBOUNDED
I20260809 12:10:01.985217 3227 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 4f441cf08efc474da096b2ebb6008ad0. 1 dirs total, 0 dirs full, 0 dirs failed
I20260809 12:10:01.985389 2927 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 4f441cf08efc474da096b2ebb6008ad0. 1 dirs total, 0 dirs full, 0 dirs failed
I20260809 12:10:01.989528 3082 tablet_service.cc:1467] Processing CreateTablet for tablet 4f441cf08efc474da096b2ebb6008ad0 (DEFAULT_TABLE table=test-table [id=654250b2eb3e43408280fab55a9c48e0]), partition=RANGE (key) PARTITION UNBOUNDED
I20260809 12:10:01.989979 3082 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 4f441cf08efc474da096b2ebb6008ad0. 1 dirs total, 0 dirs full, 0 dirs failed
I20260809 12:10:01.992100 3324 tablet_bootstrap.cc:492] T 4f441cf08efc474da096b2ebb6008ad0 P 823d98af597d4559bc2e475e2d08a5df: Bootstrap starting.
I20260809 12:10:01.992200 3371 raft_consensus.cc:2229] T 79f986af492747e8911414a8dd4f5d18 P 26d0a424d542436a8830dbe1f8d5142e [term 1 FOLLOWER]: Raft consensus shutting down.
I20260809 12:10:01.992617 3371 raft_consensus.cc:2258] T 79f986af492747e8911414a8dd4f5d18 P 26d0a424d542436a8830dbe1f8d5142e [term 1 FOLLOWER]: Raft consensus is shut down!
I20260809 12:10:01.993922 3324 tablet_bootstrap.cc:654] T 4f441cf08efc474da096b2ebb6008ad0 P 823d98af597d4559bc2e475e2d08a5df: Neither blocks nor log segments found. Creating new log.
I20260809 12:10:01.995364 3370 ts_tablet_manager.cc:1918] T 79f986af492747e8911414a8dd4f5d18 P 9cea8553e4754cd89ec242d7c7a78577: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260809 12:10:01.995726 3323 tablet_bootstrap.cc:492] T 4f441cf08efc474da096b2ebb6008ad0 P 9cea8553e4754cd89ec242d7c7a78577: Bootstrap starting.
I20260809 12:10:01.997804 3323 tablet_bootstrap.cc:654] T 4f441cf08efc474da096b2ebb6008ad0 P 9cea8553e4754cd89ec242d7c7a78577: Neither blocks nor log segments found. Creating new log.
I20260809 12:10:01.997797 3372 ts_tablet_manager.cc:1918] T 79f986af492747e8911414a8dd4f5d18 P 823d98af597d4559bc2e475e2d08a5df: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260809 12:10:01.997972 3372 log.cc:1196] T 79f986af492747e8911414a8dd4f5d18 P 823d98af597d4559bc2e475e2d08a5df: Deleting WAL directory at /tmp/dist-test-taskgH5Pjm/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786277382073938-888-0/minicluster-data/ts-2/wal/wals/79f986af492747e8911414a8dd4f5d18
I20260809 12:10:01.998558 3372 ts_tablet_manager.cc:1939] T 79f986af492747e8911414a8dd4f5d18 P 823d98af597d4559bc2e475e2d08a5df: Deleting consensus metadata
I20260809 12:10:02.000835 2762 catalog_manager.cc:4841] TS 823d98af597d4559bc2e475e2d08a5df (127.0.222.3:36309): tablet 79f986af492747e8911414a8dd4f5d18 (table test-table [id=a8bc98782b0d4ebfaa09425c27627bd3]) successfully deleted
I20260809 12:10:02.002286 3370 log.cc:1196] T 79f986af492747e8911414a8dd4f5d18 P 9cea8553e4754cd89ec242d7c7a78577: Deleting WAL directory at /tmp/dist-test-taskgH5Pjm/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786277382073938-888-0/minicluster-data/ts-0/wal/wals/79f986af492747e8911414a8dd4f5d18
I20260809 12:10:02.003016 3371 ts_tablet_manager.cc:1905] T 79f986af492747e8911414a8dd4f5d18 P 26d0a424d542436a8830dbe1f8d5142e: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260809 12:10:02.003754 3370 ts_tablet_manager.cc:1939] T 79f986af492747e8911414a8dd4f5d18 P 9cea8553e4754cd89ec242d7c7a78577: Deleting consensus metadata
I20260809 12:10:02.005765 2763 catalog_manager.cc:4841] TS 9cea8553e4754cd89ec242d7c7a78577 (127.0.222.1:35257): tablet 79f986af492747e8911414a8dd4f5d18 (table test-table [id=a8bc98782b0d4ebfaa09425c27627bd3]) successfully deleted
I20260809 12:10:02.013312 3371 ts_tablet_manager.cc:1918] T 79f986af492747e8911414a8dd4f5d18 P 26d0a424d542436a8830dbe1f8d5142e: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260809 12:10:02.013653 3371 log.cc:1196] T 79f986af492747e8911414a8dd4f5d18 P 26d0a424d542436a8830dbe1f8d5142e: Deleting WAL directory at /tmp/dist-test-taskgH5Pjm/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786277382073938-888-0/minicluster-data/ts-1/wal/wals/79f986af492747e8911414a8dd4f5d18
I20260809 12:10:02.014453 3371 ts_tablet_manager.cc:1939] T 79f986af492747e8911414a8dd4f5d18 P 26d0a424d542436a8830dbe1f8d5142e: Deleting consensus metadata
I20260809 12:10:02.015477 3324 tablet_bootstrap.cc:492] T 4f441cf08efc474da096b2ebb6008ad0 P 823d98af597d4559bc2e475e2d08a5df: No bootstrap required, opened a new log
I20260809 12:10:02.015717 3324 ts_tablet_manager.cc:1397] T 4f441cf08efc474da096b2ebb6008ad0 P 823d98af597d4559bc2e475e2d08a5df: Time spent bootstrapping tablet: real 0.024s user 0.009s sys 0.001s
I20260809 12:10:02.016693 3324 raft_consensus.cc:348] T 4f441cf08efc474da096b2ebb6008ad0 P 823d98af597d4559bc2e475e2d08a5df [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9cea8553e4754cd89ec242d7c7a78577" member_type: VOTER last_known_addr { host: "127.0.222.1" port: 35257 } } peers { permanent_uuid: "26d0a424d542436a8830dbe1f8d5142e" member_type: VOTER last_known_addr { host: "127.0.222.2" port: 37581 } } peers { permanent_uuid: "823d98af597d4559bc2e475e2d08a5df" member_type: VOTER last_known_addr { host: "127.0.222.3" port: 36309 } }
I20260809 12:10:02.016938 2762 catalog_manager.cc:4841] TS 26d0a424d542436a8830dbe1f8d5142e (127.0.222.2:37581): tablet 79f986af492747e8911414a8dd4f5d18 (table test-table [id=a8bc98782b0d4ebfaa09425c27627bd3]) successfully deleted
I20260809 12:10:02.017068 3324 raft_consensus.cc:374] T 4f441cf08efc474da096b2ebb6008ad0 P 823d98af597d4559bc2e475e2d08a5df [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260809 12:10:02.017251 3324 raft_consensus.cc:729] T 4f441cf08efc474da096b2ebb6008ad0 P 823d98af597d4559bc2e475e2d08a5df [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 823d98af597d4559bc2e475e2d08a5df, State: Initialized, Role: FOLLOWER
I20260809 12:10:02.017582 3324 consensus_queue.cc:260] T 4f441cf08efc474da096b2ebb6008ad0 P 823d98af597d4559bc2e475e2d08a5df [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9cea8553e4754cd89ec242d7c7a78577" member_type: VOTER last_known_addr { host: "127.0.222.1" port: 35257 } } peers { permanent_uuid: "26d0a424d542436a8830dbe1f8d5142e" member_type: VOTER last_known_addr { host: "127.0.222.2" port: 37581 } } peers { permanent_uuid: "823d98af597d4559bc2e475e2d08a5df" member_type: VOTER last_known_addr { host: "127.0.222.3" port: 36309 } }
I20260809 12:10:02.020438 3323 tablet_bootstrap.cc:492] T 4f441cf08efc474da096b2ebb6008ad0 P 9cea8553e4754cd89ec242d7c7a78577: No bootstrap required, opened a new log
I20260809 12:10:02.020728 3323 ts_tablet_manager.cc:1397] T 4f441cf08efc474da096b2ebb6008ad0 P 9cea8553e4754cd89ec242d7c7a78577: Time spent bootstrapping tablet: real 0.025s user 0.011s sys 0.011s
I20260809 12:10:02.020932 3324 ts_tablet_manager.cc:1428] T 4f441cf08efc474da096b2ebb6008ad0 P 823d98af597d4559bc2e475e2d08a5df: Time spent starting tablet: real 0.005s user 0.002s sys 0.000s
I20260809 12:10:02.021605 3323 raft_consensus.cc:348] T 4f441cf08efc474da096b2ebb6008ad0 P 9cea8553e4754cd89ec242d7c7a78577 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9cea8553e4754cd89ec242d7c7a78577" member_type: VOTER last_known_addr { host: "127.0.222.1" port: 35257 } } peers { permanent_uuid: "26d0a424d542436a8830dbe1f8d5142e" member_type: VOTER last_known_addr { host: "127.0.222.2" port: 37581 } } peers { permanent_uuid: "823d98af597d4559bc2e475e2d08a5df" member_type: VOTER last_known_addr { host: "127.0.222.3" port: 36309 } }
I20260809 12:10:02.021907 3323 raft_consensus.cc:374] T 4f441cf08efc474da096b2ebb6008ad0 P 9cea8553e4754cd89ec242d7c7a78577 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260809 12:10:02.022028 3323 raft_consensus.cc:729] T 4f441cf08efc474da096b2ebb6008ad0 P 9cea8553e4754cd89ec242d7c7a78577 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9cea8553e4754cd89ec242d7c7a78577, State: Initialized, Role: FOLLOWER
I20260809 12:10:02.022285 3323 consensus_queue.cc:260] T 4f441cf08efc474da096b2ebb6008ad0 P 9cea8553e4754cd89ec242d7c7a78577 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9cea8553e4754cd89ec242d7c7a78577" member_type: VOTER last_known_addr { host: "127.0.222.1" port: 35257 } } peers { permanent_uuid: "26d0a424d542436a8830dbe1f8d5142e" member_type: VOTER last_known_addr { host: "127.0.222.2" port: 37581 } } peers { permanent_uuid: "823d98af597d4559bc2e475e2d08a5df" member_type: VOTER last_known_addr { host: "127.0.222.3" port: 36309 } }
I20260809 12:10:02.022917 3323 ts_tablet_manager.cc:1428] T 4f441cf08efc474da096b2ebb6008ad0 P 9cea8553e4754cd89ec242d7c7a78577: Time spent starting tablet: real 0.002s user 0.000s sys 0.000s
I20260809 12:10:02.024626 3322 tablet_bootstrap.cc:492] T 4f441cf08efc474da096b2ebb6008ad0 P 26d0a424d542436a8830dbe1f8d5142e: Bootstrap starting.
I20260809 12:10:02.026424 3322 tablet_bootstrap.cc:654] T 4f441cf08efc474da096b2ebb6008ad0 P 26d0a424d542436a8830dbe1f8d5142e: Neither blocks nor log segments found. Creating new log.
I20260809 12:10:02.028795 3322 tablet_bootstrap.cc:492] T 4f441cf08efc474da096b2ebb6008ad0 P 26d0a424d542436a8830dbe1f8d5142e: No bootstrap required, opened a new log
I20260809 12:10:02.028992 3322 ts_tablet_manager.cc:1397] T 4f441cf08efc474da096b2ebb6008ad0 P 26d0a424d542436a8830dbe1f8d5142e: Time spent bootstrapping tablet: real 0.004s user 0.000s sys 0.003s
I20260809 12:10:02.029678 3322 raft_consensus.cc:348] T 4f441cf08efc474da096b2ebb6008ad0 P 26d0a424d542436a8830dbe1f8d5142e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9cea8553e4754cd89ec242d7c7a78577" member_type: VOTER last_known_addr { host: "127.0.222.1" port: 35257 } } peers { permanent_uuid: "26d0a424d542436a8830dbe1f8d5142e" member_type: VOTER last_known_addr { host: "127.0.222.2" port: 37581 } } peers { permanent_uuid: "823d98af597d4559bc2e475e2d08a5df" member_type: VOTER last_known_addr { host: "127.0.222.3" port: 36309 } }
I20260809 12:10:02.029927 3322 raft_consensus.cc:374] T 4f441cf08efc474da096b2ebb6008ad0 P 26d0a424d542436a8830dbe1f8d5142e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260809 12:10:02.030014 3322 raft_consensus.cc:729] T 4f441cf08efc474da096b2ebb6008ad0 P 26d0a424d542436a8830dbe1f8d5142e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 26d0a424d542436a8830dbe1f8d5142e, State: Initialized, Role: FOLLOWER
I20260809 12:10:02.030278 3322 consensus_queue.cc:260] T 4f441cf08efc474da096b2ebb6008ad0 P 26d0a424d542436a8830dbe1f8d5142e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9cea8553e4754cd89ec242d7c7a78577" member_type: VOTER last_known_addr { host: "127.0.222.1" port: 35257 } } peers { permanent_uuid: "26d0a424d542436a8830dbe1f8d5142e" member_type: VOTER last_known_addr { host: "127.0.222.2" port: 37581 } } peers { permanent_uuid: "823d98af597d4559bc2e475e2d08a5df" member_type: VOTER last_known_addr { host: "127.0.222.3" port: 36309 } }
I20260809 12:10:02.030845 3322 ts_tablet_manager.cc:1428] T 4f441cf08efc474da096b2ebb6008ad0 P 26d0a424d542436a8830dbe1f8d5142e: Time spent starting tablet: real 0.002s user 0.000s sys 0.000s
I20260809 12:10:02.179989 3330 raft_consensus.cc:482] T 4f441cf08efc474da096b2ebb6008ad0 P 26d0a424d542436a8830dbe1f8d5142e [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260809 12:10:02.180294 3330 raft_consensus.cc:504] T 4f441cf08efc474da096b2ebb6008ad0 P 26d0a424d542436a8830dbe1f8d5142e [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9cea8553e4754cd89ec242d7c7a78577" member_type: VOTER last_known_addr { host: "127.0.222.1" port: 35257 } } peers { permanent_uuid: "26d0a424d542436a8830dbe1f8d5142e" member_type: VOTER last_known_addr { host: "127.0.222.2" port: 37581 } } peers { permanent_uuid: "823d98af597d4559bc2e475e2d08a5df" member_type: VOTER last_known_addr { host: "127.0.222.3" port: 36309 } }
I20260809 12:10:02.182314 3330 leader_election.cc:288] T 4f441cf08efc474da096b2ebb6008ad0 P 26d0a424d542436a8830dbe1f8d5142e [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 9cea8553e4754cd89ec242d7c7a78577 (127.0.222.1:35257), 823d98af597d4559bc2e475e2d08a5df (127.0.222.3:36309)
I20260809 12:10:02.189916 3376 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 12:10:02.182549 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.222.3:36309 (local address 127.0.222.2:41425)
0809 12:10:02.183102 (+ 553us) negotiation.cc:107] Waiting for socket to connect
0809 12:10:02.183124 (+ 22us) client_negotiation.cc:174] Beginning negotiation
0809 12:10:02.183302 (+ 178us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 12:10:02.184513 (+ 1211us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 12:10:02.184518 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 12:10:02.184547 (+ 29us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0809 12:10:02.184856 (+ 309us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 12:10:02.184870 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 12:10:02.186046 (+ 1176us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 12:10:02.186052 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 12:10:02.187915 (+ 1863us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 12:10:02.187928 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 12:10:02.189027 (+ 1099us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 12:10:02.189033 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 12:10:02.189237 (+ 204us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 12:10:02.189264 (+ 27us) client_negotiation.cc:769] Sending connection context
0809 12:10:02.189375 (+ 111us) client_negotiation.cc:240] Negotiation successful
0809 12:10:02.189528 (+ 153us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":391,"thread_start_us":331,"threads_started":1}
I20260809 12:10:02.189909 3378 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 12:10:02.183309 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.222.2:41425 (local address 127.0.222.3:36309)
0809 12:10:02.184117 (+ 808us) server_negotiation.cc:206] Beginning negotiation
0809 12:10:02.184125 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0809 12:10:02.184147 (+ 22us) server_negotiation.cc:407] Connection header received
0809 12:10:02.184230 (+ 83us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 12:10:02.184235 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 12:10:02.184336 (+ 101us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 12:10:02.184454 (+ 118us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0809 12:10:02.185045 (+ 591us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 12:10:02.185892 (+ 847us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 12:10:02.188081 (+ 2189us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 12:10:02.188860 (+ 779us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 12:10:02.189017 (+ 157us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 12:10:02.189156 (+ 139us) server_negotiation.cc:1035] Waiting for connection context
0809 12:10:02.189405 (+ 249us) server_negotiation.cc:299] Negotiation successful
0809 12:10:02.189554 (+ 149us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":672,"thread_start_us":527,"threads_started":1}
I20260809 12:10:02.190811 3249 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4f441cf08efc474da096b2ebb6008ad0" candidate_uuid: "26d0a424d542436a8830dbe1f8d5142e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "823d98af597d4559bc2e475e2d08a5df" is_pre_election: true
I20260809 12:10:02.191170 3249 raft_consensus.cc:2454] T 4f441cf08efc474da096b2ebb6008ad0 P 823d98af597d4559bc2e475e2d08a5df [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 26d0a424d542436a8830dbe1f8d5142e in term 0.
I20260809 12:10:02.191792 3350 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 12:10:02.182815 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.222.2:43649 (local address 127.0.222.1:35257)
0809 12:10:02.183087 (+ 272us) server_negotiation.cc:206] Beginning negotiation
0809 12:10:02.183096 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0809 12:10:02.183971 (+ 875us) server_negotiation.cc:407] Connection header received
0809 12:10:02.184351 (+ 380us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 12:10:02.184357 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 12:10:02.184446 (+ 89us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 12:10:02.184541 (+ 95us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0809 12:10:02.185355 (+ 814us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 12:10:02.186490 (+ 1135us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 12:10:02.189090 (+ 2600us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 12:10:02.190545 (+ 1455us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 12:10:02.190654 (+ 109us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 12:10:02.190772 (+ 118us) server_negotiation.cc:1035] Waiting for connection context
0809 12:10:02.191352 (+ 580us) server_negotiation.cc:299] Negotiation successful
0809 12:10:02.191502 (+ 150us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":75}
I20260809 12:10:02.191869 3377 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 12:10:02.182549 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.222.1:35257 (local address 127.0.222.2:43649)
0809 12:10:02.183865 (+ 1316us) negotiation.cc:107] Waiting for socket to connect
0809 12:10:02.183884 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0809 12:10:02.184114 (+ 230us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 12:10:02.184670 (+ 556us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 12:10:02.184675 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 12:10:02.184700 (+ 25us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0809 12:10:02.185096 (+ 396us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 12:10:02.185112 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 12:10:02.186682 (+ 1570us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 12:10:02.186688 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 12:10:02.188875 (+ 2187us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 12:10:02.188892 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 12:10:02.190900 (+ 2008us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 12:10:02.190906 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 12:10:02.191129 (+ 223us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 12:10:02.191165 (+ 36us) client_negotiation.cc:769] Sending connection context
0809 12:10:02.191266 (+ 101us) client_negotiation.cc:240] Negotiation successful
0809 12:10:02.191402 (+ 136us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1183,"mutex_wait_us":147,"thread_start_us":855,"threads_started":1}
I20260809 12:10:02.191942 3024 leader_election.cc:302] T 4f441cf08efc474da096b2ebb6008ad0 P 26d0a424d542436a8830dbe1f8d5142e [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 26d0a424d542436a8830dbe1f8d5142e, 823d98af597d4559bc2e475e2d08a5df; no voters:
I20260809 12:10:02.192514 3330 raft_consensus.cc:2781] T 4f441cf08efc474da096b2ebb6008ad0 P 26d0a424d542436a8830dbe1f8d5142e [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260809 12:10:02.192659 3330 raft_consensus.cc:482] T 4f441cf08efc474da096b2ebb6008ad0 P 26d0a424d542436a8830dbe1f8d5142e [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260809 12:10:02.192767 3330 raft_consensus.cc:3037] T 4f441cf08efc474da096b2ebb6008ad0 P 26d0a424d542436a8830dbe1f8d5142e [term 0 FOLLOWER]: Advancing to term 1
I20260809 12:10:02.192917 2953 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4f441cf08efc474da096b2ebb6008ad0" candidate_uuid: "26d0a424d542436a8830dbe1f8d5142e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9cea8553e4754cd89ec242d7c7a78577" is_pre_election: true
I20260809 12:10:02.193389 2953 raft_consensus.cc:2454] T 4f441cf08efc474da096b2ebb6008ad0 P 9cea8553e4754cd89ec242d7c7a78577 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 26d0a424d542436a8830dbe1f8d5142e in term 0.
I20260809 12:10:02.196327 3330 raft_consensus.cc:504] T 4f441cf08efc474da096b2ebb6008ad0 P 26d0a424d542436a8830dbe1f8d5142e [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9cea8553e4754cd89ec242d7c7a78577" member_type: VOTER last_known_addr { host: "127.0.222.1" port: 35257 } } peers { permanent_uuid: "26d0a424d542436a8830dbe1f8d5142e" member_type: VOTER last_known_addr { host: "127.0.222.2" port: 37581 } } peers { permanent_uuid: "823d98af597d4559bc2e475e2d08a5df" member_type: VOTER last_known_addr { host: "127.0.222.3" port: 36309 } }
I20260809 12:10:02.197086 3330 leader_election.cc:288] T 4f441cf08efc474da096b2ebb6008ad0 P 26d0a424d542436a8830dbe1f8d5142e [CANDIDATE]: Term 1 election: Requested vote from peers 9cea8553e4754cd89ec242d7c7a78577 (127.0.222.1:35257), 823d98af597d4559bc2e475e2d08a5df (127.0.222.3:36309)
I20260809 12:10:02.197592 2953 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4f441cf08efc474da096b2ebb6008ad0" candidate_uuid: "26d0a424d542436a8830dbe1f8d5142e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9cea8553e4754cd89ec242d7c7a78577"
I20260809 12:10:02.197624 3249 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4f441cf08efc474da096b2ebb6008ad0" candidate_uuid: "26d0a424d542436a8830dbe1f8d5142e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "823d98af597d4559bc2e475e2d08a5df"
I20260809 12:10:02.197836 2953 raft_consensus.cc:3037] T 4f441cf08efc474da096b2ebb6008ad0 P 9cea8553e4754cd89ec242d7c7a78577 [term 0 FOLLOWER]: Advancing to term 1
I20260809 12:10:02.197862 3249 raft_consensus.cc:3037] T 4f441cf08efc474da096b2ebb6008ad0 P 823d98af597d4559bc2e475e2d08a5df [term 0 FOLLOWER]: Advancing to term 1
I20260809 12:10:02.200482 2953 raft_consensus.cc:2454] T 4f441cf08efc474da096b2ebb6008ad0 P 9cea8553e4754cd89ec242d7c7a78577 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 26d0a424d542436a8830dbe1f8d5142e in term 1.
I20260809 12:10:02.200954 3025 leader_election.cc:302] T 4f441cf08efc474da096b2ebb6008ad0 P 26d0a424d542436a8830dbe1f8d5142e [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 26d0a424d542436a8830dbe1f8d5142e, 9cea8553e4754cd89ec242d7c7a78577; no voters:
I20260809 12:10:02.201059 3249 raft_consensus.cc:2454] T 4f441cf08efc474da096b2ebb6008ad0 P 823d98af597d4559bc2e475e2d08a5df [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 26d0a424d542436a8830dbe1f8d5142e in term 1.
I20260809 12:10:02.201443 3330 raft_consensus.cc:2781] T 4f441cf08efc474da096b2ebb6008ad0 P 26d0a424d542436a8830dbe1f8d5142e [term 1 FOLLOWER]: Leader election won for term 1
I20260809 12:10:02.201951 3330 raft_consensus.cc:686] T 4f441cf08efc474da096b2ebb6008ad0 P 26d0a424d542436a8830dbe1f8d5142e [term 1 LEADER]: Becoming Leader. State: Replica: 26d0a424d542436a8830dbe1f8d5142e, State: Running, Role: LEADER
I20260809 12:10:02.202525 3330 consensus_queue.cc:237] T 4f441cf08efc474da096b2ebb6008ad0 P 26d0a424d542436a8830dbe1f8d5142e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9cea8553e4754cd89ec242d7c7a78577" member_type: VOTER last_known_addr { host: "127.0.222.1" port: 35257 } } peers { permanent_uuid: "26d0a424d542436a8830dbe1f8d5142e" member_type: VOTER last_known_addr { host: "127.0.222.2" port: 37581 } } peers { permanent_uuid: "823d98af597d4559bc2e475e2d08a5df" member_type: VOTER last_known_addr { host: "127.0.222.3" port: 36309 } }
I20260809 12:10:02.206878 2780 catalog_manager.cc:5495] T 4f441cf08efc474da096b2ebb6008ad0 P 26d0a424d542436a8830dbe1f8d5142e reported cstate change: term changed from 0 to 1, leader changed from <none> to 26d0a424d542436a8830dbe1f8d5142e (127.0.222.2). New cstate: current_term: 1 leader_uuid: "26d0a424d542436a8830dbe1f8d5142e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9cea8553e4754cd89ec242d7c7a78577" member_type: VOTER last_known_addr { host: "127.0.222.1" port: 35257 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "26d0a424d542436a8830dbe1f8d5142e" member_type: VOTER last_known_addr { host: "127.0.222.2" port: 37581 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "823d98af597d4559bc2e475e2d08a5df" member_type: VOTER last_known_addr { host: "127.0.222.3" port: 36309 } health_report { overall_health: UNKNOWN } } }
I20260809 12:10:02.252368 3381 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 12:10:02.246318 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48196 (local address 127.0.222.2:37581)
0809 12:10:02.247167 (+ 849us) server_negotiation.cc:206] Beginning negotiation
0809 12:10:02.247172 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0809 12:10:02.247189 (+ 17us) server_negotiation.cc:407] Connection header received
0809 12:10:02.247261 (+ 72us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 12:10:02.247265 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 12:10:02.247332 (+ 67us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 12:10:02.247510 (+ 178us) server_negotiation.cc:226] Negotiated authn=TOKEN
0809 12:10:02.247941 (+ 431us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 12:10:02.248716 (+ 775us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 12:10:02.250024 (+ 1308us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 12:10:02.250446 (+ 422us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 12:10:02.250580 (+ 134us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 12:10:02.251074 (+ 494us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0809 12:10:02.251638 (+ 564us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0809 12:10:02.251797 (+ 159us) server_negotiation.cc:1035] Waiting for connection context
0809 12:10:02.251898 (+ 101us) server_negotiation.cc:299] Negotiation successful
0809 12:10:02.251987 (+ 89us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":746,"thread_start_us":567,"threads_started":1}
I20260809 12:10:02.269395 3350 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 12:10:02.263226 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.222.2:42531 (local address 127.0.222.1:35257)
0809 12:10:02.263471 (+ 245us) server_negotiation.cc:206] Beginning negotiation
0809 12:10:02.263480 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0809 12:10:02.264043 (+ 563us) server_negotiation.cc:407] Connection header received
0809 12:10:02.264217 (+ 174us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 12:10:02.264220 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 12:10:02.264293 (+ 73us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 12:10:02.264368 (+ 75us) server_negotiation.cc:226] Negotiated authn=TOKEN
0809 12:10:02.264927 (+ 559us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 12:10:02.265653 (+ 726us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 12:10:02.267640 (+ 1987us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 12:10:02.268147 (+ 507us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 12:10:02.268244 (+ 97us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 12:10:02.268753 (+ 509us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0809 12:10:02.268866 (+ 113us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0809 12:10:02.268933 (+ 67us) server_negotiation.cc:1035] Waiting for connection context
0809 12:10:02.269081 (+ 148us) server_negotiation.cc:299] Negotiation successful
0809 12:10:02.269154 (+ 73us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":81}
I20260809 12:10:02.269567 3384 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 12:10:02.263202 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.222.1:35257 (local address 127.0.222.2:42531)
0809 12:10:02.263930 (+ 728us) negotiation.cc:107] Waiting for socket to connect
0809 12:10:02.263950 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0809 12:10:02.264097 (+ 147us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 12:10:02.264401 (+ 304us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 12:10:02.264406 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 12:10:02.264420 (+ 14us) client_negotiation.cc:189] Negotiated authn=TOKEN
0809 12:10:02.264771 (+ 351us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 12:10:02.264782 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 12:10:02.265768 (+ 986us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 12:10:02.265774 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 12:10:02.267502 (+ 1728us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 12:10:02.267517 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 12:10:02.268342 (+ 825us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 12:10:02.268348 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 12:10:02.268520 (+ 172us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 12:10:02.268588 (+ 68us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0809 12:10:02.268960 (+ 372us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0809 12:10:02.268971 (+ 11us) client_negotiation.cc:769] Sending connection context
0809 12:10:02.269051 (+ 80us) client_negotiation.cc:240] Negotiation successful
0809 12:10:02.269185 (+ 134us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":566,"thread_start_us":377,"threads_started":1}
I20260809 12:10:02.279253 3249 raft_consensus.cc:1261] T 4f441cf08efc474da096b2ebb6008ad0 P 823d98af597d4559bc2e475e2d08a5df [term 1 FOLLOWER]: Refusing update from remote peer 26d0a424d542436a8830dbe1f8d5142e: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260809 12:10:02.279559 2953 raft_consensus.cc:1261] T 4f441cf08efc474da096b2ebb6008ad0 P 9cea8553e4754cd89ec242d7c7a78577 [term 1 FOLLOWER]: Refusing update from remote peer 26d0a424d542436a8830dbe1f8d5142e: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260809 12:10:02.279976 3379 consensus_queue.cc:1035] T 4f441cf08efc474da096b2ebb6008ad0 P 26d0a424d542436a8830dbe1f8d5142e [LEADER]: Connected to new peer: Peer: permanent_uuid: "823d98af597d4559bc2e475e2d08a5df" member_type: VOTER last_known_addr { host: "127.0.222.3" port: 36309 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260809 12:10:02.280360 3330 consensus_queue.cc:1035] T 4f441cf08efc474da096b2ebb6008ad0 P 26d0a424d542436a8830dbe1f8d5142e [LEADER]: Connected to new peer: Peer: permanent_uuid: "9cea8553e4754cd89ec242d7c7a78577" member_type: VOTER last_known_addr { host: "127.0.222.1" port: 35257 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260809 12:10:02.312528 3381 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 12:10:02.306436 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.222.1:41947 (local address 127.0.222.2:37581)
0809 12:10:02.306597 (+ 161us) server_negotiation.cc:206] Beginning negotiation
0809 12:10:02.306603 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0809 12:10:02.306623 (+ 20us) server_negotiation.cc:407] Connection header received
0809 12:10:02.306684 (+ 61us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 12:10:02.306690 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 12:10:02.306767 (+ 77us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 12:10:02.306862 (+ 95us) server_negotiation.cc:226] Negotiated authn=TOKEN
0809 12:10:02.307512 (+ 650us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 12:10:02.308623 (+ 1111us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 12:10:02.310594 (+ 1971us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 12:10:02.311218 (+ 624us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 12:10:02.311361 (+ 143us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 12:10:02.311752 (+ 391us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0809 12:10:02.311858 (+ 106us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0809 12:10:02.311952 (+ 94us) server_negotiation.cc:1035] Waiting for connection context
0809 12:10:02.312139 (+ 187us) server_negotiation.cc:299] Negotiation successful
0809 12:10:02.312217 (+ 78us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":58}
I20260809 12:10:02.312572 3364 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 12:10:02.305810 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.222.2:37581 (local address 127.0.222.1:41947)
0809 12:10:02.306220 (+ 410us) negotiation.cc:107] Waiting for socket to connect
0809 12:10:02.306241 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0809 12:10:02.306355 (+ 114us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 12:10:02.307075 (+ 720us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 12:10:02.307079 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 12:10:02.307092 (+ 13us) client_negotiation.cc:189] Negotiated authn=TOKEN
0809 12:10:02.307340 (+ 248us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 12:10:02.307354 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 12:10:02.308781 (+ 1427us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 12:10:02.308787 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 12:10:02.310387 (+ 1600us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 12:10:02.310401 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 12:10:02.311371 (+ 970us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 12:10:02.311378 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 12:10:02.311551 (+ 173us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 12:10:02.311588 (+ 37us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0809 12:10:02.311956 (+ 368us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0809 12:10:02.311970 (+ 14us) client_negotiation.cc:769] Sending connection context
0809 12:10:02.312114 (+ 144us) client_negotiation.cc:240] Negotiation successful
0809 12:10:02.312255 (+ 141us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":273}
W20260809 12:10:02.314059 3352 tablet_replica.cc:1298] Aborted: operation has been aborted: cancelling pending write operations
I20260809 12:10:03.927187 2780 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:51984:
table { table_name: "test-table" } modify_external_catalogs: true
I20260809 12:10:03.927470 2780 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:51984:
table { table_name: "test-table" } modify_external_catalogs: true
I20260809 12:10:03.932778 2780 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 846fd7fb008b4e5ea8bd71f4646a10c4: Sending DeleteTablet for 3 replicas of tablet 4f441cf08efc474da096b2ebb6008ad0
I20260809 12:10:03.933717 2927 tablet_service.cc:1514] Processing DeleteTablet for tablet 4f441cf08efc474da096b2ebb6008ad0 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-09 12:10:03 UTC) from {username='kudu', principal='kudu/127.0.222.62@KRBTEST.COM'} at 127.0.0.1:42438
I20260809 12:10:03.933979 3082 tablet_service.cc:1514] Processing DeleteTablet for tablet 4f441cf08efc474da096b2ebb6008ad0 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-09 12:10:03 UTC) from {username='kudu', principal='kudu/127.0.222.62@KRBTEST.COM'} at 127.0.0.1:48188
I20260809 12:10:03.934010 3227 tablet_service.cc:1514] Processing DeleteTablet for tablet 4f441cf08efc474da096b2ebb6008ad0 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-09 12:10:03 UTC) from {username='kudu', principal='kudu/127.0.222.62@KRBTEST.COM'} at 127.0.0.1:42174
I20260809 12:10:03.946744 3415 tablet_replica.cc:330] T 4f441cf08efc474da096b2ebb6008ad0 P 9cea8553e4754cd89ec242d7c7a78577: stopping tablet replica
I20260809 12:10:03.946990 3416 tablet_replica.cc:330] T 4f441cf08efc474da096b2ebb6008ad0 P 26d0a424d542436a8830dbe1f8d5142e: stopping tablet replica
I20260809 12:10:03.947212 3417 tablet_replica.cc:330] T 4f441cf08efc474da096b2ebb6008ad0 P 823d98af597d4559bc2e475e2d08a5df: stopping tablet replica
I20260809 12:10:03.947278 3416 raft_consensus.cc:2229] T 4f441cf08efc474da096b2ebb6008ad0 P 26d0a424d542436a8830dbe1f8d5142e [term 1 LEADER]: Raft consensus shutting down.
I20260809 12:10:03.947254 3415 raft_consensus.cc:2229] T 4f441cf08efc474da096b2ebb6008ad0 P 9cea8553e4754cd89ec242d7c7a78577 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260809 12:10:03.947702 3415 raft_consensus.cc:2258] T 4f441cf08efc474da096b2ebb6008ad0 P 9cea8553e4754cd89ec242d7c7a78577 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260809 12:10:03.947745 3417 raft_consensus.cc:2229] T 4f441cf08efc474da096b2ebb6008ad0 P 823d98af597d4559bc2e475e2d08a5df [term 1 FOLLOWER]: Raft consensus shutting down.
I20260809 12:10:03.948094 3416 raft_consensus.cc:2258] T 4f441cf08efc474da096b2ebb6008ad0 P 26d0a424d542436a8830dbe1f8d5142e [term 1 FOLLOWER]: Raft consensus is shut down!
I20260809 12:10:03.948139 3417 raft_consensus.cc:2258] T 4f441cf08efc474da096b2ebb6008ad0 P 823d98af597d4559bc2e475e2d08a5df [term 1 FOLLOWER]: Raft consensus is shut down!
I20260809 12:10:03.949148 3415 ts_tablet_manager.cc:1905] T 4f441cf08efc474da096b2ebb6008ad0 P 9cea8553e4754cd89ec242d7c7a78577: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260809 12:10:03.949272 3416 ts_tablet_manager.cc:1905] T 4f441cf08efc474da096b2ebb6008ad0 P 26d0a424d542436a8830dbe1f8d5142e: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260809 12:10:03.949309 3417 ts_tablet_manager.cc:1905] T 4f441cf08efc474da096b2ebb6008ad0 P 823d98af597d4559bc2e475e2d08a5df: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260809 12:10:03.957451 3416 ts_tablet_manager.cc:1918] T 4f441cf08efc474da096b2ebb6008ad0 P 26d0a424d542436a8830dbe1f8d5142e: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.8
I20260809 12:10:03.957484 3417 ts_tablet_manager.cc:1918] T 4f441cf08efc474da096b2ebb6008ad0 P 823d98af597d4559bc2e475e2d08a5df: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.8
I20260809 12:10:03.957659 3417 log.cc:1196] T 4f441cf08efc474da096b2ebb6008ad0 P 823d98af597d4559bc2e475e2d08a5df: Deleting WAL directory at /tmp/dist-test-taskgH5Pjm/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786277382073938-888-0/minicluster-data/ts-2/wal/wals/4f441cf08efc474da096b2ebb6008ad0
I20260809 12:10:03.957813 3416 log.cc:1196] T 4f441cf08efc474da096b2ebb6008ad0 P 26d0a424d542436a8830dbe1f8d5142e: Deleting WAL directory at /tmp/dist-test-taskgH5Pjm/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786277382073938-888-0/minicluster-data/ts-1/wal/wals/4f441cf08efc474da096b2ebb6008ad0
I20260809 12:10:03.957847 3415 ts_tablet_manager.cc:1918] T 4f441cf08efc474da096b2ebb6008ad0 P 9cea8553e4754cd89ec242d7c7a78577: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.8
I20260809 12:10:03.958124 3415 log.cc:1196] T 4f441cf08efc474da096b2ebb6008ad0 P 9cea8553e4754cd89ec242d7c7a78577: Deleting WAL directory at /tmp/dist-test-taskgH5Pjm/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786277382073938-888-0/minicluster-data/ts-0/wal/wals/4f441cf08efc474da096b2ebb6008ad0
I20260809 12:10:03.958523 3417 ts_tablet_manager.cc:1939] T 4f441cf08efc474da096b2ebb6008ad0 P 823d98af597d4559bc2e475e2d08a5df: Deleting consensus metadata
I20260809 12:10:03.958565 3416 ts_tablet_manager.cc:1939] T 4f441cf08efc474da096b2ebb6008ad0 P 26d0a424d542436a8830dbe1f8d5142e: Deleting consensus metadata
I20260809 12:10:03.958851 3415 ts_tablet_manager.cc:1939] T 4f441cf08efc474da096b2ebb6008ad0 P 9cea8553e4754cd89ec242d7c7a78577: Deleting consensus metadata
I20260809 12:10:03.959808 2762 catalog_manager.cc:4841] TS 823d98af597d4559bc2e475e2d08a5df (127.0.222.3:36309): tablet 4f441cf08efc474da096b2ebb6008ad0 (table test-table [id=654250b2eb3e43408280fab55a9c48e0]) successfully deleted
I20260809 12:10:03.960140 2762 catalog_manager.cc:4841] TS 26d0a424d542436a8830dbe1f8d5142e (127.0.222.2:37581): tablet 4f441cf08efc474da096b2ebb6008ad0 (table test-table [id=654250b2eb3e43408280fab55a9c48e0]) successfully deleted
I20260809 12:10:03.960389 2763 catalog_manager.cc:4841] TS 9cea8553e4754cd89ec242d7c7a78577 (127.0.222.1:35257): tablet 4f441cf08efc474da096b2ebb6008ad0 (table test-table [id=654250b2eb3e43408280fab55a9c48e0]) successfully deleted
Aug 09 12:10:03 dist-test-slave-2w3w krb5kdc[2731](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786277403, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-user@KRBTEST.COM:
Aug 09 12:10:03 dist-test-slave-2w3w krb5kdc[2731](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786277403, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.0.222.62@KRBTEST.COM
I20260809 12:10:04.003080 3424 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 12:10:03.986771 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51998 (local address 127.0.222.62:42305)
0809 12:10:03.987474 (+ 703us) server_negotiation.cc:206] Beginning negotiation
0809 12:10:03.987479 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0809 12:10:03.987667 (+ 188us) server_negotiation.cc:407] Connection header received
0809 12:10:03.987875 (+ 208us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 12:10:03.987880 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 12:10:03.987948 (+ 68us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 12:10:03.988180 (+ 232us) server_negotiation.cc:226] Negotiated authn=SASL
0809 12:10:03.989526 (+ 1346us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 12:10:03.990297 (+ 771us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 12:10:03.991631 (+ 1334us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 12:10:03.992009 (+ 378us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 12:10:03.992103 (+ 94us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 12:10:03.996621 (+ 4518us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 12:10:03.996626 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 12:10:03.996629 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0809 12:10:03.996661 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0809 12:10:04.000107 (+ 3446us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 12:10:04.000786 (+ 679us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 12:10:04.000792 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 12:10:04.000796 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0809 12:10:04.000996 (+ 200us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 12:10:04.001418 (+ 422us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 12:10:04.001423 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 12:10:04.001427 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0809 12:10:04.001827 (+ 400us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 12:10:04.002094 (+ 267us) server_negotiation.cc:1035] Waiting for connection context
0809 12:10:04.002370 (+ 276us) server_negotiation.cc:299] Negotiation successful
0809 12:10:04.002609 (+ 239us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":554,"thread_start_us":375,"threads_started":1}
I20260809 12:10:04.007174 2780 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:51998:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260809 12:10:04.007975 2780 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260809 12:10:04.015228 2927 tablet_service.cc:1467] Processing CreateTablet for tablet 80d1c004871f494f92836219d15effda (DEFAULT_TABLE table=test-table [id=a1185a0d34744b67a85ecef76ff4c0cd]), partition=RANGE (key) PARTITION UNBOUNDED
I20260809 12:10:04.015806 2927 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 80d1c004871f494f92836219d15effda. 1 dirs total, 0 dirs full, 0 dirs failed
I20260809 12:10:04.015931 3082 tablet_service.cc:1467] Processing CreateTablet for tablet 80d1c004871f494f92836219d15effda (DEFAULT_TABLE table=test-table [id=a1185a0d34744b67a85ecef76ff4c0cd]), partition=RANGE (key) PARTITION UNBOUNDED
I20260809 12:10:04.016407 3082 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 80d1c004871f494f92836219d15effda. 1 dirs total, 0 dirs full, 0 dirs failed
I20260809 12:10:04.023403 3227 tablet_service.cc:1467] Processing CreateTablet for tablet 80d1c004871f494f92836219d15effda (DEFAULT_TABLE table=test-table [id=a1185a0d34744b67a85ecef76ff4c0cd]), partition=RANGE (key) PARTITION UNBOUNDED
I20260809 12:10:04.023830 3227 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 80d1c004871f494f92836219d15effda. 1 dirs total, 0 dirs full, 0 dirs failed
I20260809 12:10:04.027882 3427 tablet_bootstrap.cc:492] T 80d1c004871f494f92836219d15effda P 9cea8553e4754cd89ec242d7c7a78577: Bootstrap starting.
I20260809 12:10:04.030937 3427 tablet_bootstrap.cc:654] T 80d1c004871f494f92836219d15effda P 9cea8553e4754cd89ec242d7c7a78577: Neither blocks nor log segments found. Creating new log.
I20260809 12:10:04.032171 3428 tablet_bootstrap.cc:492] T 80d1c004871f494f92836219d15effda P 26d0a424d542436a8830dbe1f8d5142e: Bootstrap starting.
I20260809 12:10:04.032931 3429 tablet_bootstrap.cc:492] T 80d1c004871f494f92836219d15effda P 823d98af597d4559bc2e475e2d08a5df: Bootstrap starting.
I20260809 12:10:04.035271 3429 tablet_bootstrap.cc:654] T 80d1c004871f494f92836219d15effda P 823d98af597d4559bc2e475e2d08a5df: Neither blocks nor log segments found. Creating new log.
I20260809 12:10:04.036701 3428 tablet_bootstrap.cc:654] T 80d1c004871f494f92836219d15effda P 26d0a424d542436a8830dbe1f8d5142e: Neither blocks nor log segments found. Creating new log.
I20260809 12:10:04.037892 3429 tablet_bootstrap.cc:492] T 80d1c004871f494f92836219d15effda P 823d98af597d4559bc2e475e2d08a5df: No bootstrap required, opened a new log
I20260809 12:10:04.038107 3429 ts_tablet_manager.cc:1397] T 80d1c004871f494f92836219d15effda P 823d98af597d4559bc2e475e2d08a5df: Time spent bootstrapping tablet: real 0.005s user 0.001s sys 0.003s
I20260809 12:10:04.039110 3429 raft_consensus.cc:348] T 80d1c004871f494f92836219d15effda P 823d98af597d4559bc2e475e2d08a5df [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9cea8553e4754cd89ec242d7c7a78577" member_type: VOTER last_known_addr { host: "127.0.222.1" port: 35257 } } peers { permanent_uuid: "26d0a424d542436a8830dbe1f8d5142e" member_type: VOTER last_known_addr { host: "127.0.222.2" port: 37581 } } peers { permanent_uuid: "823d98af597d4559bc2e475e2d08a5df" member_type: VOTER last_known_addr { host: "127.0.222.3" port: 36309 } }
I20260809 12:10:04.039682 3429 raft_consensus.cc:374] T 80d1c004871f494f92836219d15effda P 823d98af597d4559bc2e475e2d08a5df [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260809 12:10:04.039937 3429 raft_consensus.cc:729] T 80d1c004871f494f92836219d15effda P 823d98af597d4559bc2e475e2d08a5df [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 823d98af597d4559bc2e475e2d08a5df, State: Initialized, Role: FOLLOWER
I20260809 12:10:04.040288 3429 consensus_queue.cc:260] T 80d1c004871f494f92836219d15effda P 823d98af597d4559bc2e475e2d08a5df [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9cea8553e4754cd89ec242d7c7a78577" member_type: VOTER last_known_addr { host: "127.0.222.1" port: 35257 } } peers { permanent_uuid: "26d0a424d542436a8830dbe1f8d5142e" member_type: VOTER last_known_addr { host: "127.0.222.2" port: 37581 } } peers { permanent_uuid: "823d98af597d4559bc2e475e2d08a5df" member_type: VOTER last_known_addr { host: "127.0.222.3" port: 36309 } }
I20260809 12:10:04.041023 3428 tablet_bootstrap.cc:492] T 80d1c004871f494f92836219d15effda P 26d0a424d542436a8830dbe1f8d5142e: No bootstrap required, opened a new log
I20260809 12:10:04.041257 3428 ts_tablet_manager.cc:1397] T 80d1c004871f494f92836219d15effda P 26d0a424d542436a8830dbe1f8d5142e: Time spent bootstrapping tablet: real 0.009s user 0.000s sys 0.007s
I20260809 12:10:04.042515 3429 ts_tablet_manager.cc:1428] T 80d1c004871f494f92836219d15effda P 823d98af597d4559bc2e475e2d08a5df: Time spent starting tablet: real 0.004s user 0.000s sys 0.003s
I20260809 12:10:04.042258 3428 raft_consensus.cc:348] T 80d1c004871f494f92836219d15effda P 26d0a424d542436a8830dbe1f8d5142e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9cea8553e4754cd89ec242d7c7a78577" member_type: VOTER last_known_addr { host: "127.0.222.1" port: 35257 } } peers { permanent_uuid: "26d0a424d542436a8830dbe1f8d5142e" member_type: VOTER last_known_addr { host: "127.0.222.2" port: 37581 } } peers { permanent_uuid: "823d98af597d4559bc2e475e2d08a5df" member_type: VOTER last_known_addr { host: "127.0.222.3" port: 36309 } }
I20260809 12:10:04.042657 3428 raft_consensus.cc:374] T 80d1c004871f494f92836219d15effda P 26d0a424d542436a8830dbe1f8d5142e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260809 12:10:04.042982 3428 raft_consensus.cc:729] T 80d1c004871f494f92836219d15effda P 26d0a424d542436a8830dbe1f8d5142e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 26d0a424d542436a8830dbe1f8d5142e, State: Initialized, Role: FOLLOWER
I20260809 12:10:04.043366 3428 consensus_queue.cc:260] T 80d1c004871f494f92836219d15effda P 26d0a424d542436a8830dbe1f8d5142e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9cea8553e4754cd89ec242d7c7a78577" member_type: VOTER last_known_addr { host: "127.0.222.1" port: 35257 } } peers { permanent_uuid: "26d0a424d542436a8830dbe1f8d5142e" member_type: VOTER last_known_addr { host: "127.0.222.2" port: 37581 } } peers { permanent_uuid: "823d98af597d4559bc2e475e2d08a5df" member_type: VOTER last_known_addr { host: "127.0.222.3" port: 36309 } }
I20260809 12:10:04.043771 3427 tablet_bootstrap.cc:492] T 80d1c004871f494f92836219d15effda P 9cea8553e4754cd89ec242d7c7a78577: No bootstrap required, opened a new log
I20260809 12:10:04.044003 3427 ts_tablet_manager.cc:1397] T 80d1c004871f494f92836219d15effda P 9cea8553e4754cd89ec242d7c7a78577: Time spent bootstrapping tablet: real 0.016s user 0.004s sys 0.009s
I20260809 12:10:04.044787 3428 ts_tablet_manager.cc:1428] T 80d1c004871f494f92836219d15effda P 26d0a424d542436a8830dbe1f8d5142e: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20260809 12:10:04.045348 3427 raft_consensus.cc:348] T 80d1c004871f494f92836219d15effda P 9cea8553e4754cd89ec242d7c7a78577 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9cea8553e4754cd89ec242d7c7a78577" member_type: VOTER last_known_addr { host: "127.0.222.1" port: 35257 } } peers { permanent_uuid: "26d0a424d542436a8830dbe1f8d5142e" member_type: VOTER last_known_addr { host: "127.0.222.2" port: 37581 } } peers { permanent_uuid: "823d98af597d4559bc2e475e2d08a5df" member_type: VOTER last_known_addr { host: "127.0.222.3" port: 36309 } }
I20260809 12:10:04.045715 3427 raft_consensus.cc:374] T 80d1c004871f494f92836219d15effda P 9cea8553e4754cd89ec242d7c7a78577 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260809 12:10:04.045929 3427 raft_consensus.cc:729] T 80d1c004871f494f92836219d15effda P 9cea8553e4754cd89ec242d7c7a78577 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9cea8553e4754cd89ec242d7c7a78577, State: Initialized, Role: FOLLOWER
I20260809 12:10:04.046221 3427 consensus_queue.cc:260] T 80d1c004871f494f92836219d15effda P 9cea8553e4754cd89ec242d7c7a78577 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9cea8553e4754cd89ec242d7c7a78577" member_type: VOTER last_known_addr { host: "127.0.222.1" port: 35257 } } peers { permanent_uuid: "26d0a424d542436a8830dbe1f8d5142e" member_type: VOTER last_known_addr { host: "127.0.222.2" port: 37581 } } peers { permanent_uuid: "823d98af597d4559bc2e475e2d08a5df" member_type: VOTER last_known_addr { host: "127.0.222.3" port: 36309 } }
I20260809 12:10:04.046841 3427 ts_tablet_manager.cc:1428] T 80d1c004871f494f92836219d15effda P 9cea8553e4754cd89ec242d7c7a78577: Time spent starting tablet: real 0.003s user 0.001s sys 0.002s
I20260809 12:10:04.064301 3394 raft_consensus.cc:482] T 80d1c004871f494f92836219d15effda P 9cea8553e4754cd89ec242d7c7a78577 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260809 12:10:04.064668 3394 raft_consensus.cc:504] T 80d1c004871f494f92836219d15effda P 9cea8553e4754cd89ec242d7c7a78577 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9cea8553e4754cd89ec242d7c7a78577" member_type: VOTER last_known_addr { host: "127.0.222.1" port: 35257 } } peers { permanent_uuid: "26d0a424d542436a8830dbe1f8d5142e" member_type: VOTER last_known_addr { host: "127.0.222.2" port: 37581 } } peers { permanent_uuid: "823d98af597d4559bc2e475e2d08a5df" member_type: VOTER last_known_addr { host: "127.0.222.3" port: 36309 } }
I20260809 12:10:04.065905 3394 leader_election.cc:288] T 80d1c004871f494f92836219d15effda P 9cea8553e4754cd89ec242d7c7a78577 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 26d0a424d542436a8830dbe1f8d5142e (127.0.222.2:37581), 823d98af597d4559bc2e475e2d08a5df (127.0.222.3:36309)
I20260809 12:10:04.066377 3106 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "80d1c004871f494f92836219d15effda" candidate_uuid: "9cea8553e4754cd89ec242d7c7a78577" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "26d0a424d542436a8830dbe1f8d5142e" is_pre_election: true
I20260809 12:10:04.066453 3250 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "80d1c004871f494f92836219d15effda" candidate_uuid: "9cea8553e4754cd89ec242d7c7a78577" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "823d98af597d4559bc2e475e2d08a5df" is_pre_election: true
I20260809 12:10:04.066708 3250 raft_consensus.cc:2454] T 80d1c004871f494f92836219d15effda P 823d98af597d4559bc2e475e2d08a5df [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9cea8553e4754cd89ec242d7c7a78577 in term 0.
I20260809 12:10:04.066727 3106 raft_consensus.cc:2454] T 80d1c004871f494f92836219d15effda P 26d0a424d542436a8830dbe1f8d5142e [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9cea8553e4754cd89ec242d7c7a78577 in term 0.
I20260809 12:10:04.067435 2873 leader_election.cc:302] T 80d1c004871f494f92836219d15effda P 9cea8553e4754cd89ec242d7c7a78577 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 26d0a424d542436a8830dbe1f8d5142e, 9cea8553e4754cd89ec242d7c7a78577; no voters:
I20260809 12:10:04.067830 3394 raft_consensus.cc:2781] T 80d1c004871f494f92836219d15effda P 9cea8553e4754cd89ec242d7c7a78577 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260809 12:10:04.068006 3394 raft_consensus.cc:482] T 80d1c004871f494f92836219d15effda P 9cea8553e4754cd89ec242d7c7a78577 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260809 12:10:04.068159 3394 raft_consensus.cc:3037] T 80d1c004871f494f92836219d15effda P 9cea8553e4754cd89ec242d7c7a78577 [term 0 FOLLOWER]: Advancing to term 1
I20260809 12:10:04.071092 3394 raft_consensus.cc:504] T 80d1c004871f494f92836219d15effda P 9cea8553e4754cd89ec242d7c7a78577 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9cea8553e4754cd89ec242d7c7a78577" member_type: VOTER last_known_addr { host: "127.0.222.1" port: 35257 } } peers { permanent_uuid: "26d0a424d542436a8830dbe1f8d5142e" member_type: VOTER last_known_addr { host: "127.0.222.2" port: 37581 } } peers { permanent_uuid: "823d98af597d4559bc2e475e2d08a5df" member_type: VOTER last_known_addr { host: "127.0.222.3" port: 36309 } }
I20260809 12:10:04.071870 3394 leader_election.cc:288] T 80d1c004871f494f92836219d15effda P 9cea8553e4754cd89ec242d7c7a78577 [CANDIDATE]: Term 1 election: Requested vote from peers 26d0a424d542436a8830dbe1f8d5142e (127.0.222.2:37581), 823d98af597d4559bc2e475e2d08a5df (127.0.222.3:36309)
I20260809 12:10:04.072546 3250 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "80d1c004871f494f92836219d15effda" candidate_uuid: "9cea8553e4754cd89ec242d7c7a78577" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "823d98af597d4559bc2e475e2d08a5df"
I20260809 12:10:04.072785 3250 raft_consensus.cc:3037] T 80d1c004871f494f92836219d15effda P 823d98af597d4559bc2e475e2d08a5df [term 0 FOLLOWER]: Advancing to term 1
I20260809 12:10:04.072671 3106 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "80d1c004871f494f92836219d15effda" candidate_uuid: "9cea8553e4754cd89ec242d7c7a78577" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "26d0a424d542436a8830dbe1f8d5142e"
I20260809 12:10:04.073154 3106 raft_consensus.cc:3037] T 80d1c004871f494f92836219d15effda P 26d0a424d542436a8830dbe1f8d5142e [term 0 FOLLOWER]: Advancing to term 1
I20260809 12:10:04.076298 3106 raft_consensus.cc:2454] T 80d1c004871f494f92836219d15effda P 26d0a424d542436a8830dbe1f8d5142e [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9cea8553e4754cd89ec242d7c7a78577 in term 1.
I20260809 12:10:04.076527 3250 raft_consensus.cc:2454] T 80d1c004871f494f92836219d15effda P 823d98af597d4559bc2e475e2d08a5df [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9cea8553e4754cd89ec242d7c7a78577 in term 1.
I20260809 12:10:04.076835 2873 leader_election.cc:302] T 80d1c004871f494f92836219d15effda P 9cea8553e4754cd89ec242d7c7a78577 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 26d0a424d542436a8830dbe1f8d5142e, 9cea8553e4754cd89ec242d7c7a78577; no voters:
I20260809 12:10:04.077230 3394 raft_consensus.cc:2781] T 80d1c004871f494f92836219d15effda P 9cea8553e4754cd89ec242d7c7a78577 [term 1 FOLLOWER]: Leader election won for term 1
I20260809 12:10:04.077420 3394 raft_consensus.cc:686] T 80d1c004871f494f92836219d15effda P 9cea8553e4754cd89ec242d7c7a78577 [term 1 LEADER]: Becoming Leader. State: Replica: 9cea8553e4754cd89ec242d7c7a78577, State: Running, Role: LEADER
I20260809 12:10:04.077749 3394 consensus_queue.cc:237] T 80d1c004871f494f92836219d15effda P 9cea8553e4754cd89ec242d7c7a78577 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9cea8553e4754cd89ec242d7c7a78577" member_type: VOTER last_known_addr { host: "127.0.222.1" port: 35257 } } peers { permanent_uuid: "26d0a424d542436a8830dbe1f8d5142e" member_type: VOTER last_known_addr { host: "127.0.222.2" port: 37581 } } peers { permanent_uuid: "823d98af597d4559bc2e475e2d08a5df" member_type: VOTER last_known_addr { host: "127.0.222.3" port: 36309 } }
I20260809 12:10:04.081167 2778 catalog_manager.cc:5495] T 80d1c004871f494f92836219d15effda P 9cea8553e4754cd89ec242d7c7a78577 reported cstate change: term changed from 0 to 1, leader changed from <none> to 9cea8553e4754cd89ec242d7c7a78577 (127.0.222.1). New cstate: current_term: 1 leader_uuid: "9cea8553e4754cd89ec242d7c7a78577" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9cea8553e4754cd89ec242d7c7a78577" member_type: VOTER last_known_addr { host: "127.0.222.1" port: 35257 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "26d0a424d542436a8830dbe1f8d5142e" member_type: VOTER last_known_addr { host: "127.0.222.2" port: 37581 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "823d98af597d4559bc2e475e2d08a5df" member_type: VOTER last_known_addr { host: "127.0.222.3" port: 36309 } health_report { overall_health: UNKNOWN } } }
I20260809 12:10:04.117002 3436 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 12:10:04.110627 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42472 (local address 127.0.222.1:35257)
0809 12:10:04.111635 (+ 1008us) server_negotiation.cc:206] Beginning negotiation
0809 12:10:04.111639 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0809 12:10:04.111663 (+ 24us) server_negotiation.cc:407] Connection header received
0809 12:10:04.111714 (+ 51us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 12:10:04.111718 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 12:10:04.111788 (+ 70us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 12:10:04.111884 (+ 96us) server_negotiation.cc:226] Negotiated authn=TOKEN
0809 12:10:04.112318 (+ 434us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 12:10:04.113284 (+ 966us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 12:10:04.114806 (+ 1522us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 12:10:04.115376 (+ 570us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 12:10:04.115480 (+ 104us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 12:10:04.116168 (+ 688us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0809 12:10:04.116314 (+ 146us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0809 12:10:04.116394 (+ 80us) server_negotiation.cc:1035] Waiting for connection context
0809 12:10:04.116510 (+ 116us) server_negotiation.cc:299] Negotiation successful
0809 12:10:04.116572 (+ 62us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":892,"thread_start_us":681,"threads_started":1}
I20260809 12:10:04.127825 3106 raft_consensus.cc:1261] T 80d1c004871f494f92836219d15effda P 26d0a424d542436a8830dbe1f8d5142e [term 1 FOLLOWER]: Refusing update from remote peer 9cea8553e4754cd89ec242d7c7a78577: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260809 12:10:04.128217 3250 raft_consensus.cc:1261] T 80d1c004871f494f92836219d15effda P 823d98af597d4559bc2e475e2d08a5df [term 1 FOLLOWER]: Refusing update from remote peer 9cea8553e4754cd89ec242d7c7a78577: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260809 12:10:04.128538 3439 consensus_queue.cc:1035] T 80d1c004871f494f92836219d15effda P 9cea8553e4754cd89ec242d7c7a78577 [LEADER]: Connected to new peer: Peer: permanent_uuid: "26d0a424d542436a8830dbe1f8d5142e" member_type: VOTER last_known_addr { host: "127.0.222.2" port: 37581 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20260809 12:10:04.129209 3369 consensus_queue.cc:1035] T 80d1c004871f494f92836219d15effda P 9cea8553e4754cd89ec242d7c7a78577 [LEADER]: Connected to new peer: Peer: permanent_uuid: "823d98af597d4559bc2e475e2d08a5df" member_type: VOTER last_known_addr { host: "127.0.222.3" port: 36309 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260809 12:10:04.185743 2778 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:51998:
table { table_name: "test-table" } modify_external_catalogs: true
I20260809 12:10:04.186043 2778 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:51998:
table { table_name: "test-table" } modify_external_catalogs: true
I20260809 12:10:04.189218 2778 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 846fd7fb008b4e5ea8bd71f4646a10c4: Sending DeleteTablet for 3 replicas of tablet 80d1c004871f494f92836219d15effda
I20260809 12:10:04.190188 2928 tablet_service.cc:1514] Processing DeleteTablet for tablet 80d1c004871f494f92836219d15effda with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-09 12:10:04 UTC) from {username='kudu', principal='kudu/127.0.222.62@KRBTEST.COM'} at 127.0.0.1:42438
I20260809 12:10:04.190265 3082 tablet_service.cc:1514] Processing DeleteTablet for tablet 80d1c004871f494f92836219d15effda with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-09 12:10:04 UTC) from {username='kudu', principal='kudu/127.0.222.62@KRBTEST.COM'} at 127.0.0.1:48188
I20260809 12:10:04.190372 3227 tablet_service.cc:1514] Processing DeleteTablet for tablet 80d1c004871f494f92836219d15effda with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-09 12:10:04 UTC) from {username='kudu', principal='kudu/127.0.222.62@KRBTEST.COM'} at 127.0.0.1:42174
I20260809 12:10:04.190579 3415 tablet_replica.cc:330] T 80d1c004871f494f92836219d15effda P 9cea8553e4754cd89ec242d7c7a78577: stopping tablet replica
I20260809 12:10:04.190606 3416 tablet_replica.cc:330] T 80d1c004871f494f92836219d15effda P 26d0a424d542436a8830dbe1f8d5142e: stopping tablet replica
I20260809 12:10:04.190855 3415 raft_consensus.cc:2229] T 80d1c004871f494f92836219d15effda P 9cea8553e4754cd89ec242d7c7a78577 [term 1 LEADER]: Raft consensus shutting down.
I20260809 12:10:04.190917 3417 tablet_replica.cc:330] T 80d1c004871f494f92836219d15effda P 823d98af597d4559bc2e475e2d08a5df: stopping tablet replica
I20260809 12:10:04.190919 3416 raft_consensus.cc:2229] T 80d1c004871f494f92836219d15effda P 26d0a424d542436a8830dbe1f8d5142e [term 1 FOLLOWER]: Raft consensus shutting down.
I20260809 12:10:04.191210 3416 raft_consensus.cc:2258] T 80d1c004871f494f92836219d15effda P 26d0a424d542436a8830dbe1f8d5142e [term 1 FOLLOWER]: Raft consensus is shut down!
I20260809 12:10:04.191239 3417 raft_consensus.cc:2229] T 80d1c004871f494f92836219d15effda P 823d98af597d4559bc2e475e2d08a5df [term 1 FOLLOWER]: Raft consensus shutting down.
I20260809 12:10:04.191319 3415 raft_consensus.cc:2258] T 80d1c004871f494f92836219d15effda P 9cea8553e4754cd89ec242d7c7a78577 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260809 12:10:04.191578 3417 raft_consensus.cc:2258] T 80d1c004871f494f92836219d15effda P 823d98af597d4559bc2e475e2d08a5df [term 1 FOLLOWER]: Raft consensus is shut down!
I20260809 12:10:04.191692 2778 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:51998:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20260809 12:10:04.192365 3416 ts_tablet_manager.cc:1905] T 80d1c004871f494f92836219d15effda P 26d0a424d542436a8830dbe1f8d5142e: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260809 12:10:04.192606 3415 ts_tablet_manager.cc:1905] T 80d1c004871f494f92836219d15effda P 9cea8553e4754cd89ec242d7c7a78577: Deleting tablet data with delete state TABLET_DATA_DELETED
W20260809 12:10:04.192608 2778 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260809 12:10:04.192974 3417 ts_tablet_manager.cc:1905] T 80d1c004871f494f92836219d15effda P 823d98af597d4559bc2e475e2d08a5df: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260809 12:10:04.199651 3417 ts_tablet_manager.cc:1918] T 80d1c004871f494f92836219d15effda P 823d98af597d4559bc2e475e2d08a5df: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260809 12:10:04.199816 3417 log.cc:1196] T 80d1c004871f494f92836219d15effda P 823d98af597d4559bc2e475e2d08a5df: Deleting WAL directory at /tmp/dist-test-taskgH5Pjm/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786277382073938-888-0/minicluster-data/ts-2/wal/wals/80d1c004871f494f92836219d15effda
I20260809 12:10:04.200425 3417 ts_tablet_manager.cc:1939] T 80d1c004871f494f92836219d15effda P 823d98af597d4559bc2e475e2d08a5df: Deleting consensus metadata
I20260809 12:10:04.200639 3415 ts_tablet_manager.cc:1918] T 80d1c004871f494f92836219d15effda P 9cea8553e4754cd89ec242d7c7a78577: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260809 12:10:04.200856 3415 log.cc:1196] T 80d1c004871f494f92836219d15effda P 9cea8553e4754cd89ec242d7c7a78577: Deleting WAL directory at /tmp/dist-test-taskgH5Pjm/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786277382073938-888-0/minicluster-data/ts-0/wal/wals/80d1c004871f494f92836219d15effda
I20260809 12:10:04.201335 3415 ts_tablet_manager.cc:1939] T 80d1c004871f494f92836219d15effda P 9cea8553e4754cd89ec242d7c7a78577: Deleting consensus metadata
I20260809 12:10:04.201699 2762 catalog_manager.cc:4841] TS 823d98af597d4559bc2e475e2d08a5df (127.0.222.3:36309): tablet 80d1c004871f494f92836219d15effda (table test-table [id=a1185a0d34744b67a85ecef76ff4c0cd]) successfully deleted
I20260809 12:10:04.202329 3416 ts_tablet_manager.cc:1918] T 80d1c004871f494f92836219d15effda P 26d0a424d542436a8830dbe1f8d5142e: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260809 12:10:04.202579 3416 log.cc:1196] T 80d1c004871f494f92836219d15effda P 26d0a424d542436a8830dbe1f8d5142e: Deleting WAL directory at /tmp/dist-test-taskgH5Pjm/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786277382073938-888-0/minicluster-data/ts-1/wal/wals/80d1c004871f494f92836219d15effda
I20260809 12:10:04.202554 2763 catalog_manager.cc:4841] TS 9cea8553e4754cd89ec242d7c7a78577 (127.0.222.1:35257): tablet 80d1c004871f494f92836219d15effda (table test-table [id=a1185a0d34744b67a85ecef76ff4c0cd]) successfully deleted
I20260809 12:10:04.203130 3416 ts_tablet_manager.cc:1939] T 80d1c004871f494f92836219d15effda P 26d0a424d542436a8830dbe1f8d5142e: Deleting consensus metadata
I20260809 12:10:04.204249 2762 catalog_manager.cc:4841] TS 26d0a424d542436a8830dbe1f8d5142e (127.0.222.2:37581): tablet 80d1c004871f494f92836219d15effda (table test-table [id=a1185a0d34744b67a85ecef76ff4c0cd]) successfully deleted
I20260809 12:10:04.206398 2928 tablet_service.cc:1467] Processing CreateTablet for tablet 7166bffc9e3e4059959b06db9c1c4b7c (DEFAULT_TABLE table=test-table [id=bff4ce4e6a30480b86c12cf5b4c1f9fa]), partition=RANGE (key) PARTITION UNBOUNDED
I20260809 12:10:04.206544 3082 tablet_service.cc:1467] Processing CreateTablet for tablet 7166bffc9e3e4059959b06db9c1c4b7c (DEFAULT_TABLE table=test-table [id=bff4ce4e6a30480b86c12cf5b4c1f9fa]), partition=RANGE (key) PARTITION UNBOUNDED
I20260809 12:10:04.206923 3082 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 7166bffc9e3e4059959b06db9c1c4b7c. 1 dirs total, 0 dirs full, 0 dirs failed
I20260809 12:10:04.207077 2928 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 7166bffc9e3e4059959b06db9c1c4b7c. 1 dirs total, 0 dirs full, 0 dirs failed
I20260809 12:10:04.207199 3227 tablet_service.cc:1467] Processing CreateTablet for tablet 7166bffc9e3e4059959b06db9c1c4b7c (DEFAULT_TABLE table=test-table [id=bff4ce4e6a30480b86c12cf5b4c1f9fa]), partition=RANGE (key) PARTITION UNBOUNDED
I20260809 12:10:04.207823 3227 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 7166bffc9e3e4059959b06db9c1c4b7c. 1 dirs total, 0 dirs full, 0 dirs failed
I20260809 12:10:04.214196 3428 tablet_bootstrap.cc:492] T 7166bffc9e3e4059959b06db9c1c4b7c P 26d0a424d542436a8830dbe1f8d5142e: Bootstrap starting.
I20260809 12:10:04.216152 3427 tablet_bootstrap.cc:492] T 7166bffc9e3e4059959b06db9c1c4b7c P 9cea8553e4754cd89ec242d7c7a78577: Bootstrap starting.
I20260809 12:10:04.216814 3428 tablet_bootstrap.cc:654] T 7166bffc9e3e4059959b06db9c1c4b7c P 26d0a424d542436a8830dbe1f8d5142e: Neither blocks nor log segments found. Creating new log.
I20260809 12:10:04.219151 3428 tablet_bootstrap.cc:492] T 7166bffc9e3e4059959b06db9c1c4b7c P 26d0a424d542436a8830dbe1f8d5142e: No bootstrap required, opened a new log
I20260809 12:10:04.219465 3428 ts_tablet_manager.cc:1397] T 7166bffc9e3e4059959b06db9c1c4b7c P 26d0a424d542436a8830dbe1f8d5142e: Time spent bootstrapping tablet: real 0.005s user 0.001s sys 0.004s
I20260809 12:10:04.219692 3427 tablet_bootstrap.cc:654] T 7166bffc9e3e4059959b06db9c1c4b7c P 9cea8553e4754cd89ec242d7c7a78577: Neither blocks nor log segments found. Creating new log.
I20260809 12:10:04.220428 3428 raft_consensus.cc:348] T 7166bffc9e3e4059959b06db9c1c4b7c P 26d0a424d542436a8830dbe1f8d5142e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9cea8553e4754cd89ec242d7c7a78577" member_type: VOTER last_known_addr { host: "127.0.222.1" port: 35257 } } peers { permanent_uuid: "26d0a424d542436a8830dbe1f8d5142e" member_type: VOTER last_known_addr { host: "127.0.222.2" port: 37581 } } peers { permanent_uuid: "823d98af597d4559bc2e475e2d08a5df" member_type: VOTER last_known_addr { host: "127.0.222.3" port: 36309 } }
I20260809 12:10:04.220849 3428 raft_consensus.cc:374] T 7166bffc9e3e4059959b06db9c1c4b7c P 26d0a424d542436a8830dbe1f8d5142e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260809 12:10:04.221004 3428 raft_consensus.cc:729] T 7166bffc9e3e4059959b06db9c1c4b7c P 26d0a424d542436a8830dbe1f8d5142e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 26d0a424d542436a8830dbe1f8d5142e, State: Initialized, Role: FOLLOWER
I20260809 12:10:04.221346 3428 consensus_queue.cc:260] T 7166bffc9e3e4059959b06db9c1c4b7c P 26d0a424d542436a8830dbe1f8d5142e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9cea8553e4754cd89ec242d7c7a78577" member_type: VOTER last_known_addr { host: "127.0.222.1" port: 35257 } } peers { permanent_uuid: "26d0a424d542436a8830dbe1f8d5142e" member_type: VOTER last_known_addr { host: "127.0.222.2" port: 37581 } } peers { permanent_uuid: "823d98af597d4559bc2e475e2d08a5df" member_type: VOTER last_known_addr { host: "127.0.222.3" port: 36309 } }
I20260809 12:10:04.222349 3428 ts_tablet_manager.cc:1428] T 7166bffc9e3e4059959b06db9c1c4b7c P 26d0a424d542436a8830dbe1f8d5142e: Time spent starting tablet: real 0.003s user 0.002s sys 0.000s
I20260809 12:10:04.223026 3427 tablet_bootstrap.cc:492] T 7166bffc9e3e4059959b06db9c1c4b7c P 9cea8553e4754cd89ec242d7c7a78577: No bootstrap required, opened a new log
I20260809 12:10:04.223330 3427 ts_tablet_manager.cc:1397] T 7166bffc9e3e4059959b06db9c1c4b7c P 9cea8553e4754cd89ec242d7c7a78577: Time spent bootstrapping tablet: real 0.007s user 0.000s sys 0.006s
I20260809 12:10:04.224880 3429 tablet_bootstrap.cc:492] T 7166bffc9e3e4059959b06db9c1c4b7c P 823d98af597d4559bc2e475e2d08a5df: Bootstrap starting.
I20260809 12:10:04.224366 3427 raft_consensus.cc:348] T 7166bffc9e3e4059959b06db9c1c4b7c P 9cea8553e4754cd89ec242d7c7a78577 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9cea8553e4754cd89ec242d7c7a78577" member_type: VOTER last_known_addr { host: "127.0.222.1" port: 35257 } } peers { permanent_uuid: "26d0a424d542436a8830dbe1f8d5142e" member_type: VOTER last_known_addr { host: "127.0.222.2" port: 37581 } } peers { permanent_uuid: "823d98af597d4559bc2e475e2d08a5df" member_type: VOTER last_known_addr { host: "127.0.222.3" port: 36309 } }
I20260809 12:10:04.227240 3427 raft_consensus.cc:374] T 7166bffc9e3e4059959b06db9c1c4b7c P 9cea8553e4754cd89ec242d7c7a78577 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260809 12:10:04.227406 3427 raft_consensus.cc:729] T 7166bffc9e3e4059959b06db9c1c4b7c P 9cea8553e4754cd89ec242d7c7a78577 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9cea8553e4754cd89ec242d7c7a78577, State: Initialized, Role: FOLLOWER
I20260809 12:10:04.227623 3429 tablet_bootstrap.cc:654] T 7166bffc9e3e4059959b06db9c1c4b7c P 823d98af597d4559bc2e475e2d08a5df: Neither blocks nor log segments found. Creating new log.
I20260809 12:10:04.227751 3427 consensus_queue.cc:260] T 7166bffc9e3e4059959b06db9c1c4b7c P 9cea8553e4754cd89ec242d7c7a78577 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9cea8553e4754cd89ec242d7c7a78577" member_type: VOTER last_known_addr { host: "127.0.222.1" port: 35257 } } peers { permanent_uuid: "26d0a424d542436a8830dbe1f8d5142e" member_type: VOTER last_known_addr { host: "127.0.222.2" port: 37581 } } peers { permanent_uuid: "823d98af597d4559bc2e475e2d08a5df" member_type: VOTER last_known_addr { host: "127.0.222.3" port: 36309 } }
I20260809 12:10:04.228300 3427 ts_tablet_manager.cc:1428] T 7166bffc9e3e4059959b06db9c1c4b7c P 9cea8553e4754cd89ec242d7c7a78577: Time spent starting tablet: real 0.005s user 0.000s sys 0.002s
I20260809 12:10:04.230891 3429 tablet_bootstrap.cc:492] T 7166bffc9e3e4059959b06db9c1c4b7c P 823d98af597d4559bc2e475e2d08a5df: No bootstrap required, opened a new log
I20260809 12:10:04.231231 3429 ts_tablet_manager.cc:1397] T 7166bffc9e3e4059959b06db9c1c4b7c P 823d98af597d4559bc2e475e2d08a5df: Time spent bootstrapping tablet: real 0.006s user 0.004s sys 0.002s
I20260809 12:10:04.232095 3429 raft_consensus.cc:348] T 7166bffc9e3e4059959b06db9c1c4b7c P 823d98af597d4559bc2e475e2d08a5df [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9cea8553e4754cd89ec242d7c7a78577" member_type: VOTER last_known_addr { host: "127.0.222.1" port: 35257 } } peers { permanent_uuid: "26d0a424d542436a8830dbe1f8d5142e" member_type: VOTER last_known_addr { host: "127.0.222.2" port: 37581 } } peers { permanent_uuid: "823d98af597d4559bc2e475e2d08a5df" member_type: VOTER last_known_addr { host: "127.0.222.3" port: 36309 } }
I20260809 12:10:04.232424 3429 raft_consensus.cc:374] T 7166bffc9e3e4059959b06db9c1c4b7c P 823d98af597d4559bc2e475e2d08a5df [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260809 12:10:04.232590 3429 raft_consensus.cc:729] T 7166bffc9e3e4059959b06db9c1c4b7c P 823d98af597d4559bc2e475e2d08a5df [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 823d98af597d4559bc2e475e2d08a5df, State: Initialized, Role: FOLLOWER
I20260809 12:10:04.232918 3429 consensus_queue.cc:260] T 7166bffc9e3e4059959b06db9c1c4b7c P 823d98af597d4559bc2e475e2d08a5df [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9cea8553e4754cd89ec242d7c7a78577" member_type: VOTER last_known_addr { host: "127.0.222.1" port: 35257 } } peers { permanent_uuid: "26d0a424d542436a8830dbe1f8d5142e" member_type: VOTER last_known_addr { host: "127.0.222.2" port: 37581 } } peers { permanent_uuid: "823d98af597d4559bc2e475e2d08a5df" member_type: VOTER last_known_addr { host: "127.0.222.3" port: 36309 } }
I20260809 12:10:04.233490 3429 ts_tablet_manager.cc:1428] T 7166bffc9e3e4059959b06db9c1c4b7c P 823d98af597d4559bc2e475e2d08a5df: Time spent starting tablet: real 0.002s user 0.000s sys 0.000s
I20260809 12:10:04.521842 3433 raft_consensus.cc:482] T 7166bffc9e3e4059959b06db9c1c4b7c P 823d98af597d4559bc2e475e2d08a5df [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260809 12:10:04.522068 3433 raft_consensus.cc:504] T 7166bffc9e3e4059959b06db9c1c4b7c P 823d98af597d4559bc2e475e2d08a5df [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9cea8553e4754cd89ec242d7c7a78577" member_type: VOTER last_known_addr { host: "127.0.222.1" port: 35257 } } peers { permanent_uuid: "26d0a424d542436a8830dbe1f8d5142e" member_type: VOTER last_known_addr { host: "127.0.222.2" port: 37581 } } peers { permanent_uuid: "823d98af597d4559bc2e475e2d08a5df" member_type: VOTER last_known_addr { host: "127.0.222.3" port: 36309 } }
I20260809 12:10:04.523906 3433 leader_election.cc:288] T 7166bffc9e3e4059959b06db9c1c4b7c P 823d98af597d4559bc2e475e2d08a5df [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 9cea8553e4754cd89ec242d7c7a78577 (127.0.222.1:35257), 26d0a424d542436a8830dbe1f8d5142e (127.0.222.2:37581)
I20260809 12:10:04.534539 3436 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 12:10:04.524602 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.222.3:43149 (local address 127.0.222.1:35257)
0809 12:10:04.524932 (+ 330us) server_negotiation.cc:206] Beginning negotiation
0809 12:10:04.524941 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0809 12:10:04.526646 (+ 1705us) server_negotiation.cc:407] Connection header received
0809 12:10:04.527019 (+ 373us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 12:10:04.527029 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 12:10:04.527115 (+ 86us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 12:10:04.527187 (+ 72us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0809 12:10:04.528347 (+ 1160us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 12:10:04.529225 (+ 878us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 12:10:04.532003 (+ 2778us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 12:10:04.533280 (+ 1277us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 12:10:04.533410 (+ 130us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 12:10:04.533494 (+ 84us) server_negotiation.cc:1035] Waiting for connection context
0809 12:10:04.533977 (+ 483us) server_negotiation.cc:299] Negotiation successful
0809 12:10:04.534125 (+ 148us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":98}
I20260809 12:10:04.534878 3449 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 12:10:04.524040 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.222.1:35257 (local address 127.0.222.3:43149)
0809 12:10:04.526520 (+ 2480us) negotiation.cc:107] Waiting for socket to connect
0809 12:10:04.526536 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0809 12:10:04.526770 (+ 234us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 12:10:04.527235 (+ 465us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 12:10:04.527243 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 12:10:04.527269 (+ 26us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0809 12:10:04.528065 (+ 796us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 12:10:04.528078 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 12:10:04.529418 (+ 1340us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 12:10:04.529424 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 12:10:04.531848 (+ 2424us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 12:10:04.531861 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 12:10:04.533660 (+ 1799us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 12:10:04.533665 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 12:10:04.533808 (+ 143us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 12:10:04.533833 (+ 25us) client_negotiation.cc:769] Sending connection context
0809 12:10:04.533946 (+ 113us) client_negotiation.cc:240] Negotiation successful
0809 12:10:04.534083 (+ 137us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":2356,"mutex_wait_us":44,"spinlock_wait_cycles":1408,"thread_start_us":2131,"threads_started":1}
I20260809 12:10:04.535889 3446 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 12:10:04.524255 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.222.2:37581 (local address 127.0.222.3:38311)
0809 12:10:04.526720 (+ 2465us) negotiation.cc:107] Waiting for socket to connect
0809 12:10:04.526735 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0809 12:10:04.526909 (+ 174us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 12:10:04.527306 (+ 397us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 12:10:04.527313 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 12:10:04.527343 (+ 30us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0809 12:10:04.528173 (+ 830us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 12:10:04.528187 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 12:10:04.529896 (+ 1709us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 12:10:04.529903 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 12:10:04.532311 (+ 2408us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 12:10:04.532339 (+ 28us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 12:10:04.534988 (+ 2649us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 12:10:04.534996 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 12:10:04.535209 (+ 213us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 12:10:04.535244 (+ 35us) client_negotiation.cc:769] Sending connection context
0809 12:10:04.535338 (+ 94us) client_negotiation.cc:240] Negotiation successful
0809 12:10:04.535459 (+ 121us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":2286,"thread_start_us":1770,"threads_started":1}
I20260809 12:10:04.535766 2953 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7166bffc9e3e4059959b06db9c1c4b7c" candidate_uuid: "823d98af597d4559bc2e475e2d08a5df" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9cea8553e4754cd89ec242d7c7a78577" is_pre_election: true
I20260809 12:10:04.536194 2953 raft_consensus.cc:2454] T 7166bffc9e3e4059959b06db9c1c4b7c P 9cea8553e4754cd89ec242d7c7a78577 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 823d98af597d4559bc2e475e2d08a5df in term 0.
I20260809 12:10:04.536873 3448 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 12:10:04.524408 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.222.3:38311 (local address 127.0.222.2:37581)
0809 12:10:04.525736 (+ 1328us) server_negotiation.cc:206] Beginning negotiation
0809 12:10:04.525745 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0809 12:10:04.526832 (+ 1087us) server_negotiation.cc:407] Connection header received
0809 12:10:04.527080 (+ 248us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 12:10:04.527085 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 12:10:04.527147 (+ 62us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 12:10:04.527269 (+ 122us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0809 12:10:04.528434 (+ 1165us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 12:10:04.529732 (+ 1298us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 12:10:04.532525 (+ 2793us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 12:10:04.534320 (+ 1795us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 12:10:04.534518 (+ 198us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 12:10:04.534720 (+ 202us) server_negotiation.cc:1035] Waiting for connection context
0809 12:10:04.536158 (+ 1438us) server_negotiation.cc:299] Negotiation successful
0809 12:10:04.536318 (+ 160us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1105,"thread_start_us":912,"threads_started":1}
I20260809 12:10:04.536916 3177 leader_election.cc:302] T 7166bffc9e3e4059959b06db9c1c4b7c P 823d98af597d4559bc2e475e2d08a5df [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 823d98af597d4559bc2e475e2d08a5df, 9cea8553e4754cd89ec242d7c7a78577; no voters:
I20260809 12:10:04.537537 3433 raft_consensus.cc:2781] T 7166bffc9e3e4059959b06db9c1c4b7c P 823d98af597d4559bc2e475e2d08a5df [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260809 12:10:04.537736 3433 raft_consensus.cc:482] T 7166bffc9e3e4059959b06db9c1c4b7c P 823d98af597d4559bc2e475e2d08a5df [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260809 12:10:04.537858 3433 raft_consensus.cc:3037] T 7166bffc9e3e4059959b06db9c1c4b7c P 823d98af597d4559bc2e475e2d08a5df [term 0 FOLLOWER]: Advancing to term 1
I20260809 12:10:04.542191 3433 raft_consensus.cc:504] T 7166bffc9e3e4059959b06db9c1c4b7c P 823d98af597d4559bc2e475e2d08a5df [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9cea8553e4754cd89ec242d7c7a78577" member_type: VOTER last_known_addr { host: "127.0.222.1" port: 35257 } } peers { permanent_uuid: "26d0a424d542436a8830dbe1f8d5142e" member_type: VOTER last_known_addr { host: "127.0.222.2" port: 37581 } } peers { permanent_uuid: "823d98af597d4559bc2e475e2d08a5df" member_type: VOTER last_known_addr { host: "127.0.222.3" port: 36309 } }
I20260809 12:10:04.542701 3105 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7166bffc9e3e4059959b06db9c1c4b7c" candidate_uuid: "823d98af597d4559bc2e475e2d08a5df" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "26d0a424d542436a8830dbe1f8d5142e" is_pre_election: true
I20260809 12:10:04.543136 3105 raft_consensus.cc:2454] T 7166bffc9e3e4059959b06db9c1c4b7c P 26d0a424d542436a8830dbe1f8d5142e [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 823d98af597d4559bc2e475e2d08a5df in term 0.
I20260809 12:10:04.543184 3433 leader_election.cc:288] T 7166bffc9e3e4059959b06db9c1c4b7c P 823d98af597d4559bc2e475e2d08a5df [CANDIDATE]: Term 1 election: Requested vote from peers 9cea8553e4754cd89ec242d7c7a78577 (127.0.222.1:35257), 26d0a424d542436a8830dbe1f8d5142e (127.0.222.2:37581)
I20260809 12:10:04.543747 3105 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7166bffc9e3e4059959b06db9c1c4b7c" candidate_uuid: "823d98af597d4559bc2e475e2d08a5df" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "26d0a424d542436a8830dbe1f8d5142e"
I20260809 12:10:04.543793 2953 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7166bffc9e3e4059959b06db9c1c4b7c" candidate_uuid: "823d98af597d4559bc2e475e2d08a5df" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9cea8553e4754cd89ec242d7c7a78577"
I20260809 12:10:04.544023 3105 raft_consensus.cc:3037] T 7166bffc9e3e4059959b06db9c1c4b7c P 26d0a424d542436a8830dbe1f8d5142e [term 0 FOLLOWER]: Advancing to term 1
I20260809 12:10:04.544113 2953 raft_consensus.cc:3037] T 7166bffc9e3e4059959b06db9c1c4b7c P 9cea8553e4754cd89ec242d7c7a78577 [term 0 FOLLOWER]: Advancing to term 1
I20260809 12:10:04.546846 2953 raft_consensus.cc:2454] T 7166bffc9e3e4059959b06db9c1c4b7c P 9cea8553e4754cd89ec242d7c7a78577 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 823d98af597d4559bc2e475e2d08a5df in term 1.
I20260809 12:10:04.547362 3177 leader_election.cc:302] T 7166bffc9e3e4059959b06db9c1c4b7c P 823d98af597d4559bc2e475e2d08a5df [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 823d98af597d4559bc2e475e2d08a5df, 9cea8553e4754cd89ec242d7c7a78577; no voters:
I20260809 12:10:04.547829 3433 raft_consensus.cc:2781] T 7166bffc9e3e4059959b06db9c1c4b7c P 823d98af597d4559bc2e475e2d08a5df [term 1 FOLLOWER]: Leader election won for term 1
I20260809 12:10:04.548167 3105 raft_consensus.cc:2454] T 7166bffc9e3e4059959b06db9c1c4b7c P 26d0a424d542436a8830dbe1f8d5142e [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 823d98af597d4559bc2e475e2d08a5df in term 1.
I20260809 12:10:04.548808 3433 raft_consensus.cc:686] T 7166bffc9e3e4059959b06db9c1c4b7c P 823d98af597d4559bc2e475e2d08a5df [term 1 LEADER]: Becoming Leader. State: Replica: 823d98af597d4559bc2e475e2d08a5df, State: Running, Role: LEADER
I20260809 12:10:04.549304 3433 consensus_queue.cc:237] T 7166bffc9e3e4059959b06db9c1c4b7c P 823d98af597d4559bc2e475e2d08a5df [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9cea8553e4754cd89ec242d7c7a78577" member_type: VOTER last_known_addr { host: "127.0.222.1" port: 35257 } } peers { permanent_uuid: "26d0a424d542436a8830dbe1f8d5142e" member_type: VOTER last_known_addr { host: "127.0.222.2" port: 37581 } } peers { permanent_uuid: "823d98af597d4559bc2e475e2d08a5df" member_type: VOTER last_known_addr { host: "127.0.222.3" port: 36309 } }
I20260809 12:10:04.553447 2778 catalog_manager.cc:5495] T 7166bffc9e3e4059959b06db9c1c4b7c P 823d98af597d4559bc2e475e2d08a5df reported cstate change: term changed from 0 to 1, leader changed from <none> to 823d98af597d4559bc2e475e2d08a5df (127.0.222.3). New cstate: current_term: 1 leader_uuid: "823d98af597d4559bc2e475e2d08a5df" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9cea8553e4754cd89ec242d7c7a78577" member_type: VOTER last_known_addr { host: "127.0.222.1" port: 35257 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "26d0a424d542436a8830dbe1f8d5142e" member_type: VOTER last_known_addr { host: "127.0.222.2" port: 37581 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "823d98af597d4559bc2e475e2d08a5df" member_type: VOTER last_known_addr { host: "127.0.222.3" port: 36309 } health_report { overall_health: HEALTHY } } }
I20260809 12:10:04.602744 3452 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 12:10:04.594577 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42182 (local address 127.0.222.3:36309)
0809 12:10:04.595865 (+ 1288us) server_negotiation.cc:206] Beginning negotiation
0809 12:10:04.595870 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0809 12:10:04.595910 (+ 40us) server_negotiation.cc:407] Connection header received
0809 12:10:04.595995 (+ 85us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 12:10:04.595999 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 12:10:04.596070 (+ 71us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 12:10:04.596285 (+ 215us) server_negotiation.cc:226] Negotiated authn=TOKEN
0809 12:10:04.597050 (+ 765us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 12:10:04.598108 (+ 1058us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 12:10:04.600041 (+ 1933us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 12:10:04.600644 (+ 603us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 12:10:04.600743 (+ 99us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 12:10:04.601350 (+ 607us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0809 12:10:04.601953 (+ 603us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0809 12:10:04.602062 (+ 109us) server_negotiation.cc:1035] Waiting for connection context
0809 12:10:04.602173 (+ 111us) server_negotiation.cc:299] Negotiation successful
0809 12:10:04.602237 (+ 64us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1103,"thread_start_us":913,"threads_started":1}
I20260809 12:10:04.622357 3436 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 12:10:04.614344 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.222.3:35655 (local address 127.0.222.1:35257)
0809 12:10:04.614510 (+ 166us) server_negotiation.cc:206] Beginning negotiation
0809 12:10:04.614514 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0809 12:10:04.615900 (+ 1386us) server_negotiation.cc:407] Connection header received
0809 12:10:04.616154 (+ 254us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 12:10:04.616158 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 12:10:04.616260 (+ 102us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 12:10:04.616366 (+ 106us) server_negotiation.cc:226] Negotiated authn=TOKEN
0809 12:10:04.617121 (+ 755us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 12:10:04.618248 (+ 1127us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 12:10:04.620069 (+ 1821us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 12:10:04.620693 (+ 624us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 12:10:04.620851 (+ 158us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 12:10:04.621387 (+ 536us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0809 12:10:04.621515 (+ 128us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0809 12:10:04.621619 (+ 104us) server_negotiation.cc:1035] Waiting for connection context
0809 12:10:04.621782 (+ 163us) server_negotiation.cc:299] Negotiation successful
0809 12:10:04.621864 (+ 82us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":55}
I20260809 12:10:04.622605 3455 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 12:10:04.614512 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.222.1:35257 (local address 127.0.222.3:35655)
0809 12:10:04.615787 (+ 1275us) negotiation.cc:107] Waiting for socket to connect
0809 12:10:04.615803 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0809 12:10:04.615993 (+ 190us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 12:10:04.616381 (+ 388us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 12:10:04.616386 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 12:10:04.616398 (+ 12us) client_negotiation.cc:189] Negotiated authn=TOKEN
0809 12:10:04.616895 (+ 497us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 12:10:04.616904 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 12:10:04.618426 (+ 1522us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 12:10:04.618431 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 12:10:04.619906 (+ 1475us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 12:10:04.619918 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 12:10:04.620931 (+ 1013us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 12:10:04.620940 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 12:10:04.621105 (+ 165us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 12:10:04.621173 (+ 68us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0809 12:10:04.621622 (+ 449us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0809 12:10:04.621636 (+ 14us) client_negotiation.cc:769] Sending connection context
0809 12:10:04.621734 (+ 98us) client_negotiation.cc:240] Negotiation successful
0809 12:10:04.621867 (+ 133us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1132,"thread_start_us":758,"threads_started":1}
I20260809 12:10:04.631572 2953 raft_consensus.cc:1261] T 7166bffc9e3e4059959b06db9c1c4b7c P 9cea8553e4754cd89ec242d7c7a78577 [term 1 FOLLOWER]: Refusing update from remote peer 823d98af597d4559bc2e475e2d08a5df: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260809 12:10:04.631904 3105 raft_consensus.cc:1261] T 7166bffc9e3e4059959b06db9c1c4b7c P 26d0a424d542436a8830dbe1f8d5142e [term 1 FOLLOWER]: Refusing update from remote peer 823d98af597d4559bc2e475e2d08a5df: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260809 12:10:04.632493 3450 consensus_queue.cc:1035] T 7166bffc9e3e4059959b06db9c1c4b7c P 823d98af597d4559bc2e475e2d08a5df [LEADER]: Connected to new peer: Peer: permanent_uuid: "9cea8553e4754cd89ec242d7c7a78577" member_type: VOTER last_known_addr { host: "127.0.222.1" port: 35257 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260809 12:10:04.633112 3433 consensus_queue.cc:1035] T 7166bffc9e3e4059959b06db9c1c4b7c P 823d98af597d4559bc2e475e2d08a5df [LEADER]: Connected to new peer: Peer: permanent_uuid: "26d0a424d542436a8830dbe1f8d5142e" member_type: VOTER last_known_addr { host: "127.0.222.2" port: 37581 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260809 12:10:04.670838 3452 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 12:10:04.656705 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.222.1:59031 (local address 127.0.222.3:36309)
0809 12:10:04.656930 (+ 225us) server_negotiation.cc:206] Beginning negotiation
0809 12:10:04.656940 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0809 12:10:04.664727 (+ 7787us) server_negotiation.cc:407] Connection header received
0809 12:10:04.664882 (+ 155us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 12:10:04.664886 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 12:10:04.664953 (+ 67us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 12:10:04.665026 (+ 73us) server_negotiation.cc:226] Negotiated authn=TOKEN
0809 12:10:04.665761 (+ 735us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 12:10:04.666698 (+ 937us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 12:10:04.668449 (+ 1751us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 12:10:04.669149 (+ 700us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 12:10:04.669296 (+ 147us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 12:10:04.669944 (+ 648us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0809 12:10:04.670057 (+ 113us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0809 12:10:04.670163 (+ 106us) server_negotiation.cc:1035] Waiting for connection context
0809 12:10:04.670368 (+ 205us) server_negotiation.cc:299] Negotiation successful
0809 12:10:04.670441 (+ 73us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":72}
I20260809 12:10:04.671015 3461 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 12:10:04.656471 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.222.3:36309 (local address 127.0.222.1:59031)
0809 12:10:04.664612 (+ 8141us) negotiation.cc:107] Waiting for socket to connect
0809 12:10:04.664630 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0809 12:10:04.664780 (+ 150us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 12:10:04.665059 (+ 279us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 12:10:04.665062 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 12:10:04.665072 (+ 10us) client_negotiation.cc:189] Negotiated authn=TOKEN
0809 12:10:04.665597 (+ 525us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 12:10:04.665606 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 12:10:04.666836 (+ 1230us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 12:10:04.666839 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 12:10:04.668314 (+ 1475us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 12:10:04.668327 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 12:10:04.669417 (+ 1090us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 12:10:04.669423 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 12:10:04.669693 (+ 270us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 12:10:04.669734 (+ 41us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0809 12:10:04.670214 (+ 480us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0809 12:10:04.670228 (+ 14us) client_negotiation.cc:769] Sending connection context
0809 12:10:04.670351 (+ 123us) client_negotiation.cc:240] Negotiation successful
0809 12:10:04.670511 (+ 160us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":7988,"thread_start_us":7685,"threads_started":1}
W20260809 12:10:04.672389 3395 tablet_replica.cc:1298] Aborted: operation has been aborted: cancelling pending write operations
I20260809 12:10:06.286922 2778 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:51998:
table { table_name: "test-table" } modify_external_catalogs: true
I20260809 12:10:06.287282 2778 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:51998:
table { table_name: "test-table" } modify_external_catalogs: true
I20260809 12:10:06.292402 2778 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 846fd7fb008b4e5ea8bd71f4646a10c4: Sending DeleteTablet for 3 replicas of tablet 7166bffc9e3e4059959b06db9c1c4b7c
I20260809 12:10:06.293248 2927 tablet_service.cc:1514] Processing DeleteTablet for tablet 7166bffc9e3e4059959b06db9c1c4b7c with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-09 12:10:06 UTC) from {username='kudu', principal='kudu/127.0.222.62@KRBTEST.COM'} at 127.0.0.1:42438
I20260809 12:10:06.293485 3082 tablet_service.cc:1514] Processing DeleteTablet for tablet 7166bffc9e3e4059959b06db9c1c4b7c with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-09 12:10:06 UTC) from {username='kudu', principal='kudu/127.0.222.62@KRBTEST.COM'} at 127.0.0.1:48188
I20260809 12:10:06.293978 3227 tablet_service.cc:1514] Processing DeleteTablet for tablet 7166bffc9e3e4059959b06db9c1c4b7c with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-09 12:10:06 UTC) from {username='kudu', principal='kudu/127.0.222.62@KRBTEST.COM'} at 127.0.0.1:42174
I20260809 12:10:06.294730 888 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskgH5Pjm/build/asan/bin/kudu pid 2856
I20260809 12:10:06.295193 3486 tablet_replica.cc:330] T 7166bffc9e3e4059959b06db9c1c4b7c P 9cea8553e4754cd89ec242d7c7a78577: stopping tablet replica
I20260809 12:10:06.295516 3486 raft_consensus.cc:2229] T 7166bffc9e3e4059959b06db9c1c4b7c P 9cea8553e4754cd89ec242d7c7a78577 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260809 12:10:06.295991 3486 raft_consensus.cc:2258] T 7166bffc9e3e4059959b06db9c1c4b7c P 9cea8553e4754cd89ec242d7c7a78577 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260809 12:10:06.297293 3488 tablet_replica.cc:330] T 7166bffc9e3e4059959b06db9c1c4b7c P 823d98af597d4559bc2e475e2d08a5df: stopping tablet replica
I20260809 12:10:06.297555 3488 raft_consensus.cc:2229] T 7166bffc9e3e4059959b06db9c1c4b7c P 823d98af597d4559bc2e475e2d08a5df [term 1 LEADER]: Raft consensus shutting down.
I20260809 12:10:06.297857 3487 tablet_replica.cc:330] T 7166bffc9e3e4059959b06db9c1c4b7c P 26d0a424d542436a8830dbe1f8d5142e: stopping tablet replica
I20260809 12:10:06.298062 3487 raft_consensus.cc:2229] T 7166bffc9e3e4059959b06db9c1c4b7c P 26d0a424d542436a8830dbe1f8d5142e [term 1 FOLLOWER]: Raft consensus shutting down.
I20260809 12:10:06.298255 3486 ts_tablet_manager.cc:1905] T 7166bffc9e3e4059959b06db9c1c4b7c P 9cea8553e4754cd89ec242d7c7a78577: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260809 12:10:06.298357 3488 raft_consensus.cc:2258] T 7166bffc9e3e4059959b06db9c1c4b7c P 823d98af597d4559bc2e475e2d08a5df [term 1 FOLLOWER]: Raft consensus is shut down!
I20260809 12:10:06.298466 3487 raft_consensus.cc:2258] T 7166bffc9e3e4059959b06db9c1c4b7c P 26d0a424d542436a8830dbe1f8d5142e [term 1 FOLLOWER]: Raft consensus is shut down!
I20260809 12:10:06.299685 3487 ts_tablet_manager.cc:1905] T 7166bffc9e3e4059959b06db9c1c4b7c P 26d0a424d542436a8830dbe1f8d5142e: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260809 12:10:06.299887 3488 ts_tablet_manager.cc:1905] T 7166bffc9e3e4059959b06db9c1c4b7c P 823d98af597d4559bc2e475e2d08a5df: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260809 12:10:06.308482 3486 ts_tablet_manager.cc:1918] T 7166bffc9e3e4059959b06db9c1c4b7c P 9cea8553e4754cd89ec242d7c7a78577: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.8
I20260809 12:10:06.308775 3488 ts_tablet_manager.cc:1918] T 7166bffc9e3e4059959b06db9c1c4b7c P 823d98af597d4559bc2e475e2d08a5df: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.8
I20260809 12:10:06.308753 3487 ts_tablet_manager.cc:1918] T 7166bffc9e3e4059959b06db9c1c4b7c P 26d0a424d542436a8830dbe1f8d5142e: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.8
I20260809 12:10:06.308838 3486 log.cc:1196] T 7166bffc9e3e4059959b06db9c1c4b7c P 9cea8553e4754cd89ec242d7c7a78577: Deleting WAL directory at /tmp/dist-test-taskgH5Pjm/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786277382073938-888-0/minicluster-data/ts-0/wal/wals/7166bffc9e3e4059959b06db9c1c4b7c
I20260809 12:10:06.309007 3487 log.cc:1196] T 7166bffc9e3e4059959b06db9c1c4b7c P 26d0a424d542436a8830dbe1f8d5142e: Deleting WAL directory at /tmp/dist-test-taskgH5Pjm/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786277382073938-888-0/minicluster-data/ts-1/wal/wals/7166bffc9e3e4059959b06db9c1c4b7c
I20260809 12:10:06.309062 3488 log.cc:1196] T 7166bffc9e3e4059959b06db9c1c4b7c P 823d98af597d4559bc2e475e2d08a5df: Deleting WAL directory at /tmp/dist-test-taskgH5Pjm/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786277382073938-888-0/minicluster-data/ts-2/wal/wals/7166bffc9e3e4059959b06db9c1c4b7c
I20260809 12:10:06.309670 3486 ts_tablet_manager.cc:1939] T 7166bffc9e3e4059959b06db9c1c4b7c P 9cea8553e4754cd89ec242d7c7a78577: Deleting consensus metadata
I20260809 12:10:06.309971 3487 ts_tablet_manager.cc:1939] T 7166bffc9e3e4059959b06db9c1c4b7c P 26d0a424d542436a8830dbe1f8d5142e: Deleting consensus metadata
I20260809 12:10:06.309996 3488 ts_tablet_manager.cc:1939] T 7166bffc9e3e4059959b06db9c1c4b7c P 823d98af597d4559bc2e475e2d08a5df: Deleting consensus metadata
I20260809 12:10:06.311218 2762 catalog_manager.cc:4841] TS 823d98af597d4559bc2e475e2d08a5df (127.0.222.3:36309): tablet 7166bffc9e3e4059959b06db9c1c4b7c (table test-table [id=bff4ce4e6a30480b86c12cf5b4c1f9fa]) successfully deleted
I20260809 12:10:06.311566 2763 catalog_manager.cc:4841] TS 9cea8553e4754cd89ec242d7c7a78577 (127.0.222.1:35257): tablet 7166bffc9e3e4059959b06db9c1c4b7c (table test-table [id=bff4ce4e6a30480b86c12cf5b4c1f9fa]) successfully deleted
I20260809 12:10:06.311674 2762 catalog_manager.cc:4841] TS 26d0a424d542436a8830dbe1f8d5142e (127.0.222.2:37581): tablet 7166bffc9e3e4059959b06db9c1c4b7c (table test-table [id=bff4ce4e6a30480b86c12cf5b4c1f9fa]) successfully deleted
Aug 09 12:10:06 dist-test-slave-2w3w krb5kdc[2731](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786277403, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.0.222.1@KRBTEST.COM
I20260809 12:10:06.330864 3490 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 12:10:06.298629 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42488 (local address 127.0.222.1:35257)
0809 12:10:06.311812 (+ 13183us) server_negotiation.cc:206] Beginning negotiation
0809 12:10:06.311819 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0809 12:10:06.312543 (+ 724us) server_negotiation.cc:407] Connection header received
0809 12:10:06.312686 (+ 143us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 12:10:06.312690 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 12:10:06.312753 (+ 63us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 12:10:06.312847 (+ 94us) server_negotiation.cc:226] Negotiated authn=SASL
0809 12:10:06.314545 (+ 1698us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 12:10:06.315724 (+ 1179us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 12:10:06.317061 (+ 1337us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 12:10:06.317724 (+ 663us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 12:10:06.317857 (+ 133us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 12:10:06.323529 (+ 5672us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 12:10:06.323549 (+ 20us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 12:10:06.323571 (+ 22us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0809 12:10:06.323646 (+ 75us) server_negotiation.cc:922] calling sasl_server_start()
0809 12:10:06.327953 (+ 4307us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 12:10:06.328588 (+ 635us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 12:10:06.328595 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 12:10:06.328597 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0809 12:10:06.328819 (+ 222us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 12:10:06.329174 (+ 355us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 12:10:06.329177 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 12:10:06.329179 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0809 12:10:06.329588 (+ 409us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 12:10:06.329932 (+ 344us) server_negotiation.cc:1035] Waiting for connection context
0809 12:10:06.330180 (+ 248us) server_negotiation.cc:299] Negotiation successful
0809 12:10:06.330381 (+ 201us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":13053,"thread_start_us":12802,"threads_started":1}
W20260809 12:10:06.331671 2998 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:42488
W20260809 12:10:06.332432 888 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskgH5Pjm/build/asan/bin/kudu pid 2856: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260809 12:10:06.332686 888 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskgH5Pjm/build/asan/bin/kudu with pid 2856
I20260809 12:10:06.364775 888 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskgH5Pjm/build/asan/bin/kudu pid 3007
Aug 09 12:10:06 dist-test-slave-2w3w krb5kdc[2731](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786277403, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.0.222.2@KRBTEST.COM
I20260809 12:10:06.381201 3491 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 12:10:06.365642 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48212 (local address 127.0.222.2:37581)
0809 12:10:06.366259 (+ 617us) server_negotiation.cc:206] Beginning negotiation
0809 12:10:06.366264 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0809 12:10:06.366280 (+ 16us) server_negotiation.cc:407] Connection header received
0809 12:10:06.366341 (+ 61us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 12:10:06.366345 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 12:10:06.366402 (+ 57us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 12:10:06.366525 (+ 123us) server_negotiation.cc:226] Negotiated authn=SASL
0809 12:10:06.367964 (+ 1439us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 12:10:06.368688 (+ 724us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 12:10:06.369924 (+ 1236us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 12:10:06.370341 (+ 417us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 12:10:06.370484 (+ 143us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 12:10:06.374812 (+ 4328us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 12:10:06.374829 (+ 17us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 12:10:06.374846 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0809 12:10:06.374896 (+ 50us) server_negotiation.cc:922] calling sasl_server_start()
0809 12:10:06.378277 (+ 3381us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 12:10:06.378993 (+ 716us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 12:10:06.379007 (+ 14us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 12:10:06.379011 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0809 12:10:06.379210 (+ 199us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 12:10:06.379619 (+ 409us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 12:10:06.379625 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 12:10:06.379628 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0809 12:10:06.380007 (+ 379us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 12:10:06.380321 (+ 314us) server_negotiation.cc:1035] Waiting for connection context
0809 12:10:06.380630 (+ 309us) server_negotiation.cc:299] Negotiation successful
0809 12:10:06.380867 (+ 237us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":503,"thread_start_us":343,"threads_started":1}
W20260809 12:10:06.381891 3147 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:48212
W20260809 12:10:06.382476 888 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskgH5Pjm/build/asan/bin/kudu pid 3007: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260809 12:10:06.382642 888 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskgH5Pjm/build/asan/bin/kudu with pid 3007
I20260809 12:10:06.410491 888 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskgH5Pjm/build/asan/bin/kudu pid 3158
Aug 09 12:10:06 dist-test-slave-2w3w krb5kdc[2731](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786277403, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.0.222.3@KRBTEST.COM
I20260809 12:10:06.427089 3492 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 12:10:06.411366 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42196 (local address 127.0.222.3:36309)
0809 12:10:06.411975 (+ 609us) server_negotiation.cc:206] Beginning negotiation
0809 12:10:06.411980 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0809 12:10:06.411998 (+ 18us) server_negotiation.cc:407] Connection header received
0809 12:10:06.412052 (+ 54us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 12:10:06.412056 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 12:10:06.412123 (+ 67us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 12:10:06.412218 (+ 95us) server_negotiation.cc:226] Negotiated authn=SASL
0809 12:10:06.413888 (+ 1670us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 12:10:06.414654 (+ 766us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 12:10:06.416222 (+ 1568us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 12:10:06.416675 (+ 453us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 12:10:06.416825 (+ 150us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 12:10:06.421379 (+ 4554us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 12:10:06.421394 (+ 15us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 12:10:06.421410 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0809 12:10:06.421450 (+ 40us) server_negotiation.cc:922] calling sasl_server_start()
0809 12:10:06.424217 (+ 2767us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 12:10:06.424898 (+ 681us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 12:10:06.424906 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 12:10:06.424908 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0809 12:10:06.425055 (+ 147us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 12:10:06.425434 (+ 379us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 12:10:06.425439 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 12:10:06.425442 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0809 12:10:06.425817 (+ 375us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 12:10:06.426114 (+ 297us) server_negotiation.cc:1035] Waiting for connection context
0809 12:10:06.426436 (+ 322us) server_negotiation.cc:299] Negotiation successful
0809 12:10:06.426664 (+ 228us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":506,"thread_start_us":323,"threads_started":1}
W20260809 12:10:06.427750 3295 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:42196
W20260809 12:10:06.428403 888 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskgH5Pjm/build/asan/bin/kudu pid 3158: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260809 12:10:06.428570 888 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskgH5Pjm/build/asan/bin/kudu with pid 3158
I20260809 12:10:06.454252 888 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskgH5Pjm/build/asan/bin/kudu pid 2745
I20260809 12:10:06.455586 2824 generic_service.cc:196] Checking for leaks (request via RPC)
I20260809 12:10:06.577203 888 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskgH5Pjm/build/asan/bin/kudu with pid 2745
2026-08-09T12:10:06Z chronyd exiting
[ OK ] SecurityITest.TxnSmokeWithDifferentUserTypes (9018 ms)
[ RUN ] SecurityITest.TestNoKerberosCredentials
Loading random data
Initializing database '/tmp/dist-test-taskgH5Pjm/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786277382073938-888-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 09 12:10:06 dist-test-slave-2w3w krb5kdc[3496](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 09 12:10:06 dist-test-slave-2w3w krb5kdc[3496](info): set up 2 sockets
Aug 09 12:10:06 dist-test-slave-2w3w krb5kdc[3496](info): commencing operation
krb5kdc: starting...
W20260809 12:10:07.720391 888 mini_kdc.cc:121] Time spent starting KDC: real 1.093s user 0.000s sys 0.023s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskgH5Pjm/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786277382073938-888-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskgH5Pjm/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786277382073938-888-0/krb5kdc/test-user.keytab.
Aug 09 12:10:07 dist-test-slave-2w3w krb5kdc[3496](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786277407, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-09T12:10:07Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-09T12:10:07Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.222.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.222.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.222.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskgH5Pjm/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786277382073938-888-0/krb5kdc/kudu_127.0.222.62.keytab.
Entry for principal kudu/127.0.222.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskgH5Pjm/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786277382073938-888-0/krb5kdc/kudu_127.0.222.62.keytab.
I20260809 12:10:07.934902 888 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskgH5Pjm/build/asan/bin/kudu
/tmp/dist-test-taskgH5Pjm/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskgH5Pjm/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786277382073938-888-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskgH5Pjm/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786277382073938-888-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskgH5Pjm/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786277382073938-888-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskgH5Pjm/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786277382073938-888-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.222.62:41249
--webserver_interface=127.0.222.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskgH5Pjm/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786277382073938-888-0/krb5kdc/kudu_127.0.222.62.keytab
--principal=kudu/127.0.222.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.222.20:39475
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.222.62:41249
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskgH5Pjm/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786277382073938-888-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskgH5Pjm/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786277382073938-888-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskgH5Pjm/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786277382073938-888-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260809 12:10:08.266942 3509 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260809 12:10:08.267356 3509 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260809 12:10:08.267522 3509 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260809 12:10:08.274688 3509 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260809 12:10:08.274849 3509 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260809 12:10:08.274947 3509 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260809 12:10:08.275084 3509 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260809 12:10:08.275174 3509 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260809 12:10:08.284848 3509 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.222.20:39475
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskgH5Pjm/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786277382073938-888-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskgH5Pjm/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786277382073938-888-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.222.62:41249
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskgH5Pjm/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786277382073938-888-0/krb5kdc/kudu_127.0.222.62.keytab
--principal=kudu/127.0.222.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.222.62:41249
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskgH5Pjm/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786277382073938-888-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.222.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskgH5Pjm/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786277382073938-888-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 09 Aug 2026 11:06:41 UTC on ccycloud.quasar-bwoogy.root.comops.site
build id 4575991
ASAN enabled
I20260809 12:10:08.286170 3509 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260809 12:10:08.287556 3509 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260809 12:10:08.299968 3515 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 12:10:08.300020 3518 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 12:10:08.300058 3516 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260809 12:10:08.300243 3509 server_base.cc:1034] running on GCE node
I20260809 12:10:08.301556 3509 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260809 12:10:08.303076 3509 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260809 12:10:08.304322 3509 hybrid_clock.cc:648] HybridClock initialized: now 1786277408304270 us; error 37 us; skew 500 ppm
I20260809 12:10:08.304703 3509 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 09 12:10:08 dist-test-slave-2w3w krb5kdc[3496](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786277408, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.222.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260809 12:10:08.307952 3509 init.cc:377] Logged in from keytab as kudu/127.0.222.62@KRBTEST.COM (short username kudu)
I20260809 12:10:08.309448 3509 webserver.cc:460] Webserver started at http://127.0.222.62:36901/ using document root <none> and password file <none>
I20260809 12:10:08.310148 3509 fs_manager.cc:359] Metadata directory not provided
I20260809 12:10:08.310329 3509 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260809 12:10:08.310647 3509 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260809 12:10:08.313684 3509 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskgH5Pjm/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786277382073938-888-0/minicluster-data/master-0/data/instance:
uuid: "f37e5549d9c04e46af938a5f9fe17689"
format_stamp: "Formatted at 2026-08-09 12:10:08 on dist-test-slave-2w3w"
server_key: "53693464a667b53ffe05d757eafeae7f"
server_key_iv: "e4af546871b2a45c1ee1fce1d40a095d"
server_key_version: "encryptionkey@0"
I20260809 12:10:08.314488 3509 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskgH5Pjm/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786277382073938-888-0/minicluster-data/master-0/wal/instance:
uuid: "f37e5549d9c04e46af938a5f9fe17689"
format_stamp: "Formatted at 2026-08-09 12:10:08 on dist-test-slave-2w3w"
server_key: "53693464a667b53ffe05d757eafeae7f"
server_key_iv: "e4af546871b2a45c1ee1fce1d40a095d"
server_key_version: "encryptionkey@0"
I20260809 12:10:08.319733 3509 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.007s sys 0.001s
I20260809 12:10:08.323328 3525 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260809 12:10:08.324746 3509 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.002s
I20260809 12:10:08.324965 3509 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskgH5Pjm/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786277382073938-888-0/minicluster-data/master-0/data,/tmp/dist-test-taskgH5Pjm/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786277382073938-888-0/minicluster-data/master-0/wal
uuid: "f37e5549d9c04e46af938a5f9fe17689"
format_stamp: "Formatted at 2026-08-09 12:10:08 on dist-test-slave-2w3w"
server_key: "53693464a667b53ffe05d757eafeae7f"
server_key_iv: "e4af546871b2a45c1ee1fce1d40a095d"
server_key_version: "encryptionkey@0"
I20260809 12:10:08.325173 3509 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskgH5Pjm/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786277382073938-888-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskgH5Pjm/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786277382073938-888-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskgH5Pjm/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786277382073938-888-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260809 12:10:08.345428 3509 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260809 12:10:08.349327 3509 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260809 12:10:08.349620 3509 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260809 12:10:08.368234 3509 rpc_server.cc:307] RPC server started. Bound to: 127.0.222.62:41249
I20260809 12:10:08.368276 3581 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.222.62:41249 every 8 connection(s)
I20260809 12:10:08.369617 3509 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskgH5Pjm/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786277382073938-888-0/minicluster-data/master-0/data/info.pb
I20260809 12:10:08.373421 3582 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260809 12:10:08.374183 888 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskgH5Pjm/build/asan/bin/kudu as pid 3509
I20260809 12:10:08.374432 888 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskgH5Pjm/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786277382073938-888-0/minicluster-data/master-0/wal/instance
I20260809 12:10:08.375303 888 external_mini_cluster.cc:1428] Setting key 79431e4e8c4d9f15d42ffd7dc0d48455
I20260809 12:10:08.389063 3582 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P f37e5549d9c04e46af938a5f9fe17689: Bootstrap starting.
Aug 09 12:10:08 dist-test-slave-2w3w krb5kdc[3496](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786277407, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.222.62@KRBTEST.COM
I20260809 12:10:08.392795 3582 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P f37e5549d9c04e46af938a5f9fe17689: Neither blocks nor log segments found. Creating new log.
I20260809 12:10:08.393919 3582 log.cc:824] T 00000000000000000000000000000000 P f37e5549d9c04e46af938a5f9fe17689: Log is configured to *not* fsync() on all Append() calls
I20260809 12:10:08.397421 3582 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P f37e5549d9c04e46af938a5f9fe17689: No bootstrap required, opened a new log
I20260809 12:10:08.400861 3585 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 12:10:08.377568 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38492 (local address 127.0.222.62:41249)
0809 12:10:08.379260 (+ 1692us) server_negotiation.cc:206] Beginning negotiation
0809 12:10:08.379275 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0809 12:10:08.379315 (+ 40us) server_negotiation.cc:407] Connection header received
0809 12:10:08.380931 (+ 1616us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 12:10:08.380956 (+ 25us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 12:10:08.381320 (+ 364us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 12:10:08.381643 (+ 323us) server_negotiation.cc:226] Negotiated authn=SASL
0809 12:10:08.383499 (+ 1856us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 12:10:08.384776 (+ 1277us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 12:10:08.386654 (+ 1878us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 12:10:08.387423 (+ 769us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 12:10:08.387552 (+ 129us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 12:10:08.392345 (+ 4793us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 12:10:08.392353 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 12:10:08.392367 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0809 12:10:08.392403 (+ 36us) server_negotiation.cc:922] calling sasl_server_start()
0809 12:10:08.396098 (+ 3695us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 12:10:08.396852 (+ 754us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 12:10:08.396861 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 12:10:08.396869 (+ 8us) server_negotiation.cc:967] calling sasl_server_step()
0809 12:10:08.397085 (+ 216us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 12:10:08.397471 (+ 386us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 12:10:08.397477 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 12:10:08.397489 (+ 12us) server_negotiation.cc:967] calling sasl_server_step()
0809 12:10:08.398211 (+ 722us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 12:10:08.398523 (+ 312us) server_negotiation.cc:1035] Waiting for connection context
0809 12:10:08.398913 (+ 390us) server_negotiation.cc:299] Negotiation successful
0809 12:10:08.399393 (+ 480us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":3,"server-negotiator.queue_time_us":1490,"thread_start_us":1228,"threads_started":1}
I20260809 12:10:08.406883 3582 raft_consensus.cc:348] T 00000000000000000000000000000000 P f37e5549d9c04e46af938a5f9fe17689 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f37e5549d9c04e46af938a5f9fe17689" member_type: VOTER last_known_addr { host: "127.0.222.62" port: 41249 } }
I20260809 12:10:08.407344 3582 raft_consensus.cc:374] T 00000000000000000000000000000000 P f37e5549d9c04e46af938a5f9fe17689 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260809 12:10:08.407434 3582 raft_consensus.cc:729] T 00000000000000000000000000000000 P f37e5549d9c04e46af938a5f9fe17689 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f37e5549d9c04e46af938a5f9fe17689, State: Initialized, Role: FOLLOWER
I20260809 12:10:08.408002 3582 consensus_queue.cc:260] T 00000000000000000000000000000000 P f37e5549d9c04e46af938a5f9fe17689 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f37e5549d9c04e46af938a5f9fe17689" member_type: VOTER last_known_addr { host: "127.0.222.62" port: 41249 } }
I20260809 12:10:08.408222 3582 raft_consensus.cc:388] T 00000000000000000000000000000000 P f37e5549d9c04e46af938a5f9fe17689 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260809 12:10:08.408370 3582 raft_consensus.cc:482] T 00000000000000000000000000000000 P f37e5549d9c04e46af938a5f9fe17689 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260809 12:10:08.408551 3582 raft_consensus.cc:3037] T 00000000000000000000000000000000 P f37e5549d9c04e46af938a5f9fe17689 [term 0 FOLLOWER]: Advancing to term 1
I20260809 12:10:08.411154 3582 raft_consensus.cc:504] T 00000000000000000000000000000000 P f37e5549d9c04e46af938a5f9fe17689 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f37e5549d9c04e46af938a5f9fe17689" member_type: VOTER last_known_addr { host: "127.0.222.62" port: 41249 } }
I20260809 12:10:08.411690 3582 leader_election.cc:302] T 00000000000000000000000000000000 P f37e5549d9c04e46af938a5f9fe17689 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: f37e5549d9c04e46af938a5f9fe17689; no voters:
I20260809 12:10:08.412281 3582 leader_election.cc:288] T 00000000000000000000000000000000 P f37e5549d9c04e46af938a5f9fe17689 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260809 12:10:08.412437 3587 raft_consensus.cc:2781] T 00000000000000000000000000000000 P f37e5549d9c04e46af938a5f9fe17689 [term 1 FOLLOWER]: Leader election won for term 1
I20260809 12:10:08.413136 3587 raft_consensus.cc:686] T 00000000000000000000000000000000 P f37e5549d9c04e46af938a5f9fe17689 [term 1 LEADER]: Becoming Leader. State: Replica: f37e5549d9c04e46af938a5f9fe17689, State: Running, Role: LEADER
I20260809 12:10:08.413714 3587 consensus_queue.cc:237] T 00000000000000000000000000000000 P f37e5549d9c04e46af938a5f9fe17689 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f37e5549d9c04e46af938a5f9fe17689" member_type: VOTER last_known_addr { host: "127.0.222.62" port: 41249 } }
I20260809 12:10:08.414532 3582 sys_catalog.cc:564] T 00000000000000000000000000000000 P f37e5549d9c04e46af938a5f9fe17689 [sys.catalog]: configured and running, proceeding with master startup.
I20260809 12:10:08.416749 3589 sys_catalog.cc:455] T 00000000000000000000000000000000 P f37e5549d9c04e46af938a5f9fe17689 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "f37e5549d9c04e46af938a5f9fe17689" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f37e5549d9c04e46af938a5f9fe17689" member_type: VOTER last_known_addr { host: "127.0.222.62" port: 41249 } } }
I20260809 12:10:08.417207 3589 sys_catalog.cc:458] T 00000000000000000000000000000000 P f37e5549d9c04e46af938a5f9fe17689 [sys.catalog]: This master's current role is: LEADER
I20260809 12:10:08.417119 3591 sys_catalog.cc:455] T 00000000000000000000000000000000 P f37e5549d9c04e46af938a5f9fe17689 [sys.catalog]: SysCatalogTable state changed. Reason: New leader f37e5549d9c04e46af938a5f9fe17689. Latest consensus state: current_term: 1 leader_uuid: "f37e5549d9c04e46af938a5f9fe17689" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f37e5549d9c04e46af938a5f9fe17689" member_type: VOTER last_known_addr { host: "127.0.222.62" port: 41249 } } }
I20260809 12:10:08.417516 3591 sys_catalog.cc:458] T 00000000000000000000000000000000 P f37e5549d9c04e46af938a5f9fe17689 [sys.catalog]: This master's current role is: LEADER
I20260809 12:10:08.418803 3597 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260809 12:10:08.425019 3597 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260809 12:10:08.434510 3597 catalog_manager.cc:1347] Generated new cluster ID: 76035549985f4195becac3a5b7fd7ca2
I20260809 12:10:08.434641 3597 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260809 12:10:08.449115 3597 catalog_manager.cc:1370] Generated new certificate authority record
I20260809 12:10:08.450657 3597 catalog_manager.cc:1504] Loading token signing keys...
I20260809 12:10:08.464524 3597 catalog_manager.cc:5868] T 00000000000000000000000000000000 P f37e5549d9c04e46af938a5f9fe17689: Generated new TSK 0
I20260809 12:10:08.465394 3597 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.0.222.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.222.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.222.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskgH5Pjm/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786277382073938-888-0/krb5kdc/kudu_127.0.222.1.keytab.
Entry for principal kudu/127.0.222.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskgH5Pjm/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786277382073938-888-0/krb5kdc/kudu_127.0.222.1.keytab.
I20260809 12:10:08.530297 888 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskgH5Pjm/build/asan/bin/kudu
/tmp/dist-test-taskgH5Pjm/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskgH5Pjm/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786277382073938-888-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskgH5Pjm/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786277382073938-888-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskgH5Pjm/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786277382073938-888-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskgH5Pjm/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786277382073938-888-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.222.1:0
--local_ip_for_outbound_sockets=127.0.222.1
--webserver_interface=127.0.222.1
--webserver_port=0
--tserver_master_addrs=127.0.222.62:41249
--keytab_file=/tmp/dist-test-taskgH5Pjm/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786277382073938-888-0/krb5kdc/kudu_127.0.222.1.keytab
--principal=kudu/127.0.222.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.222.20:39475
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskgH5Pjm/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786277382073938-888-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskgH5Pjm/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786277382073938-888-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskgH5Pjm/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786277382073938-888-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260809 12:10:08.865065 3610 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260809 12:10:08.865348 3610 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260809 12:10:08.865451 3610 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260809 12:10:08.872694 3610 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260809 12:10:08.872807 3610 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260809 12:10:08.872929 3610 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.222.1
I20260809 12:10:08.882544 3610 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.222.20:39475
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskgH5Pjm/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786277382073938-888-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskgH5Pjm/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786277382073938-888-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskgH5Pjm/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786277382073938-888-0/krb5kdc/kudu_127.0.222.1.keytab
--principal=kudu/127.0.222.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.222.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskgH5Pjm/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786277382073938-888-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.222.1
--webserver_port=0
--tserver_master_addrs=127.0.222.62:41249
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.222.1
--log_dir=/tmp/dist-test-taskgH5Pjm/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786277382073938-888-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 09 Aug 2026 11:06:41 UTC on ccycloud.quasar-bwoogy.root.comops.site
build id 4575991
ASAN enabled
I20260809 12:10:08.883734 3610 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260809 12:10:08.884967 3610 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260809 12:10:08.898355 3616 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 12:10:08.898355 3617 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 12:10:08.898396 3619 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260809 12:10:08.899710 3610 server_base.cc:1034] running on GCE node
I20260809 12:10:08.900394 3610 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260809 12:10:08.901356 3610 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260809 12:10:08.902559 3610 hybrid_clock.cc:648] HybridClock initialized: now 1786277408902513 us; error 36 us; skew 500 ppm
I20260809 12:10:08.902894 3610 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 09 12:10:08 dist-test-slave-2w3w krb5kdc[3496](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786277408, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.222.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260809 12:10:08.906162 3610 init.cc:377] Logged in from keytab as kudu/127.0.222.1@KRBTEST.COM (short username kudu)
I20260809 12:10:08.907732 3610 webserver.cc:460] Webserver started at http://127.0.222.1:38893/ using document root <none> and password file <none>
I20260809 12:10:08.908425 3610 fs_manager.cc:359] Metadata directory not provided
I20260809 12:10:08.908617 3610 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260809 12:10:08.909020 3610 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260809 12:10:08.912441 3610 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskgH5Pjm/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786277382073938-888-0/minicluster-data/ts-0/data/instance:
uuid: "b1e53687861c45b384467299f0f3d19e"
format_stamp: "Formatted at 2026-08-09 12:10:08 on dist-test-slave-2w3w"
server_key: "beede2247a1723719b7a141c2f7b0e5c"
server_key_iv: "37014b4445f0dce1db2d111088f27b81"
server_key_version: "encryptionkey@0"
I20260809 12:10:08.913236 3610 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskgH5Pjm/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786277382073938-888-0/minicluster-data/ts-0/wal/instance:
uuid: "b1e53687861c45b384467299f0f3d19e"
format_stamp: "Formatted at 2026-08-09 12:10:08 on dist-test-slave-2w3w"
server_key: "beede2247a1723719b7a141c2f7b0e5c"
server_key_iv: "37014b4445f0dce1db2d111088f27b81"
server_key_version: "encryptionkey@0"
I20260809 12:10:08.918617 3610 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.000s
I20260809 12:10:08.922406 3626 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260809 12:10:08.923753 3610 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.000s sys 0.003s
I20260809 12:10:08.923957 3610 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskgH5Pjm/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786277382073938-888-0/minicluster-data/ts-0/data,/tmp/dist-test-taskgH5Pjm/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786277382073938-888-0/minicluster-data/ts-0/wal
uuid: "b1e53687861c45b384467299f0f3d19e"
format_stamp: "Formatted at 2026-08-09 12:10:08 on dist-test-slave-2w3w"
server_key: "beede2247a1723719b7a141c2f7b0e5c"
server_key_iv: "37014b4445f0dce1db2d111088f27b81"
server_key_version: "encryptionkey@0"
I20260809 12:10:08.924165 3610 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskgH5Pjm/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786277382073938-888-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskgH5Pjm/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786277382073938-888-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskgH5Pjm/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786277382073938-888-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260809 12:10:08.948380 3610 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260809 12:10:08.952683 3610 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260809 12:10:08.953003 3610 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260809 12:10:08.954000 3610 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260809 12:10:08.955413 3610 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260809 12:10:08.955515 3610 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260809 12:10:08.955667 3610 ts_tablet_manager.cc:610] Registered 0 tablets
I20260809 12:10:08.955739 3610 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260809 12:10:08.991525 3610 rpc_server.cc:307] RPC server started. Bound to: 127.0.222.1:32941
I20260809 12:10:08.991613 3752 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.222.1:32941 every 8 connection(s)
I20260809 12:10:08.992869 3610 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskgH5Pjm/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786277382073938-888-0/minicluster-data/ts-0/data/info.pb
I20260809 12:10:09.003357 888 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskgH5Pjm/build/asan/bin/kudu as pid 3610
I20260809 12:10:09.003615 888 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskgH5Pjm/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786277382073938-888-0/minicluster-data/ts-0/wal/instance
I20260809 12:10:09.004427 888 external_mini_cluster.cc:1428] Setting key 94c7c80e503d095bb1503e3605512476
Aug 09 12:10:09 dist-test-slave-2w3w krb5kdc[3496](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786277408, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.222.1@KRBTEST.COM for kudu/127.0.222.62@KRBTEST.COM
I20260809 12:10:09.014856 3755 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 12:10:08.995502 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.222.1:50739 (local address 127.0.222.62:41249)
0809 12:10:08.995955 (+ 453us) server_negotiation.cc:206] Beginning negotiation
0809 12:10:08.995961 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0809 12:10:08.997018 (+ 1057us) server_negotiation.cc:407] Connection header received
0809 12:10:08.998783 (+ 1765us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 12:10:08.998787 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 12:10:08.998867 (+ 80us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 12:10:08.999010 (+ 143us) server_negotiation.cc:226] Negotiated authn=SASL
0809 12:10:09.000696 (+ 1686us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 12:10:09.001449 (+ 753us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 12:10:09.003330 (+ 1881us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 12:10:09.003854 (+ 524us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 12:10:09.003977 (+ 123us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 12:10:09.008446 (+ 4469us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 12:10:09.008451 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 12:10:09.008454 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0809 12:10:09.008486 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0809 12:10:09.011272 (+ 2786us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 12:10:09.012234 (+ 962us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 12:10:09.012238 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 12:10:09.012240 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0809 12:10:09.012357 (+ 117us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 12:10:09.012838 (+ 481us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 12:10:09.012841 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 12:10:09.012843 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0809 12:10:09.013104 (+ 261us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 12:10:09.013213 (+ 109us) server_negotiation.cc:1035] Waiting for connection context
0809 12:10:09.014135 (+ 922us) server_negotiation.cc:299] Negotiation successful
0809 12:10:09.014374 (+ 239us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":329,"thread_start_us":203,"threads_started":1}
I20260809 12:10:09.015960 3756 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 12:10:08.996120 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.222.62:41249 (local address 127.0.222.1:50739)
0809 12:10:08.996833 (+ 713us) negotiation.cc:107] Waiting for socket to connect
0809 12:10:08.996892 (+ 59us) client_negotiation.cc:174] Beginning negotiation
0809 12:10:08.998538 (+ 1646us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 12:10:08.999297 (+ 759us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 12:10:08.999308 (+ 11us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 12:10:08.999881 (+ 573us) client_negotiation.cc:189] Negotiated authn=SASL
0809 12:10:09.000497 (+ 616us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 12:10:09.000516 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 12:10:09.001645 (+ 1129us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 12:10:09.001649 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 12:10:09.003172 (+ 1523us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 12:10:09.003190 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 12:10:09.004109 (+ 919us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 12:10:09.004118 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 12:10:09.004449 (+ 331us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 12:10:09.005178 (+ 729us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0809 12:10:09.005201 (+ 23us) client_negotiation.cc:656] calling sasl_client_start()
0809 12:10:09.008215 (+ 3014us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0809 12:10:09.011417 (+ 3202us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0809 12:10:09.011429 (+ 12us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0809 12:10:09.011450 (+ 21us) client_negotiation.cc:760] calling sasl_client_step()
0809 12:10:09.012110 (+ 660us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0809 12:10:09.012462 (+ 352us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0809 12:10:09.012469 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0809 12:10:09.012475 (+ 6us) client_negotiation.cc:760] calling sasl_client_step()
0809 12:10:09.012725 (+ 250us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0809 12:10:09.013236 (+ 511us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0809 12:10:09.013247 (+ 11us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0809 12:10:09.013721 (+ 474us) client_negotiation.cc:769] Sending connection context
0809 12:10:09.014033 (+ 312us) client_negotiation.cc:240] Negotiation successful
0809 12:10:09.014617 (+ 584us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":487,"thread_start_us":285,"threads_started":1}
I20260809 12:10:09.017536 3753 heartbeater.cc:347] Connected to a master server at 127.0.222.62:41249
I20260809 12:10:09.017870 3753 heartbeater.cc:464] Registering TS with master...
I20260809 12:10:09.018787 3753 heartbeater.cc:511] Master 127.0.222.62:41249 requested a full tablet report, sending...
I20260809 12:10:09.021062 3544 ts_manager.cc:194] Registered new tserver with Master: b1e53687861c45b384467299f0f3d19e (127.0.222.1:32941)
I20260809 12:10:09.022490 3544 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.222.1@KRBTEST.COM'} at 127.0.222.1:50739
WARNING: no policy specified for kudu/127.0.222.2@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.222.2@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.222.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskgH5Pjm/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786277382073938-888-0/krb5kdc/kudu_127.0.222.2.keytab.
Entry for principal kudu/127.0.222.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskgH5Pjm/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786277382073938-888-0/krb5kdc/kudu_127.0.222.2.keytab.
I20260809 12:10:09.055925 888 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskgH5Pjm/build/asan/bin/kudu
/tmp/dist-test-taskgH5Pjm/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskgH5Pjm/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786277382073938-888-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskgH5Pjm/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786277382073938-888-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskgH5Pjm/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786277382073938-888-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskgH5Pjm/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786277382073938-888-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.222.2:0
--local_ip_for_outbound_sockets=127.0.222.2
--webserver_interface=127.0.222.2
--webserver_port=0
--tserver_master_addrs=127.0.222.62:41249
--keytab_file=/tmp/dist-test-taskgH5Pjm/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786277382073938-888-0/krb5kdc/kudu_127.0.222.2.keytab
--principal=kudu/127.0.222.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.222.20:39475
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskgH5Pjm/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786277382073938-888-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskgH5Pjm/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786277382073938-888-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskgH5Pjm/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786277382073938-888-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260809 12:10:09.406008 3759 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260809 12:10:09.406405 3759 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260809 12:10:09.406594 3759 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260809 12:10:09.414933 3759 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260809 12:10:09.415138 3759 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260809 12:10:09.415339 3759 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.222.2
I20260809 12:10:09.426645 3759 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.222.20:39475
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskgH5Pjm/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786277382073938-888-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskgH5Pjm/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786277382073938-888-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskgH5Pjm/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786277382073938-888-0/krb5kdc/kudu_127.0.222.2.keytab
--principal=kudu/127.0.222.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.222.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskgH5Pjm/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786277382073938-888-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.222.2
--webserver_port=0
--tserver_master_addrs=127.0.222.62:41249
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.222.2
--log_dir=/tmp/dist-test-taskgH5Pjm/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786277382073938-888-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 09 Aug 2026 11:06:41 UTC on ccycloud.quasar-bwoogy.root.comops.site
build id 4575991
ASAN enabled
I20260809 12:10:09.428118 3759 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260809 12:10:09.429772 3759 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260809 12:10:09.444535 3764 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 12:10:09.444572 3765 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 12:10:09.444598 3768 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260809 12:10:09.445971 3759 server_base.cc:1034] running on GCE node
I20260809 12:10:09.446694 3759 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260809 12:10:09.447878 3759 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260809 12:10:09.449157 3759 hybrid_clock.cc:648] HybridClock initialized: now 1786277409449115 us; error 61 us; skew 500 ppm
I20260809 12:10:09.449514 3759 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 09 12:10:09 dist-test-slave-2w3w krb5kdc[3496](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786277409, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.222.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260809 12:10:09.452883 3759 init.cc:377] Logged in from keytab as kudu/127.0.222.2@KRBTEST.COM (short username kudu)
I20260809 12:10:09.454545 3759 webserver.cc:460] Webserver started at http://127.0.222.2:35959/ using document root <none> and password file <none>
I20260809 12:10:09.455381 3759 fs_manager.cc:359] Metadata directory not provided
I20260809 12:10:09.455544 3759 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260809 12:10:09.455880 3759 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260809 12:10:09.458917 3759 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskgH5Pjm/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786277382073938-888-0/minicluster-data/ts-1/data/instance:
uuid: "82e6a16580fe478386e473b476e72fb6"
format_stamp: "Formatted at 2026-08-09 12:10:09 on dist-test-slave-2w3w"
server_key: "02d0453624fb004eb6f65b4b44e0132a"
server_key_iv: "65881e87ebe3dc0f45b1f8b31cb658da"
server_key_version: "encryptionkey@0"
I20260809 12:10:09.459820 3759 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskgH5Pjm/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786277382073938-888-0/minicluster-data/ts-1/wal/instance:
uuid: "82e6a16580fe478386e473b476e72fb6"
format_stamp: "Formatted at 2026-08-09 12:10:09 on dist-test-slave-2w3w"
server_key: "02d0453624fb004eb6f65b4b44e0132a"
server_key_iv: "65881e87ebe3dc0f45b1f8b31cb658da"
server_key_version: "encryptionkey@0"
I20260809 12:10:09.465437 3759 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.000s
I20260809 12:10:09.469132 3777 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260809 12:10:09.470569 3759 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260809 12:10:09.470753 3759 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskgH5Pjm/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786277382073938-888-0/minicluster-data/ts-1/data,/tmp/dist-test-taskgH5Pjm/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786277382073938-888-0/minicluster-data/ts-1/wal
uuid: "82e6a16580fe478386e473b476e72fb6"
format_stamp: "Formatted at 2026-08-09 12:10:09 on dist-test-slave-2w3w"
server_key: "02d0453624fb004eb6f65b4b44e0132a"
server_key_iv: "65881e87ebe3dc0f45b1f8b31cb658da"
server_key_version: "encryptionkey@0"
I20260809 12:10:09.470968 3759 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskgH5Pjm/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786277382073938-888-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskgH5Pjm/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786277382073938-888-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskgH5Pjm/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786277382073938-888-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260809 12:10:09.501842 3759 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260809 12:10:09.506054 3759 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260809 12:10:09.506363 3759 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260809 12:10:09.507432 3759 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260809 12:10:09.508873 3759 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260809 12:10:09.508989 3759 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260809 12:10:09.509143 3759 ts_tablet_manager.cc:610] Registered 0 tablets
I20260809 12:10:09.509224 3759 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260809 12:10:09.545717 3759 rpc_server.cc:307] RPC server started. Bound to: 127.0.222.2:39061
I20260809 12:10:09.545814 3897 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.222.2:39061 every 8 connection(s)
I20260809 12:10:09.547097 3759 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskgH5Pjm/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786277382073938-888-0/minicluster-data/ts-1/data/info.pb
I20260809 12:10:09.548251 888 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskgH5Pjm/build/asan/bin/kudu as pid 3759
I20260809 12:10:09.548473 888 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskgH5Pjm/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786277382073938-888-0/minicluster-data/ts-1/wal/instance
I20260809 12:10:09.549038 888 external_mini_cluster.cc:1428] Setting key 28fa6f1c0ed12a649cdc71616eca3900
Aug 09 12:10:09 dist-test-slave-2w3w krb5kdc[3496](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786277409, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.222.2@KRBTEST.COM for kudu/127.0.222.62@KRBTEST.COM
I20260809 12:10:09.569545 3901 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 12:10:09.551119 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.222.2:43205 (local address 127.0.222.62:41249)
0809 12:10:09.552103 (+ 984us) server_negotiation.cc:206] Beginning negotiation
0809 12:10:09.552109 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0809 12:10:09.552131 (+ 22us) server_negotiation.cc:407] Connection header received
0809 12:10:09.553532 (+ 1401us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 12:10:09.553538 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 12:10:09.553632 (+ 94us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 12:10:09.553750 (+ 118us) server_negotiation.cc:226] Negotiated authn=SASL
0809 12:10:09.555355 (+ 1605us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 12:10:09.556104 (+ 749us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 12:10:09.557660 (+ 1556us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 12:10:09.558103 (+ 443us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 12:10:09.558242 (+ 139us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 12:10:09.562277 (+ 4035us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 12:10:09.562287 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 12:10:09.562295 (+ 8us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0809 12:10:09.562342 (+ 47us) server_negotiation.cc:922] calling sasl_server_start()
0809 12:10:09.566052 (+ 3710us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 12:10:09.566827 (+ 775us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 12:10:09.566836 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 12:10:09.566844 (+ 8us) server_negotiation.cc:967] calling sasl_server_step()
0809 12:10:09.567082 (+ 238us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 12:10:09.567567 (+ 485us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 12:10:09.567576 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 12:10:09.567583 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0809 12:10:09.568042 (+ 459us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 12:10:09.568224 (+ 182us) server_negotiation.cc:1035] Waiting for connection context
0809 12:10:09.568931 (+ 707us) server_negotiation.cc:299] Negotiation successful
0809 12:10:09.569156 (+ 225us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":864,"thread_start_us":748,"threads_started":1}
I20260809 12:10:09.570494 3900 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 12:10:09.550549 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.222.62:41249 (local address 127.0.222.2:43205)
0809 12:10:09.551387 (+ 838us) negotiation.cc:107] Waiting for socket to connect
0809 12:10:09.551434 (+ 47us) client_negotiation.cc:174] Beginning negotiation
0809 12:10:09.553261 (+ 1827us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 12:10:09.553995 (+ 734us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 12:10:09.554003 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 12:10:09.554528 (+ 525us) client_negotiation.cc:189] Negotiated authn=SASL
0809 12:10:09.555187 (+ 659us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 12:10:09.555207 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 12:10:09.556261 (+ 1054us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 12:10:09.556266 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 12:10:09.557512 (+ 1246us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 12:10:09.557524 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 12:10:09.558333 (+ 809us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 12:10:09.558342 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 12:10:09.558582 (+ 240us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 12:10:09.559336 (+ 754us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0809 12:10:09.559360 (+ 24us) client_negotiation.cc:656] calling sasl_client_start()
0809 12:10:09.562019 (+ 2659us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0809 12:10:09.566218 (+ 4199us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0809 12:10:09.566224 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0809 12:10:09.566237 (+ 13us) client_negotiation.cc:760] calling sasl_client_step()
0809 12:10:09.566660 (+ 423us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0809 12:10:09.567217 (+ 557us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0809 12:10:09.567220 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0809 12:10:09.567222 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0809 12:10:09.567422 (+ 200us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0809 12:10:09.568188 (+ 766us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0809 12:10:09.568194 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0809 12:10:09.568526 (+ 332us) client_negotiation.cc:769] Sending connection context
0809 12:10:09.568789 (+ 263us) client_negotiation.cc:240] Negotiation successful
0809 12:10:09.569336 (+ 547us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":563,"thread_start_us":376,"threads_started":1}
I20260809 12:10:09.572018 3898 heartbeater.cc:347] Connected to a master server at 127.0.222.62:41249
I20260809 12:10:09.572342 3898 heartbeater.cc:464] Registering TS with master...
I20260809 12:10:09.573056 3898 heartbeater.cc:511] Master 127.0.222.62:41249 requested a full tablet report, sending...
WARNING: no policy specified for kudu/127.0.222.3@KRBTEST.COM; defaulting to no policy
I20260809 12:10:09.574626 3544 ts_manager.cc:194] Registered new tserver with Master: 82e6a16580fe478386e473b476e72fb6 (127.0.222.2:39061)
I20260809 12:10:09.575646 3544 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.222.2@KRBTEST.COM'} at 127.0.222.2:43205
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.222.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.222.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskgH5Pjm/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786277382073938-888-0/krb5kdc/kudu_127.0.222.3.keytab.
Entry for principal kudu/127.0.222.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskgH5Pjm/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786277382073938-888-0/krb5kdc/kudu_127.0.222.3.keytab.
I20260809 12:10:09.602308 888 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskgH5Pjm/build/asan/bin/kudu
/tmp/dist-test-taskgH5Pjm/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskgH5Pjm/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786277382073938-888-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskgH5Pjm/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786277382073938-888-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskgH5Pjm/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786277382073938-888-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskgH5Pjm/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786277382073938-888-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.222.3:0
--local_ip_for_outbound_sockets=127.0.222.3
--webserver_interface=127.0.222.3
--webserver_port=0
--tserver_master_addrs=127.0.222.62:41249
--keytab_file=/tmp/dist-test-taskgH5Pjm/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786277382073938-888-0/krb5kdc/kudu_127.0.222.3.keytab
--principal=kudu/127.0.222.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.222.20:39475
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskgH5Pjm/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786277382073938-888-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskgH5Pjm/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786277382073938-888-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskgH5Pjm/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786277382073938-888-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260809 12:10:09.935158 3904 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260809 12:10:09.935451 3904 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260809 12:10:09.935550 3904 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260809 12:10:09.943130 3904 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260809 12:10:09.943259 3904 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260809 12:10:09.943377 3904 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.222.3
I20260809 12:10:09.953202 3904 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.222.20:39475
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskgH5Pjm/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786277382073938-888-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskgH5Pjm/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786277382073938-888-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskgH5Pjm/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786277382073938-888-0/krb5kdc/kudu_127.0.222.3.keytab
--principal=kudu/127.0.222.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.222.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskgH5Pjm/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786277382073938-888-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.222.3
--webserver_port=0
--tserver_master_addrs=127.0.222.62:41249
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.222.3
--log_dir=/tmp/dist-test-taskgH5Pjm/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786277382073938-888-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 09 Aug 2026 11:06:41 UTC on ccycloud.quasar-bwoogy.root.comops.site
build id 4575991
ASAN enabled
I20260809 12:10:09.954479 3904 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260809 12:10:09.955935 3904 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260809 12:10:09.969449 3910 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 12:10:09.969409 3909 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 12:10:09.969489 3912 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260809 12:10:09.970381 3904 server_base.cc:1034] running on GCE node
I20260809 12:10:09.971089 3904 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260809 12:10:09.972213 3904 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260809 12:10:09.973424 3904 hybrid_clock.cc:648] HybridClock initialized: now 1786277409973347 us; error 69 us; skew 500 ppm
I20260809 12:10:09.973793 3904 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 09 12:10:09 dist-test-slave-2w3w krb5kdc[3496](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786277409, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.222.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260809 12:10:09.977244 3904 init.cc:377] Logged in from keytab as kudu/127.0.222.3@KRBTEST.COM (short username kudu)
I20260809 12:10:09.978755 3904 webserver.cc:460] Webserver started at http://127.0.222.3:45839/ using document root <none> and password file <none>
I20260809 12:10:09.979506 3904 fs_manager.cc:359] Metadata directory not provided
I20260809 12:10:09.979645 3904 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260809 12:10:09.979992 3904 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260809 12:10:09.983193 3904 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskgH5Pjm/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786277382073938-888-0/minicluster-data/ts-2/data/instance:
uuid: "4b6e2a559de94f9ca228e08d068c9eac"
format_stamp: "Formatted at 2026-08-09 12:10:09 on dist-test-slave-2w3w"
server_key: "309803daec97cde6d1d2ebd086ad1452"
server_key_iv: "397233c18ae48202a0977b2973e54597"
server_key_version: "encryptionkey@0"
I20260809 12:10:09.984016 3904 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskgH5Pjm/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786277382073938-888-0/minicluster-data/ts-2/wal/instance:
uuid: "4b6e2a559de94f9ca228e08d068c9eac"
format_stamp: "Formatted at 2026-08-09 12:10:09 on dist-test-slave-2w3w"
server_key: "309803daec97cde6d1d2ebd086ad1452"
server_key_iv: "397233c18ae48202a0977b2973e54597"
server_key_version: "encryptionkey@0"
I20260809 12:10:09.990476 3904 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.006s sys 0.000s
I20260809 12:10:09.995610 3920 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260809 12:10:09.997124 3904 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260809 12:10:09.997319 3904 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskgH5Pjm/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786277382073938-888-0/minicluster-data/ts-2/data,/tmp/dist-test-taskgH5Pjm/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786277382073938-888-0/minicluster-data/ts-2/wal
uuid: "4b6e2a559de94f9ca228e08d068c9eac"
format_stamp: "Formatted at 2026-08-09 12:10:09 on dist-test-slave-2w3w"
server_key: "309803daec97cde6d1d2ebd086ad1452"
server_key_iv: "397233c18ae48202a0977b2973e54597"
server_key_version: "encryptionkey@0"
I20260809 12:10:09.997537 3904 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskgH5Pjm/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786277382073938-888-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskgH5Pjm/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786277382073938-888-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskgH5Pjm/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786277382073938-888-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260809 12:10:10.019965 3904 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260809 12:10:10.024363 3904 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260809 12:10:10.024677 3904 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260809 12:10:10.025727 3904 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260809 12:10:10.026260 3753 heartbeater.cc:503] Master 127.0.222.62:41249 was elected leader, sending a full tablet report...
I20260809 12:10:10.027488 3904 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260809 12:10:10.027565 3904 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260809 12:10:10.027719 3904 ts_tablet_manager.cc:610] Registered 0 tablets
I20260809 12:10:10.027776 3904 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260809 12:10:10.061476 3904 rpc_server.cc:307] RPC server started. Bound to: 127.0.222.3:45749
I20260809 12:10:10.061573 4041 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.222.3:45749 every 8 connection(s)
I20260809 12:10:10.062860 3904 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskgH5Pjm/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786277382073938-888-0/minicluster-data/ts-2/data/info.pb
I20260809 12:10:10.065608 888 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskgH5Pjm/build/asan/bin/kudu as pid 3904
I20260809 12:10:10.065816 888 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskgH5Pjm/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786277382073938-888-0/minicluster-data/ts-2/wal/instance
I20260809 12:10:10.066350 888 external_mini_cluster.cc:1428] Setting key 1ab229f0c6bde7ccfbf8c1faac873e78
Aug 09 12:10:10 dist-test-slave-2w3w krb5kdc[3496](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786277409, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.222.3@KRBTEST.COM for kudu/127.0.222.62@KRBTEST.COM
I20260809 12:10:10.083352 3901 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 12:10:10.065429 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.222.3:55905 (local address 127.0.222.62:41249)
0809 12:10:10.065614 (+ 185us) server_negotiation.cc:206] Beginning negotiation
0809 12:10:10.065619 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0809 12:10:10.067138 (+ 1519us) server_negotiation.cc:407] Connection header received
0809 12:10:10.069576 (+ 2438us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 12:10:10.069580 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 12:10:10.069660 (+ 80us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 12:10:10.069811 (+ 151us) server_negotiation.cc:226] Negotiated authn=SASL
0809 12:10:10.071311 (+ 1500us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 12:10:10.072185 (+ 874us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 12:10:10.073679 (+ 1494us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 12:10:10.074073 (+ 394us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 12:10:10.074169 (+ 96us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 12:10:10.077601 (+ 3432us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 12:10:10.077605 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 12:10:10.077608 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0809 12:10:10.077640 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0809 12:10:10.080166 (+ 2526us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 12:10:10.081031 (+ 865us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 12:10:10.081035 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 12:10:10.081037 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0809 12:10:10.081161 (+ 124us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 12:10:10.081623 (+ 462us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 12:10:10.081626 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 12:10:10.081628 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0809 12:10:10.081994 (+ 366us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 12:10:10.082107 (+ 113us) server_negotiation.cc:1035] Waiting for connection context
0809 12:10:10.082871 (+ 764us) server_negotiation.cc:299] Negotiation successful
0809 12:10:10.083108 (+ 237us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":65}
I20260809 12:10:10.084306 4044 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 12:10:10.066077 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.222.62:41249 (local address 127.0.222.3:55905)
0809 12:10:10.066886 (+ 809us) negotiation.cc:107] Waiting for socket to connect
0809 12:10:10.066952 (+ 66us) client_negotiation.cc:174] Beginning negotiation
0809 12:10:10.069299 (+ 2347us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 12:10:10.070001 (+ 702us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 12:10:10.070009 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 12:10:10.070515 (+ 506us) client_negotiation.cc:189] Negotiated authn=SASL
0809 12:10:10.071142 (+ 627us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 12:10:10.071162 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 12:10:10.072317 (+ 1155us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 12:10:10.072320 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 12:10:10.073539 (+ 1219us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 12:10:10.073550 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 12:10:10.074262 (+ 712us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 12:10:10.074267 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 12:10:10.074451 (+ 184us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 12:10:10.075204 (+ 753us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0809 12:10:10.075234 (+ 30us) client_negotiation.cc:656] calling sasl_client_start()
0809 12:10:10.077398 (+ 2164us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0809 12:10:10.080329 (+ 2931us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0809 12:10:10.080336 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0809 12:10:10.080347 (+ 11us) client_negotiation.cc:760] calling sasl_client_step()
0809 12:10:10.080901 (+ 554us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0809 12:10:10.081297 (+ 396us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0809 12:10:10.081301 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0809 12:10:10.081305 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0809 12:10:10.081524 (+ 219us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0809 12:10:10.082126 (+ 602us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0809 12:10:10.082132 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0809 12:10:10.082539 (+ 407us) client_negotiation.cc:769] Sending connection context
0809 12:10:10.082760 (+ 221us) client_negotiation.cc:240] Negotiation successful
0809 12:10:10.083201 (+ 441us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":571,"thread_start_us":368,"threads_started":1}
I20260809 12:10:10.085623 4042 heartbeater.cc:347] Connected to a master server at 127.0.222.62:41249
I20260809 12:10:10.085947 4042 heartbeater.cc:464] Registering TS with master...
I20260809 12:10:10.086637 4042 heartbeater.cc:511] Master 127.0.222.62:41249 requested a full tablet report, sending...
I20260809 12:10:10.088029 3544 ts_manager.cc:194] Registered new tserver with Master: 4b6e2a559de94f9ca228e08d068c9eac (127.0.222.3:45749)
I20260809 12:10:10.088793 3544 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.222.3@KRBTEST.COM'} at 127.0.222.3:55905
I20260809 12:10:10.094292 888 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
W20260809 12:10:10.134732 3901 negotiation.cc:337] Failed RPC negotiation. Trace:
0809 12:10:10.132063 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38502 (local address 127.0.222.62:41249)
0809 12:10:10.132332 (+ 269us) server_negotiation.cc:206] Beginning negotiation
0809 12:10:10.132342 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0809 12:10:10.132719 (+ 377us) server_negotiation.cc:407] Connection header received
0809 12:10:10.132871 (+ 152us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 12:10:10.132875 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 12:10:10.132957 (+ 82us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 12:10:10.133061 (+ 104us) server_negotiation.cc:226] Negotiated authn=SASL
0809 12:10:10.134415 (+ 1354us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:38502: BlockingRecv error: recv got EOF from 127.0.0.1:38502 (error 108)
Metrics: {"server-negotiator.queue_time_us":72}
I20260809 12:10:10.138499 888 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskgH5Pjm/build/asan/bin/kudu pid 3610
W20260809 12:10:10.141288 888 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskgH5Pjm/build/asan/bin/kudu pid 3610: Not authorized: Client connection negotiation failed: client connection to 127.0.222.1:32941: server requires authentication, but client does not have Kerberos credentials available
I20260809 12:10:10.141489 888 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskgH5Pjm/build/asan/bin/kudu with pid 3610
I20260809 12:10:10.160848 888 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskgH5Pjm/build/asan/bin/kudu pid 3759
W20260809 12:10:10.163645 888 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskgH5Pjm/build/asan/bin/kudu pid 3759: Not authorized: Client connection negotiation failed: client connection to 127.0.222.2:39061: server requires authentication, but client does not have Kerberos credentials available
I20260809 12:10:10.163828 888 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskgH5Pjm/build/asan/bin/kudu with pid 3759
I20260809 12:10:10.182322 888 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskgH5Pjm/build/asan/bin/kudu pid 3904
W20260809 12:10:10.185424 888 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskgH5Pjm/build/asan/bin/kudu pid 3904: Not authorized: Client connection negotiation failed: client connection to 127.0.222.3:45749: server requires authentication, but client does not have Kerberos credentials available
I20260809 12:10:10.185557 888 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskgH5Pjm/build/asan/bin/kudu with pid 3904
I20260809 12:10:10.206422 888 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskgH5Pjm/build/asan/bin/kudu pid 3509
I20260809 12:10:10.207901 3577 generic_service.cc:196] Checking for leaks (request via RPC)
I20260809 12:10:10.313177 888 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskgH5Pjm/build/asan/bin/kudu with pid 3509
2026-08-09T12:10:10Z chronyd exiting
[ OK ] SecurityITest.TestNoKerberosCredentials (3733 ms)
[ RUN ] SecurityITest.SaslPlainFallback
Loading random data
Initializing database '/tmp/dist-test-taskgH5Pjm/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1786277382073938-888-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 09 12:10:10 dist-test-slave-2w3w krb5kdc[4061](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 09 12:10:10 dist-test-slave-2w3w krb5kdc[4061](info): set up 2 sockets
Aug 09 12:10:10 dist-test-slave-2w3w krb5kdc[4061](info): commencing operation
krb5kdc: starting...
W20260809 12:10:11.460245 888 mini_kdc.cc:121] Time spent starting KDC: real 1.100s user 0.002s sys 0.024s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskgH5Pjm/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1786277382073938-888-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskgH5Pjm/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1786277382073938-888-0/krb5kdc/test-user.keytab.
Aug 09 12:10:11 dist-test-slave-2w3w krb5kdc[4061](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786277411, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-09T12:10:11Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-09T12:10:11Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.222.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.222.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.222.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskgH5Pjm/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1786277382073938-888-0/krb5kdc/kudu_127.0.222.62.keytab.
Entry for principal kudu/127.0.222.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskgH5Pjm/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1786277382073938-888-0/krb5kdc/kudu_127.0.222.62.keytab.
I20260809 12:10:11.684962 888 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskgH5Pjm/build/asan/bin/kudu
/tmp/dist-test-taskgH5Pjm/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskgH5Pjm/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1786277382073938-888-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskgH5Pjm/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1786277382073938-888-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskgH5Pjm/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1786277382073938-888-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskgH5Pjm/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1786277382073938-888-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.222.62:43919
--webserver_interface=127.0.222.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskgH5Pjm/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1786277382073938-888-0/krb5kdc/kudu_127.0.222.62.keytab
--principal=kudu/127.0.222.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.222.20:43747
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.222.62:43919
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--rpc-authentication=optional
--user-acl=* with env {KRB5CCNAME=/tmp/dist-test-taskgH5Pjm/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1786277382073938-888-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskgH5Pjm/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1786277382073938-888-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskgH5Pjm/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1786277382073938-888-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260809 12:10:12.026827 4076 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260809 12:10:12.027130 4076 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260809 12:10:12.027333 4076 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260809 12:10:12.034394 4076 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260809 12:10:12.034494 4076 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260809 12:10:12.034585 4076 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260809 12:10:12.034688 4076 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260809 12:10:12.034807 4076 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260809 12:10:12.044703 4076 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.222.20:43747
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskgH5Pjm/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1786277382073938-888-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskgH5Pjm/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1786277382073938-888-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.222.62:43919
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskgH5Pjm/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1786277382073938-888-0/krb5kdc/kudu_127.0.222.62.keytab
--principal=kudu/127.0.222.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.222.62:43919
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskgH5Pjm/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1786277382073938-888-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.222.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskgH5Pjm/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1786277382073938-888-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 09 Aug 2026 11:06:41 UTC on ccycloud.quasar-bwoogy.root.comops.site
build id 4575991
ASAN enabled
I20260809 12:10:12.045969 4076 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260809 12:10:12.047351 4076 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260809 12:10:12.059490 4081 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 12:10:12.059573 4082 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 12:10:12.059495 4084 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260809 12:10:12.060341 4076 server_base.cc:1034] running on GCE node
I20260809 12:10:12.061144 4076 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260809 12:10:12.062543 4076 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260809 12:10:12.063762 4076 hybrid_clock.cc:648] HybridClock initialized: now 1786277412063711 us; error 65 us; skew 500 ppm
I20260809 12:10:12.064128 4076 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 09 12:10:12 dist-test-slave-2w3w krb5kdc[4061](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786277412, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.222.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260809 12:10:12.067384 4076 init.cc:377] Logged in from keytab as kudu/127.0.222.62@KRBTEST.COM (short username kudu)
I20260809 12:10:12.068838 4076 webserver.cc:460] Webserver started at http://127.0.222.62:38473/ using document root <none> and password file <none>
I20260809 12:10:12.069532 4076 fs_manager.cc:359] Metadata directory not provided
I20260809 12:10:12.069685 4076 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260809 12:10:12.069999 4076 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260809 12:10:12.073065 4076 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskgH5Pjm/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1786277382073938-888-0/minicluster-data/master-0/data/instance:
uuid: "199ad630cddf42a7946fa94bc03a71ae"
format_stamp: "Formatted at 2026-08-09 12:10:12 on dist-test-slave-2w3w"
server_key: "8a4978931281774e149ffa9ace94fd87"
server_key_iv: "e25ceded6ad1b23295de3ea27f0358ff"
server_key_version: "encryptionkey@0"
I20260809 12:10:12.073879 4076 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskgH5Pjm/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1786277382073938-888-0/minicluster-data/master-0/wal/instance:
uuid: "199ad630cddf42a7946fa94bc03a71ae"
format_stamp: "Formatted at 2026-08-09 12:10:12 on dist-test-slave-2w3w"
server_key: "8a4978931281774e149ffa9ace94fd87"
server_key_iv: "e25ceded6ad1b23295de3ea27f0358ff"
server_key_version: "encryptionkey@0"
I20260809 12:10:12.079358 4076 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.001s
I20260809 12:10:12.082933 4091 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260809 12:10:12.084349 4076 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260809 12:10:12.084532 4076 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskgH5Pjm/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1786277382073938-888-0/minicluster-data/master-0/data,/tmp/dist-test-taskgH5Pjm/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1786277382073938-888-0/minicluster-data/master-0/wal
uuid: "199ad630cddf42a7946fa94bc03a71ae"
format_stamp: "Formatted at 2026-08-09 12:10:12 on dist-test-slave-2w3w"
server_key: "8a4978931281774e149ffa9ace94fd87"
server_key_iv: "e25ceded6ad1b23295de3ea27f0358ff"
server_key_version: "encryptionkey@0"
I20260809 12:10:12.084735 4076 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskgH5Pjm/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1786277382073938-888-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskgH5Pjm/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1786277382073938-888-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskgH5Pjm/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1786277382073938-888-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260809 12:10:12.097854 4076 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260809 12:10:12.101992 4076 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260809 12:10:12.102291 4076 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260809 12:10:12.121661 4076 rpc_server.cc:307] RPC server started. Bound to: 127.0.222.62:43919
I20260809 12:10:12.121706 4149 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.222.62:43919 every 8 connection(s)
I20260809 12:10:12.123109 4076 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskgH5Pjm/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1786277382073938-888-0/minicluster-data/master-0/data/info.pb
I20260809 12:10:12.126896 4150 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260809 12:10:12.127398 888 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskgH5Pjm/build/asan/bin/kudu as pid 4076
I20260809 12:10:12.127611 888 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskgH5Pjm/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1786277382073938-888-0/minicluster-data/master-0/wal/instance
I20260809 12:10:12.128312 888 external_mini_cluster.cc:1428] Setting key a06352b938ab5d643eb5d0b0e4bed7ad
I20260809 12:10:12.140861 4150 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 199ad630cddf42a7946fa94bc03a71ae: Bootstrap starting.
Aug 09 12:10:12 dist-test-slave-2w3w krb5kdc[4061](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786277411, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.222.62@KRBTEST.COM
I20260809 12:10:12.144594 4150 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 199ad630cddf42a7946fa94bc03a71ae: Neither blocks nor log segments found. Creating new log.
I20260809 12:10:12.145680 4150 log.cc:824] T 00000000000000000000000000000000 P 199ad630cddf42a7946fa94bc03a71ae: Log is configured to *not* fsync() on all Append() calls
I20260809 12:10:12.148554 4150 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 199ad630cddf42a7946fa94bc03a71ae: No bootstrap required, opened a new log
I20260809 12:10:12.151757 4153 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 12:10:12.130137 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43028 (local address 127.0.222.62:43919)
0809 12:10:12.130921 (+ 784us) server_negotiation.cc:206] Beginning negotiation
0809 12:10:12.130932 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0809 12:10:12.131042 (+ 110us) server_negotiation.cc:407] Connection header received
0809 12:10:12.132954 (+ 1912us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 12:10:12.132977 (+ 23us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 12:10:12.133385 (+ 408us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 12:10:12.133710 (+ 325us) server_negotiation.cc:226] Negotiated authn=SASL
0809 12:10:12.135549 (+ 1839us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 12:10:12.136807 (+ 1258us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 12:10:12.138617 (+ 1810us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 12:10:12.139344 (+ 727us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 12:10:12.139474 (+ 130us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 12:10:12.144192 (+ 4718us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 12:10:12.144201 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 12:10:12.144214 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0809 12:10:12.144251 (+ 37us) server_negotiation.cc:922] calling sasl_server_start()
0809 12:10:12.147478 (+ 3227us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 12:10:12.148206 (+ 728us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 12:10:12.148213 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 12:10:12.148220 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0809 12:10:12.148406 (+ 186us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 12:10:12.148828 (+ 422us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 12:10:12.148836 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 12:10:12.148840 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0809 12:10:12.149394 (+ 554us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 12:10:12.149713 (+ 319us) server_negotiation.cc:1035] Waiting for connection context
0809 12:10:12.150218 (+ 505us) server_negotiation.cc:299] Negotiation successful
0809 12:10:12.150661 (+ 443us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":568,"thread_start_us":363,"threads_started":1}
I20260809 12:10:12.155781 888 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260809 12:10:12.157352 4150 raft_consensus.cc:348] T 00000000000000000000000000000000 P 199ad630cddf42a7946fa94bc03a71ae [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "199ad630cddf42a7946fa94bc03a71ae" member_type: VOTER last_known_addr { host: "127.0.222.62" port: 43919 } }
I20260809 12:10:12.157735 4150 raft_consensus.cc:374] T 00000000000000000000000000000000 P 199ad630cddf42a7946fa94bc03a71ae [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260809 12:10:12.157856 4150 raft_consensus.cc:729] T 00000000000000000000000000000000 P 199ad630cddf42a7946fa94bc03a71ae [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 199ad630cddf42a7946fa94bc03a71ae, State: Initialized, Role: FOLLOWER
I20260809 12:10:12.158409 4150 consensus_queue.cc:260] T 00000000000000000000000000000000 P 199ad630cddf42a7946fa94bc03a71ae [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "199ad630cddf42a7946fa94bc03a71ae" member_type: VOTER last_known_addr { host: "127.0.222.62" port: 43919 } }
I20260809 12:10:12.158656 4150 raft_consensus.cc:388] T 00000000000000000000000000000000 P 199ad630cddf42a7946fa94bc03a71ae [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260809 12:10:12.158811 4150 raft_consensus.cc:482] T 00000000000000000000000000000000 P 199ad630cddf42a7946fa94bc03a71ae [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260809 12:10:12.159032 4150 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 199ad630cddf42a7946fa94bc03a71ae [term 0 FOLLOWER]: Advancing to term 1
I20260809 12:10:12.162194 4150 raft_consensus.cc:504] T 00000000000000000000000000000000 P 199ad630cddf42a7946fa94bc03a71ae [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "199ad630cddf42a7946fa94bc03a71ae" member_type: VOTER last_known_addr { host: "127.0.222.62" port: 43919 } }
I20260809 12:10:12.162762 4150 leader_election.cc:302] T 00000000000000000000000000000000 P 199ad630cddf42a7946fa94bc03a71ae [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 199ad630cddf42a7946fa94bc03a71ae; no voters:
I20260809 12:10:12.163452 4150 leader_election.cc:288] T 00000000000000000000000000000000 P 199ad630cddf42a7946fa94bc03a71ae [CANDIDATE]: Term 1 election: Requested vote from peers
I20260809 12:10:12.163587 4155 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 199ad630cddf42a7946fa94bc03a71ae [term 1 FOLLOWER]: Leader election won for term 1
I20260809 12:10:12.166005 4155 raft_consensus.cc:686] T 00000000000000000000000000000000 P 199ad630cddf42a7946fa94bc03a71ae [term 1 LEADER]: Becoming Leader. State: Replica: 199ad630cddf42a7946fa94bc03a71ae, State: Running, Role: LEADER
I20260809 12:10:12.166721 4155 consensus_queue.cc:237] T 00000000000000000000000000000000 P 199ad630cddf42a7946fa94bc03a71ae [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "199ad630cddf42a7946fa94bc03a71ae" member_type: VOTER last_known_addr { host: "127.0.222.62" port: 43919 } }
I20260809 12:10:12.167119 4150 sys_catalog.cc:564] T 00000000000000000000000000000000 P 199ad630cddf42a7946fa94bc03a71ae [sys.catalog]: configured and running, proceeding with master startup.
I20260809 12:10:12.173187 4157 sys_catalog.cc:455] T 00000000000000000000000000000000 P 199ad630cddf42a7946fa94bc03a71ae [sys.catalog]: SysCatalogTable state changed. Reason: New leader 199ad630cddf42a7946fa94bc03a71ae. Latest consensus state: current_term: 1 leader_uuid: "199ad630cddf42a7946fa94bc03a71ae" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "199ad630cddf42a7946fa94bc03a71ae" member_type: VOTER last_known_addr { host: "127.0.222.62" port: 43919 } } }
I20260809 12:10:12.173605 4157 sys_catalog.cc:458] T 00000000000000000000000000000000 P 199ad630cddf42a7946fa94bc03a71ae [sys.catalog]: This master's current role is: LEADER
I20260809 12:10:12.174014 4156 sys_catalog.cc:455] T 00000000000000000000000000000000 P 199ad630cddf42a7946fa94bc03a71ae [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "199ad630cddf42a7946fa94bc03a71ae" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "199ad630cddf42a7946fa94bc03a71ae" member_type: VOTER last_known_addr { host: "127.0.222.62" port: 43919 } } }
I20260809 12:10:12.174286 4156 sys_catalog.cc:458] T 00000000000000000000000000000000 P 199ad630cddf42a7946fa94bc03a71ae [sys.catalog]: This master's current role is: LEADER
I20260809 12:10:12.179121 4166 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260809 12:10:12.185034 4166 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260809 12:10:12.195566 4166 catalog_manager.cc:1347] Generated new cluster ID: dffd796f46ed4da180c8ae9b5d95130d
I20260809 12:10:12.195755 4166 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260809 12:10:12.210592 4166 catalog_manager.cc:1370] Generated new certificate authority record
I20260809 12:10:12.212430 4166 catalog_manager.cc:1504] Loading token signing keys...
I20260809 12:10:12.214283 4153 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 12:10:12.205299 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43040 (local address 127.0.222.62:43919)
0809 12:10:12.205518 (+ 219us) server_negotiation.cc:206] Beginning negotiation
0809 12:10:12.205527 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0809 12:10:12.206090 (+ 563us) server_negotiation.cc:407] Connection header received
0809 12:10:12.206384 (+ 294us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 12:10:12.206391 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 12:10:12.206482 (+ 91us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 12:10:12.206567 (+ 85us) server_negotiation.cc:226] Negotiated authn=SASL
0809 12:10:12.207977 (+ 1410us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 12:10:12.208988 (+ 1011us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 12:10:12.210584 (+ 1596us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 12:10:12.211210 (+ 626us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 12:10:12.211358 (+ 148us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 12:10:12.212609 (+ 1251us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 12:10:12.212614 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 12:10:12.212618 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0809 12:10:12.213401 (+ 783us) server_negotiation.cc:922] calling sasl_server_start()
0809 12:10:12.213469 (+ 68us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0809 12:10:12.213499 (+ 30us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 12:10:12.213626 (+ 127us) server_negotiation.cc:1035] Waiting for connection context
0809 12:10:12.213836 (+ 210us) server_negotiation.cc:299] Negotiation successful
0809 12:10:12.213965 (+ 129us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":1,"server-negotiator.queue_time_us":71}
I20260809 12:10:12.221223 4166 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 199ad630cddf42a7946fa94bc03a71ae: Generated new TSK 0
I20260809 12:10:12.222047 4166 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260809 12:10:12.234932 888 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskgH5Pjm/build/asan/bin/kudu pid 4076
I20260809 12:10:12.236116 4145 generic_service.cc:196] Checking for leaks (request via RPC)
I20260809 12:10:12.327925 888 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskgH5Pjm/build/asan/bin/kudu with pid 4076
2026-08-09T12:10:12Z chronyd exiting
[ OK ] SecurityITest.SaslPlainFallback (2009 ms)
[ RUN ] SecurityITest.TestUnauthorizedClientKerberosCredentials
Loading random data
Initializing database '/tmp/dist-test-taskgH5Pjm/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786277382073938-888-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 09 12:10:12 dist-test-slave-2w3w krb5kdc[4187](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 09 12:10:12 dist-test-slave-2w3w krb5kdc[4187](info): set up 2 sockets
Aug 09 12:10:12 dist-test-slave-2w3w krb5kdc[4187](info): commencing operation
krb5kdc: starting...
W20260809 12:10:13.483465 888 mini_kdc.cc:121] Time spent starting KDC: real 1.114s user 0.000s sys 0.026s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskgH5Pjm/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786277382073938-888-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskgH5Pjm/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786277382073938-888-0/krb5kdc/test-user.keytab.
Aug 09 12:10:13 dist-test-slave-2w3w krb5kdc[4187](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786277413, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-09T12:10:13Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-09T12:10:13Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.222.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.222.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.222.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskgH5Pjm/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786277382073938-888-0/krb5kdc/kudu_127.0.222.62.keytab.
Entry for principal kudu/127.0.222.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskgH5Pjm/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786277382073938-888-0/krb5kdc/kudu_127.0.222.62.keytab.
I20260809 12:10:13.712962 888 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskgH5Pjm/build/asan/bin/kudu
/tmp/dist-test-taskgH5Pjm/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskgH5Pjm/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786277382073938-888-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskgH5Pjm/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786277382073938-888-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskgH5Pjm/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786277382073938-888-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskgH5Pjm/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786277382073938-888-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.222.62:37839
--webserver_interface=127.0.222.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskgH5Pjm/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786277382073938-888-0/krb5kdc/kudu_127.0.222.62.keytab
--principal=kudu/127.0.222.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.222.20:45373
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.222.62:37839
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskgH5Pjm/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786277382073938-888-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskgH5Pjm/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786277382073938-888-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskgH5Pjm/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786277382073938-888-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260809 12:10:14.045833 4201 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260809 12:10:14.046186 4201 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260809 12:10:14.046350 4201 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260809 12:10:14.054188 4201 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260809 12:10:14.054344 4201 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260809 12:10:14.054452 4201 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260809 12:10:14.054559 4201 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260809 12:10:14.054658 4201 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260809 12:10:14.064677 4201 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.222.20:45373
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskgH5Pjm/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786277382073938-888-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskgH5Pjm/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786277382073938-888-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.222.62:37839
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskgH5Pjm/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786277382073938-888-0/krb5kdc/kudu_127.0.222.62.keytab
--principal=kudu/127.0.222.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.222.62:37839
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskgH5Pjm/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786277382073938-888-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.222.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskgH5Pjm/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786277382073938-888-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 09 Aug 2026 11:06:41 UTC on ccycloud.quasar-bwoogy.root.comops.site
build id 4575991
ASAN enabled
I20260809 12:10:14.066079 4201 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260809 12:10:14.067482 4201 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260809 12:10:14.079396 4207 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 12:10:14.079470 4208 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 12:10:14.079444 4211 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260809 12:10:14.080329 4201 server_base.cc:1034] running on GCE node
I20260809 12:10:14.081061 4201 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260809 12:10:14.082437 4201 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260809 12:10:14.083667 4201 hybrid_clock.cc:648] HybridClock initialized: now 1786277414083621 us; error 56 us; skew 500 ppm
I20260809 12:10:14.084017 4201 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 09 12:10:14 dist-test-slave-2w3w krb5kdc[4187](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786277414, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.222.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260809 12:10:14.087354 4201 init.cc:377] Logged in from keytab as kudu/127.0.222.62@KRBTEST.COM (short username kudu)
I20260809 12:10:14.088997 4201 webserver.cc:460] Webserver started at http://127.0.222.62:41003/ using document root <none> and password file <none>
I20260809 12:10:14.089686 4201 fs_manager.cc:359] Metadata directory not provided
I20260809 12:10:14.089838 4201 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260809 12:10:14.090152 4201 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260809 12:10:14.093348 4201 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskgH5Pjm/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786277382073938-888-0/minicluster-data/master-0/data/instance:
uuid: "4eee0b48fca245919dd1f63f2b19466e"
format_stamp: "Formatted at 2026-08-09 12:10:14 on dist-test-slave-2w3w"
server_key: "b801b61f3c93b8e19bd3ccb61331c792"
server_key_iv: "2ad2e7ec6c68695e27115d0bb4435b0e"
server_key_version: "encryptionkey@0"
I20260809 12:10:14.094122 4201 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskgH5Pjm/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786277382073938-888-0/minicluster-data/master-0/wal/instance:
uuid: "4eee0b48fca245919dd1f63f2b19466e"
format_stamp: "Formatted at 2026-08-09 12:10:14 on dist-test-slave-2w3w"
server_key: "b801b61f3c93b8e19bd3ccb61331c792"
server_key_iv: "2ad2e7ec6c68695e27115d0bb4435b0e"
server_key_version: "encryptionkey@0"
I20260809 12:10:14.099356 4201 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.002s sys 0.005s
I20260809 12:10:14.103094 4219 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260809 12:10:14.104480 4201 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260809 12:10:14.104664 4201 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskgH5Pjm/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786277382073938-888-0/minicluster-data/master-0/data,/tmp/dist-test-taskgH5Pjm/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786277382073938-888-0/minicluster-data/master-0/wal
uuid: "4eee0b48fca245919dd1f63f2b19466e"
format_stamp: "Formatted at 2026-08-09 12:10:14 on dist-test-slave-2w3w"
server_key: "b801b61f3c93b8e19bd3ccb61331c792"
server_key_iv: "2ad2e7ec6c68695e27115d0bb4435b0e"
server_key_version: "encryptionkey@0"
I20260809 12:10:14.104899 4201 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskgH5Pjm/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786277382073938-888-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskgH5Pjm/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786277382073938-888-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskgH5Pjm/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786277382073938-888-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260809 12:10:14.122165 4201 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260809 12:10:14.126771 4201 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260809 12:10:14.127130 4201 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260809 12:10:14.146456 4201 rpc_server.cc:307] RPC server started. Bound to: 127.0.222.62:37839
I20260809 12:10:14.146489 4273 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.222.62:37839 every 8 connection(s)
I20260809 12:10:14.147900 4201 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskgH5Pjm/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786277382073938-888-0/minicluster-data/master-0/data/info.pb
I20260809 12:10:14.151603 4274 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260809 12:10:14.156039 888 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskgH5Pjm/build/asan/bin/kudu as pid 4201
I20260809 12:10:14.156273 888 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskgH5Pjm/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786277382073938-888-0/minicluster-data/master-0/wal/instance
I20260809 12:10:14.157114 888 external_mini_cluster.cc:1428] Setting key 922b9c3516b992cbb1f9e69c391bedb8
I20260809 12:10:14.163708 4274 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 4eee0b48fca245919dd1f63f2b19466e: Bootstrap starting.
I20260809 12:10:14.167461 4274 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 4eee0b48fca245919dd1f63f2b19466e: Neither blocks nor log segments found. Creating new log.
I20260809 12:10:14.168723 4274 log.cc:824] T 00000000000000000000000000000000 P 4eee0b48fca245919dd1f63f2b19466e: Log is configured to *not* fsync() on all Append() calls
I20260809 12:10:14.172248 4274 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 4eee0b48fca245919dd1f63f2b19466e: No bootstrap required, opened a new log
Aug 09 12:10:14 dist-test-slave-2w3w krb5kdc[4187](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786277413, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.222.62@KRBTEST.COM
I20260809 12:10:14.182508 4277 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 12:10:14.159249 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33524 (local address 127.0.222.62:37839)
0809 12:10:14.160003 (+ 754us) server_negotiation.cc:206] Beginning negotiation
0809 12:10:14.160017 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0809 12:10:14.160062 (+ 45us) server_negotiation.cc:407] Connection header received
0809 12:10:14.162039 (+ 1977us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 12:10:14.162061 (+ 22us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 12:10:14.162419 (+ 358us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 12:10:14.162725 (+ 306us) server_negotiation.cc:226] Negotiated authn=SASL
0809 12:10:14.164684 (+ 1959us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 12:10:14.165973 (+ 1289us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 12:10:14.167818 (+ 1845us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 12:10:14.168532 (+ 714us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 12:10:14.168661 (+ 129us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 12:10:14.173396 (+ 4735us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 12:10:14.173408 (+ 12us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 12:10:14.173426 (+ 18us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0809 12:10:14.173470 (+ 44us) server_negotiation.cc:922] calling sasl_server_start()
0809 12:10:14.177591 (+ 4121us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 12:10:14.178261 (+ 670us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 12:10:14.178270 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 12:10:14.178283 (+ 13us) server_negotiation.cc:967] calling sasl_server_step()
0809 12:10:14.178503 (+ 220us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 12:10:14.178868 (+ 365us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 12:10:14.178874 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 12:10:14.178878 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0809 12:10:14.179652 (+ 774us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 12:10:14.180000 (+ 348us) server_negotiation.cc:1035] Waiting for connection context
0809 12:10:14.180319 (+ 319us) server_negotiation.cc:299] Negotiation successful
0809 12:10:14.180789 (+ 470us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":541,"thread_start_us":336,"threads_started":1}
I20260809 12:10:14.183741 4274 raft_consensus.cc:348] T 00000000000000000000000000000000 P 4eee0b48fca245919dd1f63f2b19466e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4eee0b48fca245919dd1f63f2b19466e" member_type: VOTER last_known_addr { host: "127.0.222.62" port: 37839 } }
I20260809 12:10:14.184167 4274 raft_consensus.cc:374] T 00000000000000000000000000000000 P 4eee0b48fca245919dd1f63f2b19466e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260809 12:10:14.184309 4274 raft_consensus.cc:729] T 00000000000000000000000000000000 P 4eee0b48fca245919dd1f63f2b19466e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4eee0b48fca245919dd1f63f2b19466e, State: Initialized, Role: FOLLOWER
I20260809 12:10:14.184888 4274 consensus_queue.cc:260] T 00000000000000000000000000000000 P 4eee0b48fca245919dd1f63f2b19466e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4eee0b48fca245919dd1f63f2b19466e" member_type: VOTER last_known_addr { host: "127.0.222.62" port: 37839 } }
I20260809 12:10:14.185135 4274 raft_consensus.cc:388] T 00000000000000000000000000000000 P 4eee0b48fca245919dd1f63f2b19466e [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260809 12:10:14.185285 4274 raft_consensus.cc:482] T 00000000000000000000000000000000 P 4eee0b48fca245919dd1f63f2b19466e [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260809 12:10:14.185488 4274 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 4eee0b48fca245919dd1f63f2b19466e [term 0 FOLLOWER]: Advancing to term 1
I20260809 12:10:14.187881 4274 raft_consensus.cc:504] T 00000000000000000000000000000000 P 4eee0b48fca245919dd1f63f2b19466e [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4eee0b48fca245919dd1f63f2b19466e" member_type: VOTER last_known_addr { host: "127.0.222.62" port: 37839 } }
I20260809 12:10:14.188470 4274 leader_election.cc:302] T 00000000000000000000000000000000 P 4eee0b48fca245919dd1f63f2b19466e [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 4eee0b48fca245919dd1f63f2b19466e; no voters:
I20260809 12:10:14.189217 4274 leader_election.cc:288] T 00000000000000000000000000000000 P 4eee0b48fca245919dd1f63f2b19466e [CANDIDATE]: Term 1 election: Requested vote from peers
I20260809 12:10:14.189355 4279 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 4eee0b48fca245919dd1f63f2b19466e [term 1 FOLLOWER]: Leader election won for term 1
I20260809 12:10:14.190150 4279 raft_consensus.cc:686] T 00000000000000000000000000000000 P 4eee0b48fca245919dd1f63f2b19466e [term 1 LEADER]: Becoming Leader. State: Replica: 4eee0b48fca245919dd1f63f2b19466e, State: Running, Role: LEADER
I20260809 12:10:14.190706 4279 consensus_queue.cc:237] T 00000000000000000000000000000000 P 4eee0b48fca245919dd1f63f2b19466e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4eee0b48fca245919dd1f63f2b19466e" member_type: VOTER last_known_addr { host: "127.0.222.62" port: 37839 } }
I20260809 12:10:14.191428 4274 sys_catalog.cc:564] T 00000000000000000000000000000000 P 4eee0b48fca245919dd1f63f2b19466e [sys.catalog]: configured and running, proceeding with master startup.
I20260809 12:10:14.193656 4281 sys_catalog.cc:455] T 00000000000000000000000000000000 P 4eee0b48fca245919dd1f63f2b19466e [sys.catalog]: SysCatalogTable state changed. Reason: New leader 4eee0b48fca245919dd1f63f2b19466e. Latest consensus state: current_term: 1 leader_uuid: "4eee0b48fca245919dd1f63f2b19466e" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4eee0b48fca245919dd1f63f2b19466e" member_type: VOTER last_known_addr { host: "127.0.222.62" port: 37839 } } }
I20260809 12:10:14.194149 4281 sys_catalog.cc:458] T 00000000000000000000000000000000 P 4eee0b48fca245919dd1f63f2b19466e [sys.catalog]: This master's current role is: LEADER
I20260809 12:10:14.193682 4280 sys_catalog.cc:455] T 00000000000000000000000000000000 P 4eee0b48fca245919dd1f63f2b19466e [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "4eee0b48fca245919dd1f63f2b19466e" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4eee0b48fca245919dd1f63f2b19466e" member_type: VOTER last_known_addr { host: "127.0.222.62" port: 37839 } } }
I20260809 12:10:14.196084 4280 sys_catalog.cc:458] T 00000000000000000000000000000000 P 4eee0b48fca245919dd1f63f2b19466e [sys.catalog]: This master's current role is: LEADER
I20260809 12:10:14.196838 4287 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260809 12:10:14.202390 4287 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260809 12:10:14.213070 4287 catalog_manager.cc:1347] Generated new cluster ID: bf052564e59644ccab4e5b668290da91
I20260809 12:10:14.213192 4287 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260809 12:10:14.249063 4287 catalog_manager.cc:1370] Generated new certificate authority record
I20260809 12:10:14.250644 4287 catalog_manager.cc:1504] Loading token signing keys...
I20260809 12:10:14.270342 4287 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 4eee0b48fca245919dd1f63f2b19466e: Generated new TSK 0
I20260809 12:10:14.271333 4287 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.0.222.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.222.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.222.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskgH5Pjm/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786277382073938-888-0/krb5kdc/kudu_127.0.222.1.keytab.
Entry for principal kudu/127.0.222.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskgH5Pjm/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786277382073938-888-0/krb5kdc/kudu_127.0.222.1.keytab.
I20260809 12:10:14.346351 888 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskgH5Pjm/build/asan/bin/kudu
/tmp/dist-test-taskgH5Pjm/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskgH5Pjm/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786277382073938-888-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskgH5Pjm/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786277382073938-888-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskgH5Pjm/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786277382073938-888-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskgH5Pjm/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786277382073938-888-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.222.1:0
--local_ip_for_outbound_sockets=127.0.222.1
--webserver_interface=127.0.222.1
--webserver_port=0
--tserver_master_addrs=127.0.222.62:37839
--keytab_file=/tmp/dist-test-taskgH5Pjm/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786277382073938-888-0/krb5kdc/kudu_127.0.222.1.keytab
--principal=kudu/127.0.222.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.222.20:45373
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskgH5Pjm/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786277382073938-888-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskgH5Pjm/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786277382073938-888-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskgH5Pjm/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786277382073938-888-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260809 12:10:14.706947 4303 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260809 12:10:14.707330 4303 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260809 12:10:14.707516 4303 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260809 12:10:14.715289 4303 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260809 12:10:14.715426 4303 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260809 12:10:14.715615 4303 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.222.1
I20260809 12:10:14.725783 4303 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.222.20:45373
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskgH5Pjm/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786277382073938-888-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskgH5Pjm/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786277382073938-888-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskgH5Pjm/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786277382073938-888-0/krb5kdc/kudu_127.0.222.1.keytab
--principal=kudu/127.0.222.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.222.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskgH5Pjm/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786277382073938-888-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.222.1
--webserver_port=0
--tserver_master_addrs=127.0.222.62:37839
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.222.1
--log_dir=/tmp/dist-test-taskgH5Pjm/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786277382073938-888-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 09 Aug 2026 11:06:41 UTC on ccycloud.quasar-bwoogy.root.comops.site
build id 4575991
ASAN enabled
I20260809 12:10:14.727223 4303 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260809 12:10:14.728546 4303 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260809 12:10:14.741613 4312 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 12:10:14.741535 4310 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 12:10:14.741806 4309 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260809 12:10:14.744287 4303 server_base.cc:1034] running on GCE node
I20260809 12:10:14.744889 4303 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260809 12:10:14.745811 4303 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260809 12:10:14.747047 4303 hybrid_clock.cc:648] HybridClock initialized: now 1786277414746981 us; error 58 us; skew 500 ppm
I20260809 12:10:14.747313 4303 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 09 12:10:14 dist-test-slave-2w3w krb5kdc[4187](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786277414, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.222.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260809 12:10:14.750406 4303 init.cc:377] Logged in from keytab as kudu/127.0.222.1@KRBTEST.COM (short username kudu)
I20260809 12:10:14.751894 4303 webserver.cc:460] Webserver started at http://127.0.222.1:43593/ using document root <none> and password file <none>
I20260809 12:10:14.752616 4303 fs_manager.cc:359] Metadata directory not provided
I20260809 12:10:14.752763 4303 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260809 12:10:14.753059 4303 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260809 12:10:14.755970 4303 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskgH5Pjm/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786277382073938-888-0/minicluster-data/ts-0/data/instance:
uuid: "7535146c4c054806b879b8cfcceed76c"
format_stamp: "Formatted at 2026-08-09 12:10:14 on dist-test-slave-2w3w"
server_key: "de9e97cc0df3bed3715568c605bf5187"
server_key_iv: "4f89eb3f85759601c321fc4f249072a9"
server_key_version: "encryptionkey@0"
I20260809 12:10:14.756764 4303 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskgH5Pjm/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786277382073938-888-0/minicluster-data/ts-0/wal/instance:
uuid: "7535146c4c054806b879b8cfcceed76c"
format_stamp: "Formatted at 2026-08-09 12:10:14 on dist-test-slave-2w3w"
server_key: "de9e97cc0df3bed3715568c605bf5187"
server_key_iv: "4f89eb3f85759601c321fc4f249072a9"
server_key_version: "encryptionkey@0"
I20260809 12:10:14.761726 4303 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.000s
I20260809 12:10:14.764993 4320 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260809 12:10:14.766227 4303 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260809 12:10:14.766402 4303 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskgH5Pjm/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786277382073938-888-0/minicluster-data/ts-0/data,/tmp/dist-test-taskgH5Pjm/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786277382073938-888-0/minicluster-data/ts-0/wal
uuid: "7535146c4c054806b879b8cfcceed76c"
format_stamp: "Formatted at 2026-08-09 12:10:14 on dist-test-slave-2w3w"
server_key: "de9e97cc0df3bed3715568c605bf5187"
server_key_iv: "4f89eb3f85759601c321fc4f249072a9"
server_key_version: "encryptionkey@0"
I20260809 12:10:14.766619 4303 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskgH5Pjm/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786277382073938-888-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskgH5Pjm/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786277382073938-888-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskgH5Pjm/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786277382073938-888-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260809 12:10:14.783241 4303 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260809 12:10:14.786846 4303 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260809 12:10:14.787158 4303 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260809 12:10:14.788159 4303 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260809 12:10:14.789460 4303 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260809 12:10:14.789574 4303 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260809 12:10:14.789721 4303 ts_tablet_manager.cc:610] Registered 0 tablets
I20260809 12:10:14.789795 4303 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260809 12:10:14.824297 4303 rpc_server.cc:307] RPC server started. Bound to: 127.0.222.1:38879
I20260809 12:10:14.824330 4441 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.222.1:38879 every 8 connection(s)
I20260809 12:10:14.825618 4303 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskgH5Pjm/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786277382073938-888-0/minicluster-data/ts-0/data/info.pb
I20260809 12:10:14.833045 888 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskgH5Pjm/build/asan/bin/kudu as pid 4303
I20260809 12:10:14.833244 888 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskgH5Pjm/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786277382073938-888-0/minicluster-data/ts-0/wal/instance
I20260809 12:10:14.833897 888 external_mini_cluster.cc:1428] Setting key f4b4bde627d994f95b7f42ec2f957bad
Aug 09 12:10:14 dist-test-slave-2w3w krb5kdc[4187](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786277414, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.222.1@KRBTEST.COM for kudu/127.0.222.62@KRBTEST.COM
I20260809 12:10:14.846017 4445 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 12:10:14.828297 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.222.1:55093 (local address 127.0.222.62:37839)
0809 12:10:14.828806 (+ 509us) server_negotiation.cc:206] Beginning negotiation
0809 12:10:14.828811 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0809 12:10:14.829721 (+ 910us) server_negotiation.cc:407] Connection header received
0809 12:10:14.831651 (+ 1930us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 12:10:14.831656 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 12:10:14.831744 (+ 88us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 12:10:14.831868 (+ 124us) server_negotiation.cc:226] Negotiated authn=SASL
0809 12:10:14.833414 (+ 1546us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 12:10:14.834223 (+ 809us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 12:10:14.836564 (+ 2341us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 12:10:14.836974 (+ 410us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 12:10:14.837117 (+ 143us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 12:10:14.840538 (+ 3421us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 12:10:14.840543 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 12:10:14.840546 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0809 12:10:14.840577 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0809 12:10:14.843077 (+ 2500us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 12:10:14.843775 (+ 698us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 12:10:14.843782 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 12:10:14.843786 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0809 12:10:14.843911 (+ 125us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 12:10:14.844325 (+ 414us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 12:10:14.844330 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 12:10:14.844334 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0809 12:10:14.844609 (+ 275us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 12:10:14.844766 (+ 157us) server_negotiation.cc:1035] Waiting for connection context
0809 12:10:14.845433 (+ 667us) server_negotiation.cc:299] Negotiation successful
0809 12:10:14.845650 (+ 217us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":362,"thread_start_us":239,"threads_started":1}
I20260809 12:10:14.846886 4446 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 12:10:14.828834 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.222.62:37839 (local address 127.0.222.1:55093)
0809 12:10:14.829545 (+ 711us) negotiation.cc:107] Waiting for socket to connect
0809 12:10:14.829590 (+ 45us) client_negotiation.cc:174] Beginning negotiation
0809 12:10:14.831379 (+ 1789us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 12:10:14.832106 (+ 727us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 12:10:14.832114 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 12:10:14.832609 (+ 495us) client_negotiation.cc:189] Negotiated authn=SASL
0809 12:10:14.833192 (+ 583us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 12:10:14.833220 (+ 28us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 12:10:14.834389 (+ 1169us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 12:10:14.834399 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 12:10:14.836394 (+ 1995us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 12:10:14.836406 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 12:10:14.837207 (+ 801us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 12:10:14.837217 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 12:10:14.837460 (+ 243us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 12:10:14.838138 (+ 678us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0809 12:10:14.838165 (+ 27us) client_negotiation.cc:656] calling sasl_client_start()
0809 12:10:14.840323 (+ 2158us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0809 12:10:14.843211 (+ 2888us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0809 12:10:14.843219 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0809 12:10:14.843234 (+ 15us) client_negotiation.cc:760] calling sasl_client_step()
0809 12:10:14.843635 (+ 401us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0809 12:10:14.844019 (+ 384us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0809 12:10:14.844024 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0809 12:10:14.844028 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0809 12:10:14.844203 (+ 175us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0809 12:10:14.844738 (+ 535us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0809 12:10:14.844746 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0809 12:10:14.845062 (+ 316us) client_negotiation.cc:769] Sending connection context
0809 12:10:14.845316 (+ 254us) client_negotiation.cc:240] Negotiation successful
0809 12:10:14.845808 (+ 492us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":484,"thread_start_us":283,"threads_started":1}
I20260809 12:10:14.848471 4442 heartbeater.cc:347] Connected to a master server at 127.0.222.62:37839
I20260809 12:10:14.848839 4442 heartbeater.cc:464] Registering TS with master...
I20260809 12:10:14.849699 4442 heartbeater.cc:511] Master 127.0.222.62:37839 requested a full tablet report, sending...
I20260809 12:10:14.852185 4238 ts_manager.cc:194] Registered new tserver with Master: 7535146c4c054806b879b8cfcceed76c (127.0.222.1:38879)
I20260809 12:10:14.854512 4238 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.222.1@KRBTEST.COM'} at 127.0.222.1:55093
WARNING: no policy specified for kudu/127.0.222.2@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.222.2@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.222.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskgH5Pjm/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786277382073938-888-0/krb5kdc/kudu_127.0.222.2.keytab.
Entry for principal kudu/127.0.222.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskgH5Pjm/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786277382073938-888-0/krb5kdc/kudu_127.0.222.2.keytab.
I20260809 12:10:14.883800 888 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskgH5Pjm/build/asan/bin/kudu
/tmp/dist-test-taskgH5Pjm/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskgH5Pjm/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786277382073938-888-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskgH5Pjm/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786277382073938-888-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskgH5Pjm/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786277382073938-888-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskgH5Pjm/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786277382073938-888-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.222.2:0
--local_ip_for_outbound_sockets=127.0.222.2
--webserver_interface=127.0.222.2
--webserver_port=0
--tserver_master_addrs=127.0.222.62:37839
--keytab_file=/tmp/dist-test-taskgH5Pjm/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786277382073938-888-0/krb5kdc/kudu_127.0.222.2.keytab
--principal=kudu/127.0.222.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.222.20:45373
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskgH5Pjm/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786277382073938-888-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskgH5Pjm/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786277382073938-888-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskgH5Pjm/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786277382073938-888-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260809 12:10:15.218586 4449 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260809 12:10:15.218874 4449 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260809 12:10:15.219080 4449 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260809 12:10:15.226505 4449 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260809 12:10:15.226622 4449 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260809 12:10:15.226826 4449 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.222.2
I20260809 12:10:15.236922 4449 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.222.20:45373
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskgH5Pjm/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786277382073938-888-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskgH5Pjm/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786277382073938-888-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskgH5Pjm/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786277382073938-888-0/krb5kdc/kudu_127.0.222.2.keytab
--principal=kudu/127.0.222.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.222.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskgH5Pjm/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786277382073938-888-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.222.2
--webserver_port=0
--tserver_master_addrs=127.0.222.62:37839
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.222.2
--log_dir=/tmp/dist-test-taskgH5Pjm/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786277382073938-888-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 09 Aug 2026 11:06:41 UTC on ccycloud.quasar-bwoogy.root.comops.site
build id 4575991
ASAN enabled
I20260809 12:10:15.238277 4449 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260809 12:10:15.239667 4449 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260809 12:10:15.252640 4455 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 12:10:15.252672 4457 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 12:10:15.252771 4459 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260809 12:10:15.253890 4449 server_base.cc:1034] running on GCE node
I20260809 12:10:15.254457 4449 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260809 12:10:15.255422 4449 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260809 12:10:15.256628 4449 hybrid_clock.cc:648] HybridClock initialized: now 1786277415256584 us; error 34 us; skew 500 ppm
I20260809 12:10:15.256935 4449 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 09 12:10:15 dist-test-slave-2w3w krb5kdc[4187](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786277415, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.222.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260809 12:10:15.260001 4449 init.cc:377] Logged in from keytab as kudu/127.0.222.2@KRBTEST.COM (short username kudu)
I20260809 12:10:15.261307 4449 webserver.cc:460] Webserver started at http://127.0.222.2:35475/ using document root <none> and password file <none>
I20260809 12:10:15.261936 4449 fs_manager.cc:359] Metadata directory not provided
I20260809 12:10:15.262035 4449 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260809 12:10:15.262277 4449 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260809 12:10:15.265141 4449 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskgH5Pjm/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786277382073938-888-0/minicluster-data/ts-1/data/instance:
uuid: "09834e6a78124679bc338df8a4698e84"
format_stamp: "Formatted at 2026-08-09 12:10:15 on dist-test-slave-2w3w"
server_key: "e8896fc380681a96da3577aa11e9ec78"
server_key_iv: "ef35cc5d5ae826c32a70fc698493e325"
server_key_version: "encryptionkey@0"
I20260809 12:10:15.265880 4449 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskgH5Pjm/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786277382073938-888-0/minicluster-data/ts-1/wal/instance:
uuid: "09834e6a78124679bc338df8a4698e84"
format_stamp: "Formatted at 2026-08-09 12:10:15 on dist-test-slave-2w3w"
server_key: "e8896fc380681a96da3577aa11e9ec78"
server_key_iv: "ef35cc5d5ae826c32a70fc698493e325"
server_key_version: "encryptionkey@0"
I20260809 12:10:15.271152 4449 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.004s sys 0.004s
I20260809 12:10:15.274678 4466 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260809 12:10:15.276033 4449 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260809 12:10:15.276203 4449 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskgH5Pjm/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786277382073938-888-0/minicluster-data/ts-1/data,/tmp/dist-test-taskgH5Pjm/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786277382073938-888-0/minicluster-data/ts-1/wal
uuid: "09834e6a78124679bc338df8a4698e84"
format_stamp: "Formatted at 2026-08-09 12:10:15 on dist-test-slave-2w3w"
server_key: "e8896fc380681a96da3577aa11e9ec78"
server_key_iv: "ef35cc5d5ae826c32a70fc698493e325"
server_key_version: "encryptionkey@0"
I20260809 12:10:15.276398 4449 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskgH5Pjm/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786277382073938-888-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskgH5Pjm/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786277382073938-888-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskgH5Pjm/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786277382073938-888-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260809 12:10:15.298934 4449 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260809 12:10:15.303575 4449 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260809 12:10:15.303930 4449 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260809 12:10:15.304932 4449 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260809 12:10:15.306356 4449 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260809 12:10:15.306465 4449 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260809 12:10:15.306622 4449 ts_tablet_manager.cc:610] Registered 0 tablets
I20260809 12:10:15.306699 4449 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260809 12:10:15.339545 4449 rpc_server.cc:307] RPC server started. Bound to: 127.0.222.2:39339
I20260809 12:10:15.339592 4587 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.222.2:39339 every 8 connection(s)
I20260809 12:10:15.340885 4449 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskgH5Pjm/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786277382073938-888-0/minicluster-data/ts-1/data/info.pb
I20260809 12:10:15.344835 888 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskgH5Pjm/build/asan/bin/kudu as pid 4449
I20260809 12:10:15.345026 888 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskgH5Pjm/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786277382073938-888-0/minicluster-data/ts-1/wal/instance
I20260809 12:10:15.345542 888 external_mini_cluster.cc:1428] Setting key c2a345e9aa4230bcf01f5d803bc3c652
Aug 09 12:10:15 dist-test-slave-2w3w krb5kdc[4187](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786277415, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.222.2@KRBTEST.COM for kudu/127.0.222.62@KRBTEST.COM
I20260809 12:10:15.363119 4445 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 12:10:15.343666 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.222.2:42977 (local address 127.0.222.62:37839)
0809 12:10:15.343856 (+ 190us) server_negotiation.cc:206] Beginning negotiation
0809 12:10:15.343860 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0809 12:10:15.345032 (+ 1172us) server_negotiation.cc:407] Connection header received
0809 12:10:15.347112 (+ 2080us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 12:10:15.347117 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 12:10:15.347184 (+ 67us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 12:10:15.347271 (+ 87us) server_negotiation.cc:226] Negotiated authn=SASL
0809 12:10:15.349320 (+ 2049us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 12:10:15.350025 (+ 705us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 12:10:15.352541 (+ 2516us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 12:10:15.353053 (+ 512us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 12:10:15.353156 (+ 103us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 12:10:15.357303 (+ 4147us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 12:10:15.357309 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 12:10:15.357312 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0809 12:10:15.357345 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0809 12:10:15.360017 (+ 2672us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 12:10:15.360773 (+ 756us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 12:10:15.360779 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 12:10:15.360781 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0809 12:10:15.360916 (+ 135us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 12:10:15.361362 (+ 446us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 12:10:15.361372 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 12:10:15.361374 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0809 12:10:15.361629 (+ 255us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 12:10:15.361788 (+ 159us) server_negotiation.cc:1035] Waiting for connection context
0809 12:10:15.362481 (+ 693us) server_negotiation.cc:299] Negotiation successful
0809 12:10:15.362706 (+ 225us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":67}
I20260809 12:10:15.364166 4590 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 12:10:15.344093 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.222.62:37839 (local address 127.0.222.2:42977)
0809 12:10:15.344833 (+ 740us) negotiation.cc:107] Waiting for socket to connect
0809 12:10:15.344877 (+ 44us) client_negotiation.cc:174] Beginning negotiation
0809 12:10:15.346793 (+ 1916us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 12:10:15.347532 (+ 739us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 12:10:15.347541 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 12:10:15.348260 (+ 719us) client_negotiation.cc:189] Negotiated authn=SASL
0809 12:10:15.349088 (+ 828us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 12:10:15.349122 (+ 34us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 12:10:15.350197 (+ 1075us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 12:10:15.350206 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 12:10:15.352368 (+ 2162us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 12:10:15.352388 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 12:10:15.353294 (+ 906us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 12:10:15.353304 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 12:10:15.353630 (+ 326us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 12:10:15.354370 (+ 740us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0809 12:10:15.354405 (+ 35us) client_negotiation.cc:656] calling sasl_client_start()
0809 12:10:15.357071 (+ 2666us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0809 12:10:15.360166 (+ 3095us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0809 12:10:15.360175 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0809 12:10:15.360191 (+ 16us) client_negotiation.cc:760] calling sasl_client_step()
0809 12:10:15.360635 (+ 444us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0809 12:10:15.361034 (+ 399us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0809 12:10:15.361040 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0809 12:10:15.361043 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0809 12:10:15.361228 (+ 185us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0809 12:10:15.361763 (+ 535us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0809 12:10:15.361771 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0809 12:10:15.362103 (+ 332us) client_negotiation.cc:769] Sending connection context
0809 12:10:15.362374 (+ 271us) client_negotiation.cc:240] Negotiation successful
0809 12:10:15.362892 (+ 518us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":518,"thread_start_us":331,"threads_started":1}
I20260809 12:10:15.365703 4588 heartbeater.cc:347] Connected to a master server at 127.0.222.62:37839
I20260809 12:10:15.366057 4588 heartbeater.cc:464] Registering TS with master...
I20260809 12:10:15.367025 4588 heartbeater.cc:511] Master 127.0.222.62:37839 requested a full tablet report, sending...
WARNING: no policy specified for kudu/127.0.222.3@KRBTEST.COM; defaulting to no policy
I20260809 12:10:15.368789 4238 ts_manager.cc:194] Registered new tserver with Master: 09834e6a78124679bc338df8a4698e84 (127.0.222.2:39339)
I20260809 12:10:15.369658 4238 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.222.2@KRBTEST.COM'} at 127.0.222.2:42977
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.222.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.222.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskgH5Pjm/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786277382073938-888-0/krb5kdc/kudu_127.0.222.3.keytab.
Entry for principal kudu/127.0.222.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskgH5Pjm/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786277382073938-888-0/krb5kdc/kudu_127.0.222.3.keytab.
I20260809 12:10:15.400415 888 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskgH5Pjm/build/asan/bin/kudu
/tmp/dist-test-taskgH5Pjm/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskgH5Pjm/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786277382073938-888-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskgH5Pjm/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786277382073938-888-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskgH5Pjm/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786277382073938-888-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskgH5Pjm/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786277382073938-888-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.222.3:0
--local_ip_for_outbound_sockets=127.0.222.3
--webserver_interface=127.0.222.3
--webserver_port=0
--tserver_master_addrs=127.0.222.62:37839
--keytab_file=/tmp/dist-test-taskgH5Pjm/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786277382073938-888-0/krb5kdc/kudu_127.0.222.3.keytab
--principal=kudu/127.0.222.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.222.20:45373
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskgH5Pjm/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786277382073938-888-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskgH5Pjm/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786277382073938-888-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskgH5Pjm/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786277382073938-888-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260809 12:10:15.749135 4593 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260809 12:10:15.749421 4593 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260809 12:10:15.749517 4593 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260809 12:10:15.756772 4593 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260809 12:10:15.756879 4593 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260809 12:10:15.756992 4593 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.222.3
I20260809 12:10:15.766430 4593 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.222.20:45373
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskgH5Pjm/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786277382073938-888-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskgH5Pjm/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786277382073938-888-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskgH5Pjm/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786277382073938-888-0/krb5kdc/kudu_127.0.222.3.keytab
--principal=kudu/127.0.222.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.222.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskgH5Pjm/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786277382073938-888-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.222.3
--webserver_port=0
--tserver_master_addrs=127.0.222.62:37839
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.222.3
--log_dir=/tmp/dist-test-taskgH5Pjm/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786277382073938-888-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 09 Aug 2026 11:06:41 UTC on ccycloud.quasar-bwoogy.root.comops.site
build id 4575991
ASAN enabled
I20260809 12:10:15.767689 4593 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260809 12:10:15.768996 4593 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260809 12:10:15.782490 4601 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 12:10:15.782461 4599 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 12:10:15.783075 4603 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260809 12:10:15.783829 4593 server_base.cc:1034] running on GCE node
I20260809 12:10:15.784407 4593 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260809 12:10:15.785386 4593 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260809 12:10:15.786599 4593 hybrid_clock.cc:648] HybridClock initialized: now 1786277415786548 us; error 37 us; skew 500 ppm
I20260809 12:10:15.786897 4593 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 09 12:10:15 dist-test-slave-2w3w krb5kdc[4187](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786277415, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.222.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260809 12:10:15.790096 4593 init.cc:377] Logged in from keytab as kudu/127.0.222.3@KRBTEST.COM (short username kudu)
I20260809 12:10:15.791699 4593 webserver.cc:460] Webserver started at http://127.0.222.3:35795/ using document root <none> and password file <none>
I20260809 12:10:15.792420 4593 fs_manager.cc:359] Metadata directory not provided
I20260809 12:10:15.792520 4593 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260809 12:10:15.792845 4593 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260809 12:10:15.795948 4593 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskgH5Pjm/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786277382073938-888-0/minicluster-data/ts-2/data/instance:
uuid: "b6cfc4027efc4994a653216b364efabf"
format_stamp: "Formatted at 2026-08-09 12:10:15 on dist-test-slave-2w3w"
server_key: "dcdf5a7c39ccd28989bd95d56c5a2be8"
server_key_iv: "c01f5409eda20e2f56d667702fc4e514"
server_key_version: "encryptionkey@0"
I20260809 12:10:15.796674 4593 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskgH5Pjm/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786277382073938-888-0/minicluster-data/ts-2/wal/instance:
uuid: "b6cfc4027efc4994a653216b364efabf"
format_stamp: "Formatted at 2026-08-09 12:10:15 on dist-test-slave-2w3w"
server_key: "dcdf5a7c39ccd28989bd95d56c5a2be8"
server_key_iv: "c01f5409eda20e2f56d667702fc4e514"
server_key_version: "encryptionkey@0"
I20260809 12:10:15.801996 4593 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.002s sys 0.004s
I20260809 12:10:15.805523 4611 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260809 12:10:15.806914 4593 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260809 12:10:15.807170 4593 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskgH5Pjm/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786277382073938-888-0/minicluster-data/ts-2/data,/tmp/dist-test-taskgH5Pjm/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786277382073938-888-0/minicluster-data/ts-2/wal
uuid: "b6cfc4027efc4994a653216b364efabf"
format_stamp: "Formatted at 2026-08-09 12:10:15 on dist-test-slave-2w3w"
server_key: "dcdf5a7c39ccd28989bd95d56c5a2be8"
server_key_iv: "c01f5409eda20e2f56d667702fc4e514"
server_key_version: "encryptionkey@0"
I20260809 12:10:15.807379 4593 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskgH5Pjm/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786277382073938-888-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskgH5Pjm/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786277382073938-888-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskgH5Pjm/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786277382073938-888-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260809 12:10:15.831281 4593 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260809 12:10:15.835356 4593 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260809 12:10:15.835659 4593 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260809 12:10:15.836640 4593 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260809 12:10:15.838029 4593 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260809 12:10:15.838145 4593 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260809 12:10:15.838291 4593 ts_tablet_manager.cc:610] Registered 0 tablets
I20260809 12:10:15.838366 4593 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260809 12:10:15.858142 4442 heartbeater.cc:503] Master 127.0.222.62:37839 was elected leader, sending a full tablet report...
I20260809 12:10:15.874521 4593 rpc_server.cc:307] RPC server started. Bound to: 127.0.222.3:34371
I20260809 12:10:15.874571 4730 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.222.3:34371 every 8 connection(s)
I20260809 12:10:15.875974 4593 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskgH5Pjm/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786277382073938-888-0/minicluster-data/ts-2/data/info.pb
I20260809 12:10:15.885600 888 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskgH5Pjm/build/asan/bin/kudu as pid 4593
I20260809 12:10:15.885785 888 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskgH5Pjm/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786277382073938-888-0/minicluster-data/ts-2/wal/instance
I20260809 12:10:15.886365 888 external_mini_cluster.cc:1428] Setting key f6f5705613e6f8a3a397bfff467001c2
Aug 09 12:10:15 dist-test-slave-2w3w krb5kdc[4187](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786277415, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.222.3@KRBTEST.COM for kudu/127.0.222.62@KRBTEST.COM
I20260809 12:10:15.897603 4733 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 12:10:15.878754 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.222.3:39319 (local address 127.0.222.62:37839)
0809 12:10:15.879269 (+ 515us) server_negotiation.cc:206] Beginning negotiation
0809 12:10:15.879275 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0809 12:10:15.880235 (+ 960us) server_negotiation.cc:407] Connection header received
0809 12:10:15.882187 (+ 1952us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 12:10:15.882192 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 12:10:15.882260 (+ 68us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 12:10:15.882368 (+ 108us) server_negotiation.cc:226] Negotiated authn=SASL
0809 12:10:15.883923 (+ 1555us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 12:10:15.884720 (+ 797us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 12:10:15.887115 (+ 2395us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 12:10:15.887660 (+ 545us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 12:10:15.887806 (+ 146us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 12:10:15.891794 (+ 3988us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 12:10:15.891832 (+ 38us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 12:10:15.891835 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0809 12:10:15.891871 (+ 36us) server_negotiation.cc:922] calling sasl_server_start()
0809 12:10:15.894579 (+ 2708us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 12:10:15.895541 (+ 962us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 12:10:15.895545 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 12:10:15.895548 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0809 12:10:15.895688 (+ 140us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 12:10:15.896099 (+ 411us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 12:10:15.896102 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 12:10:15.896104 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0809 12:10:15.896370 (+ 266us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 12:10:15.896485 (+ 115us) server_negotiation.cc:1035] Waiting for connection context
0809 12:10:15.897165 (+ 680us) server_negotiation.cc:299] Negotiation successful
0809 12:10:15.897338 (+ 173us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":407,"thread_start_us":301,"threads_started":1}
I20260809 12:10:15.898552 4734 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 12:10:15.879232 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.222.62:37839 (local address 127.0.222.3:39319)
0809 12:10:15.880026 (+ 794us) negotiation.cc:107] Waiting for socket to connect
0809 12:10:15.880064 (+ 38us) client_negotiation.cc:174] Beginning negotiation
0809 12:10:15.881875 (+ 1811us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 12:10:15.882580 (+ 705us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 12:10:15.882587 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 12:10:15.883180 (+ 593us) client_negotiation.cc:189] Negotiated authn=SASL
0809 12:10:15.883737 (+ 557us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 12:10:15.883754 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 12:10:15.884851 (+ 1097us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 12:10:15.884854 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 12:10:15.886884 (+ 2030us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 12:10:15.886901 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 12:10:15.887933 (+ 1032us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 12:10:15.887944 (+ 11us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 12:10:15.888254 (+ 310us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 12:10:15.889065 (+ 811us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0809 12:10:15.889089 (+ 24us) client_negotiation.cc:656] calling sasl_client_start()
0809 12:10:15.891550 (+ 2461us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0809 12:10:15.894729 (+ 3179us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0809 12:10:15.894736 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0809 12:10:15.894765 (+ 29us) client_negotiation.cc:760] calling sasl_client_step()
0809 12:10:15.895415 (+ 650us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0809 12:10:15.895789 (+ 374us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0809 12:10:15.895792 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0809 12:10:15.895795 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0809 12:10:15.895994 (+ 199us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0809 12:10:15.896478 (+ 484us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0809 12:10:15.896483 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0809 12:10:15.896805 (+ 322us) client_negotiation.cc:769] Sending connection context
0809 12:10:15.897057 (+ 252us) client_negotiation.cc:240] Negotiation successful
0809 12:10:15.897462 (+ 405us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":495,"thread_start_us":320,"threads_started":1}
I20260809 12:10:15.900417 4731 heartbeater.cc:347] Connected to a master server at 127.0.222.62:37839
I20260809 12:10:15.900804 4731 heartbeater.cc:464] Registering TS with master...
I20260809 12:10:15.901751 4731 heartbeater.cc:511] Master 127.0.222.62:37839 requested a full tablet report, sending...
I20260809 12:10:15.903298 4238 ts_manager.cc:194] Registered new tserver with Master: b6cfc4027efc4994a653216b364efabf (127.0.222.3:34371)
I20260809 12:10:15.904084 4238 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.222.3@KRBTEST.COM'} at 127.0.222.3:39319
I20260809 12:10:15.914350 888 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
Aug 09 12:10:15 dist-test-slave-2w3w krb5kdc[4187](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786277415, etypes {rep=17 tkt=17 ses=17}, joe-interloper@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for joe-interloper@KRBTEST.COM:
Aug 09 12:10:15 dist-test-slave-2w3w krb5kdc[4187](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786277415, etypes {rep=17 tkt=17 ses=17}, joe-interloper@KRBTEST.COM for kudu/127.0.222.62@KRBTEST.COM
I20260809 12:10:15.973315 4733 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 12:10:15.958860 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33534 (local address 127.0.222.62:37839)
0809 12:10:15.959158 (+ 298us) server_negotiation.cc:206] Beginning negotiation
0809 12:10:15.959169 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0809 12:10:15.959543 (+ 374us) server_negotiation.cc:407] Connection header received
0809 12:10:15.959691 (+ 148us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 12:10:15.959695 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 12:10:15.959774 (+ 79us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 12:10:15.959877 (+ 103us) server_negotiation.cc:226] Negotiated authn=SASL
0809 12:10:15.961274 (+ 1397us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 12:10:15.962092 (+ 818us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 12:10:15.963415 (+ 1323us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 12:10:15.963864 (+ 449us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 12:10:15.963971 (+ 107us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 12:10:15.967777 (+ 3806us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 12:10:15.967782 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 12:10:15.967785 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0809 12:10:15.967825 (+ 40us) server_negotiation.cc:922] calling sasl_server_start()
0809 12:10:15.970743 (+ 2918us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 12:10:15.971531 (+ 788us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 12:10:15.971535 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 12:10:15.971537 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0809 12:10:15.971669 (+ 132us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 12:10:15.972010 (+ 341us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 12:10:15.972014 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 12:10:15.972016 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0809 12:10:15.972370 (+ 354us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 12:10:15.972568 (+ 198us) server_negotiation.cc:1035] Waiting for connection context
0809 12:10:15.972796 (+ 228us) server_negotiation.cc:299] Negotiation successful
0809 12:10:15.972975 (+ 179us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":76}
W20260809 12:10:15.973933 4238 server_base.cc:1116] Unauthorized access attempt to method kudu.master.MasterService.ConnectToMaster from {username='joe-interloper', principal='joe-interloper@KRBTEST.COM'} at 127.0.0.1:33534
I20260809 12:10:15.975900 888 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskgH5Pjm/build/asan/bin/kudu pid 4303
Aug 09 12:10:15 dist-test-slave-2w3w krb5kdc[4187](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786277415, etypes {rep=17 tkt=17 ses=17}, joe-interloper@KRBTEST.COM for kudu/127.0.222.1@KRBTEST.COM
I20260809 12:10:15.993309 4743 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 12:10:15.977459 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39066 (local address 127.0.222.1:38879)
0809 12:10:15.978096 (+ 637us) server_negotiation.cc:206] Beginning negotiation
0809 12:10:15.978106 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0809 12:10:15.978133 (+ 27us) server_negotiation.cc:407] Connection header received
0809 12:10:15.978202 (+ 69us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 12:10:15.978208 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 12:10:15.978405 (+ 197us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 12:10:15.978589 (+ 184us) server_negotiation.cc:226] Negotiated authn=SASL
0809 12:10:15.979814 (+ 1225us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 12:10:15.980918 (+ 1104us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 12:10:15.982214 (+ 1296us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 12:10:15.983055 (+ 841us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 12:10:15.983187 (+ 132us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 12:10:15.987411 (+ 4224us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 12:10:15.987424 (+ 13us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 12:10:15.987439 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0809 12:10:15.987481 (+ 42us) server_negotiation.cc:922] calling sasl_server_start()
0809 12:10:15.990270 (+ 2789us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 12:10:15.991165 (+ 895us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 12:10:15.991173 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 12:10:15.991175 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0809 12:10:15.991332 (+ 157us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 12:10:15.991683 (+ 351us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 12:10:15.991686 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 12:10:15.991688 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0809 12:10:15.992118 (+ 430us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 12:10:15.992375 (+ 257us) server_negotiation.cc:1035] Waiting for connection context
0809 12:10:15.992708 (+ 333us) server_negotiation.cc:299] Negotiation successful
0809 12:10:15.992967 (+ 259us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":452,"thread_start_us":278,"threads_started":1}
W20260809 12:10:15.994520 4437 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='joe-interloper', principal='joe-interloper@KRBTEST.COM'} at 127.0.0.1:39066
W20260809 12:10:15.995705 888 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskgH5Pjm/build/asan/bin/kudu pid 4303: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260809 12:10:15.995925 888 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskgH5Pjm/build/asan/bin/kudu with pid 4303
I20260809 12:10:16.018110 888 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskgH5Pjm/build/asan/bin/kudu pid 4449
Aug 09 12:10:16 dist-test-slave-2w3w krb5kdc[4187](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786277415, etypes {rep=17 tkt=17 ses=17}, joe-interloper@KRBTEST.COM for kudu/127.0.222.2@KRBTEST.COM
I20260809 12:10:16.034297 4744 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 12:10:16.019169 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54592 (local address 127.0.222.2:39339)
0809 12:10:16.019811 (+ 642us) server_negotiation.cc:206] Beginning negotiation
0809 12:10:16.019819 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0809 12:10:16.019847 (+ 28us) server_negotiation.cc:407] Connection header received
0809 12:10:16.019910 (+ 63us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 12:10:16.019916 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 12:10:16.020154 (+ 238us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 12:10:16.020282 (+ 128us) server_negotiation.cc:226] Negotiated authn=SASL
0809 12:10:16.021544 (+ 1262us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 12:10:16.022552 (+ 1008us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 12:10:16.023979 (+ 1427us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 12:10:16.024562 (+ 583us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 12:10:16.024664 (+ 102us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 12:10:16.028809 (+ 4145us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 12:10:16.028820 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 12:10:16.028835 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0809 12:10:16.028878 (+ 43us) server_negotiation.cc:922] calling sasl_server_start()
0809 12:10:16.031704 (+ 2826us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 12:10:16.032325 (+ 621us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 12:10:16.032332 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 12:10:16.032334 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0809 12:10:16.032463 (+ 129us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 12:10:16.032841 (+ 378us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 12:10:16.032844 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 12:10:16.032846 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0809 12:10:16.033243 (+ 397us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 12:10:16.033462 (+ 219us) server_negotiation.cc:1035] Waiting for connection context
0809 12:10:16.033790 (+ 328us) server_negotiation.cc:299] Negotiation successful
0809 12:10:16.034004 (+ 214us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":488,"thread_start_us":344,"threads_started":1}
W20260809 12:10:16.035631 4583 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='joe-interloper', principal='joe-interloper@KRBTEST.COM'} at 127.0.0.1:54592
W20260809 12:10:16.037078 888 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskgH5Pjm/build/asan/bin/kudu pid 4449: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260809 12:10:16.037225 888 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskgH5Pjm/build/asan/bin/kudu with pid 4449
I20260809 12:10:16.056756 888 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskgH5Pjm/build/asan/bin/kudu pid 4593
Aug 09 12:10:16 dist-test-slave-2w3w krb5kdc[4187](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786277415, etypes {rep=17 tkt=17 ses=17}, joe-interloper@KRBTEST.COM for kudu/127.0.222.3@KRBTEST.COM
I20260809 12:10:16.073196 4745 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 12:10:16.057677 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33216 (local address 127.0.222.3:34371)
0809 12:10:16.058273 (+ 596us) server_negotiation.cc:206] Beginning negotiation
0809 12:10:16.058280 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0809 12:10:16.058309 (+ 29us) server_negotiation.cc:407] Connection header received
0809 12:10:16.058372 (+ 63us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 12:10:16.058378 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 12:10:16.058599 (+ 221us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 12:10:16.058780 (+ 181us) server_negotiation.cc:226] Negotiated authn=SASL
0809 12:10:16.060171 (+ 1391us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 12:10:16.061217 (+ 1046us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 12:10:16.062455 (+ 1238us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 12:10:16.063107 (+ 652us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 12:10:16.063243 (+ 136us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 12:10:16.067469 (+ 4226us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 12:10:16.067486 (+ 17us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 12:10:16.067500 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0809 12:10:16.067534 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0809 12:10:16.070256 (+ 2722us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 12:10:16.070923 (+ 667us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 12:10:16.070934 (+ 11us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 12:10:16.070942 (+ 8us) server_negotiation.cc:967] calling sasl_server_step()
0809 12:10:16.071121 (+ 179us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 12:10:16.071523 (+ 402us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 12:10:16.071529 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 12:10:16.071532 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0809 12:10:16.071882 (+ 350us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 12:10:16.072213 (+ 331us) server_negotiation.cc:1035] Waiting for connection context
0809 12:10:16.072555 (+ 342us) server_negotiation.cc:299] Negotiation successful
0809 12:10:16.072837 (+ 282us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":446,"thread_start_us":267,"threads_started":1}
W20260809 12:10:16.074621 4726 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='joe-interloper', principal='joe-interloper@KRBTEST.COM'} at 127.0.0.1:33216
W20260809 12:10:16.075830 888 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskgH5Pjm/build/asan/bin/kudu pid 4593: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260809 12:10:16.076006 888 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskgH5Pjm/build/asan/bin/kudu with pid 4593
I20260809 12:10:16.095191 888 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskgH5Pjm/build/asan/bin/kudu pid 4201
I20260809 12:10:16.096426 4269 generic_service.cc:196] Checking for leaks (request via RPC)
I20260809 12:10:16.203439 888 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskgH5Pjm/build/asan/bin/kudu with pid 4201
2026-08-09T12:10:16Z chronyd exiting
[ OK ] SecurityITest.TestUnauthorizedClientKerberosCredentials (3881 ms)
[ RUN ] SecurityITest.TestAuthorizedSuperuser
Loading random data
Initializing database '/tmp/dist-test-taskgH5Pjm/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786277382073938-888-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 09 12:10:16 dist-test-slave-2w3w krb5kdc[4749](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 09 12:10:16 dist-test-slave-2w3w krb5kdc[4749](info): set up 2 sockets
Aug 09 12:10:16 dist-test-slave-2w3w krb5kdc[4749](info): commencing operation
krb5kdc: starting...
W20260809 12:10:17.352347 888 mini_kdc.cc:121] Time spent starting KDC: real 1.101s user 0.002s sys 0.024s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskgH5Pjm/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786277382073938-888-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskgH5Pjm/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786277382073938-888-0/krb5kdc/test-user.keytab.
Aug 09 12:10:17 dist-test-slave-2w3w krb5kdc[4749](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786277417, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-09T12:10:17Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-09T12:10:17Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.222.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.222.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.222.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskgH5Pjm/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786277382073938-888-0/krb5kdc/kudu_127.0.222.62.keytab.
Entry for principal kudu/127.0.222.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskgH5Pjm/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786277382073938-888-0/krb5kdc/kudu_127.0.222.62.keytab.
I20260809 12:10:17.582756 888 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskgH5Pjm/build/asan/bin/kudu
/tmp/dist-test-taskgH5Pjm/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskgH5Pjm/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786277382073938-888-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskgH5Pjm/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786277382073938-888-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskgH5Pjm/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786277382073938-888-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskgH5Pjm/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786277382073938-888-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.222.62:42731
--webserver_interface=127.0.222.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskgH5Pjm/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786277382073938-888-0/krb5kdc/kudu_127.0.222.62.keytab
--principal=kudu/127.0.222.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.222.20:39623
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.222.62:42731
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskgH5Pjm/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786277382073938-888-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskgH5Pjm/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786277382073938-888-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskgH5Pjm/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786277382073938-888-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260809 12:10:17.918586 4764 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260809 12:10:17.918949 4764 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260809 12:10:17.919137 4764 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260809 12:10:17.926314 4764 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260809 12:10:17.926460 4764 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260809 12:10:17.926581 4764 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260809 12:10:17.926680 4764 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260809 12:10:17.926767 4764 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260809 12:10:17.936787 4764 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.222.20:39623
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskgH5Pjm/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786277382073938-888-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskgH5Pjm/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786277382073938-888-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.222.62:42731
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskgH5Pjm/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786277382073938-888-0/krb5kdc/kudu_127.0.222.62.keytab
--principal=kudu/127.0.222.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.222.62:42731
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskgH5Pjm/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786277382073938-888-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.222.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskgH5Pjm/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786277382073938-888-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 09 Aug 2026 11:06:41 UTC on ccycloud.quasar-bwoogy.root.comops.site
build id 4575991
ASAN enabled
I20260809 12:10:17.938320 4764 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260809 12:10:17.939754 4764 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260809 12:10:17.952419 4770 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 12:10:17.952454 4773 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 12:10:17.952517 4771 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260809 12:10:17.953327 4764 server_base.cc:1034] running on GCE node
I20260809 12:10:17.954207 4764 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260809 12:10:17.955971 4764 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260809 12:10:17.957257 4764 hybrid_clock.cc:648] HybridClock initialized: now 1786277417957206 us; error 49 us; skew 500 ppm
I20260809 12:10:17.957660 4764 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 09 12:10:17 dist-test-slave-2w3w krb5kdc[4749](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786277417, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.222.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260809 12:10:17.961229 4764 init.cc:377] Logged in from keytab as kudu/127.0.222.62@KRBTEST.COM (short username kudu)
I20260809 12:10:17.963044 4764 webserver.cc:460] Webserver started at http://127.0.222.62:35157/ using document root <none> and password file <none>
I20260809 12:10:17.963840 4764 fs_manager.cc:359] Metadata directory not provided
I20260809 12:10:17.964006 4764 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260809 12:10:17.964370 4764 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260809 12:10:17.967957 4764 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskgH5Pjm/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786277382073938-888-0/minicluster-data/master-0/data/instance:
uuid: "6902b73aabbb49c08b476bb53895ca37"
format_stamp: "Formatted at 2026-08-09 12:10:17 on dist-test-slave-2w3w"
server_key: "f2bcc7eb15ddb9e6e6b096663f644b08"
server_key_iv: "c00cffc70c6fa2d84db6d44fa6e931be"
server_key_version: "encryptionkey@0"
I20260809 12:10:17.968844 4764 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskgH5Pjm/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786277382073938-888-0/minicluster-data/master-0/wal/instance:
uuid: "6902b73aabbb49c08b476bb53895ca37"
format_stamp: "Formatted at 2026-08-09 12:10:17 on dist-test-slave-2w3w"
server_key: "f2bcc7eb15ddb9e6e6b096663f644b08"
server_key_iv: "c00cffc70c6fa2d84db6d44fa6e931be"
server_key_version: "encryptionkey@0"
I20260809 12:10:17.974665 4764 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.001s
I20260809 12:10:17.978564 4781 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260809 12:10:17.980027 4764 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260809 12:10:17.980229 4764 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskgH5Pjm/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786277382073938-888-0/minicluster-data/master-0/data,/tmp/dist-test-taskgH5Pjm/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786277382073938-888-0/minicluster-data/master-0/wal
uuid: "6902b73aabbb49c08b476bb53895ca37"
format_stamp: "Formatted at 2026-08-09 12:10:17 on dist-test-slave-2w3w"
server_key: "f2bcc7eb15ddb9e6e6b096663f644b08"
server_key_iv: "c00cffc70c6fa2d84db6d44fa6e931be"
server_key_version: "encryptionkey@0"
I20260809 12:10:17.980479 4764 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskgH5Pjm/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786277382073938-888-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskgH5Pjm/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786277382073938-888-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskgH5Pjm/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786277382073938-888-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260809 12:10:18.008489 4764 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260809 12:10:18.013064 4764 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260809 12:10:18.013402 4764 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260809 12:10:18.033771 4764 rpc_server.cc:307] RPC server started. Bound to: 127.0.222.62:42731
I20260809 12:10:18.033802 4835 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.222.62:42731 every 8 connection(s)
I20260809 12:10:18.035317 4764 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskgH5Pjm/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786277382073938-888-0/minicluster-data/master-0/data/info.pb
I20260809 12:10:18.036617 888 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskgH5Pjm/build/asan/bin/kudu as pid 4764
I20260809 12:10:18.036870 888 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskgH5Pjm/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786277382073938-888-0/minicluster-data/master-0/wal/instance
I20260809 12:10:18.037748 888 external_mini_cluster.cc:1428] Setting key d896edc13ff793cccc9abc4c154e6122
I20260809 12:10:18.040243 4836 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
Aug 09 12:10:18 dist-test-slave-2w3w krb5kdc[4749](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786277417, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.222.62@KRBTEST.COM
I20260809 12:10:18.053511 4836 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 6902b73aabbb49c08b476bb53895ca37: Bootstrap starting.
I20260809 12:10:18.056667 4836 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 6902b73aabbb49c08b476bb53895ca37: Neither blocks nor log segments found. Creating new log.
I20260809 12:10:18.057711 4836 log.cc:824] T 00000000000000000000000000000000 P 6902b73aabbb49c08b476bb53895ca37: Log is configured to *not* fsync() on all Append() calls
I20260809 12:10:18.060497 4836 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 6902b73aabbb49c08b476bb53895ca37: No bootstrap required, opened a new log
I20260809 12:10:18.061782 4839 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 12:10:18.039925 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50500 (local address 127.0.222.62:42731)
0809 12:10:18.040708 (+ 783us) server_negotiation.cc:206] Beginning negotiation
0809 12:10:18.040718 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0809 12:10:18.040757 (+ 39us) server_negotiation.cc:407] Connection header received
0809 12:10:18.042401 (+ 1644us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 12:10:18.042421 (+ 20us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 12:10:18.042766 (+ 345us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 12:10:18.043185 (+ 419us) server_negotiation.cc:226] Negotiated authn=SASL
0809 12:10:18.045035 (+ 1850us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 12:10:18.046409 (+ 1374us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 12:10:18.048212 (+ 1803us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 12:10:18.048870 (+ 658us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 12:10:18.049002 (+ 132us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 12:10:18.053862 (+ 4860us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 12:10:18.053872 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 12:10:18.053886 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0809 12:10:18.053929 (+ 43us) server_negotiation.cc:922] calling sasl_server_start()
0809 12:10:18.057432 (+ 3503us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 12:10:18.058241 (+ 809us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 12:10:18.058254 (+ 13us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 12:10:18.058261 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0809 12:10:18.058458 (+ 197us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 12:10:18.058906 (+ 448us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 12:10:18.058910 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 12:10:18.058912 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0809 12:10:18.059509 (+ 597us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 12:10:18.059831 (+ 322us) server_negotiation.cc:1035] Waiting for connection context
0809 12:10:18.060332 (+ 501us) server_negotiation.cc:299] Negotiation successful
0809 12:10:18.060766 (+ 434us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":582,"thread_start_us":367,"threads_started":1}
I20260809 12:10:18.068584 4836 raft_consensus.cc:348] T 00000000000000000000000000000000 P 6902b73aabbb49c08b476bb53895ca37 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6902b73aabbb49c08b476bb53895ca37" member_type: VOTER last_known_addr { host: "127.0.222.62" port: 42731 } }
I20260809 12:10:18.068939 4836 raft_consensus.cc:374] T 00000000000000000000000000000000 P 6902b73aabbb49c08b476bb53895ca37 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260809 12:10:18.069046 4836 raft_consensus.cc:729] T 00000000000000000000000000000000 P 6902b73aabbb49c08b476bb53895ca37 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6902b73aabbb49c08b476bb53895ca37, State: Initialized, Role: FOLLOWER
I20260809 12:10:18.069589 4836 consensus_queue.cc:260] T 00000000000000000000000000000000 P 6902b73aabbb49c08b476bb53895ca37 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6902b73aabbb49c08b476bb53895ca37" member_type: VOTER last_known_addr { host: "127.0.222.62" port: 42731 } }
I20260809 12:10:18.069813 4836 raft_consensus.cc:388] T 00000000000000000000000000000000 P 6902b73aabbb49c08b476bb53895ca37 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260809 12:10:18.069938 4836 raft_consensus.cc:482] T 00000000000000000000000000000000 P 6902b73aabbb49c08b476bb53895ca37 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260809 12:10:18.070094 4836 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 6902b73aabbb49c08b476bb53895ca37 [term 0 FOLLOWER]: Advancing to term 1
I20260809 12:10:18.072376 4836 raft_consensus.cc:504] T 00000000000000000000000000000000 P 6902b73aabbb49c08b476bb53895ca37 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6902b73aabbb49c08b476bb53895ca37" member_type: VOTER last_known_addr { host: "127.0.222.62" port: 42731 } }
I20260809 12:10:18.072853 4836 leader_election.cc:302] T 00000000000000000000000000000000 P 6902b73aabbb49c08b476bb53895ca37 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 6902b73aabbb49c08b476bb53895ca37; no voters:
I20260809 12:10:18.073457 4836 leader_election.cc:288] T 00000000000000000000000000000000 P 6902b73aabbb49c08b476bb53895ca37 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260809 12:10:18.073622 4841 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 6902b73aabbb49c08b476bb53895ca37 [term 1 FOLLOWER]: Leader election won for term 1
I20260809 12:10:18.074309 4841 raft_consensus.cc:686] T 00000000000000000000000000000000 P 6902b73aabbb49c08b476bb53895ca37 [term 1 LEADER]: Becoming Leader. State: Replica: 6902b73aabbb49c08b476bb53895ca37, State: Running, Role: LEADER
I20260809 12:10:18.074888 4841 consensus_queue.cc:237] T 00000000000000000000000000000000 P 6902b73aabbb49c08b476bb53895ca37 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6902b73aabbb49c08b476bb53895ca37" member_type: VOTER last_known_addr { host: "127.0.222.62" port: 42731 } }
I20260809 12:10:18.075608 4836 sys_catalog.cc:564] T 00000000000000000000000000000000 P 6902b73aabbb49c08b476bb53895ca37 [sys.catalog]: configured and running, proceeding with master startup.
I20260809 12:10:18.077764 4843 sys_catalog.cc:455] T 00000000000000000000000000000000 P 6902b73aabbb49c08b476bb53895ca37 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 6902b73aabbb49c08b476bb53895ca37. Latest consensus state: current_term: 1 leader_uuid: "6902b73aabbb49c08b476bb53895ca37" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6902b73aabbb49c08b476bb53895ca37" member_type: VOTER last_known_addr { host: "127.0.222.62" port: 42731 } } }
I20260809 12:10:18.078229 4843 sys_catalog.cc:458] T 00000000000000000000000000000000 P 6902b73aabbb49c08b476bb53895ca37 [sys.catalog]: This master's current role is: LEADER
I20260809 12:10:18.078883 4842 sys_catalog.cc:455] T 00000000000000000000000000000000 P 6902b73aabbb49c08b476bb53895ca37 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "6902b73aabbb49c08b476bb53895ca37" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6902b73aabbb49c08b476bb53895ca37" member_type: VOTER last_known_addr { host: "127.0.222.62" port: 42731 } } }
I20260809 12:10:18.079207 4842 sys_catalog.cc:458] T 00000000000000000000000000000000 P 6902b73aabbb49c08b476bb53895ca37 [sys.catalog]: This master's current role is: LEADER
I20260809 12:10:18.084044 4852 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260809 12:10:18.089432 4852 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260809 12:10:18.098044 4852 catalog_manager.cc:1347] Generated new cluster ID: 86e3fe988751455894bba674acac384c
I20260809 12:10:18.098145 4852 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260809 12:10:18.120189 4852 catalog_manager.cc:1370] Generated new certificate authority record
I20260809 12:10:18.121467 4852 catalog_manager.cc:1504] Loading token signing keys...
I20260809 12:10:18.128705 4852 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 6902b73aabbb49c08b476bb53895ca37: Generated new TSK 0
I20260809 12:10:18.129443 4852 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.0.222.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.222.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.222.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskgH5Pjm/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786277382073938-888-0/krb5kdc/kudu_127.0.222.1.keytab.
Entry for principal kudu/127.0.222.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskgH5Pjm/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786277382073938-888-0/krb5kdc/kudu_127.0.222.1.keytab.
I20260809 12:10:18.196967 888 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskgH5Pjm/build/asan/bin/kudu
/tmp/dist-test-taskgH5Pjm/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskgH5Pjm/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786277382073938-888-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskgH5Pjm/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786277382073938-888-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskgH5Pjm/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786277382073938-888-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskgH5Pjm/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786277382073938-888-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.222.1:0
--local_ip_for_outbound_sockets=127.0.222.1
--webserver_interface=127.0.222.1
--webserver_port=0
--tserver_master_addrs=127.0.222.62:42731
--keytab_file=/tmp/dist-test-taskgH5Pjm/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786277382073938-888-0/krb5kdc/kudu_127.0.222.1.keytab
--principal=kudu/127.0.222.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.222.20:39623
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskgH5Pjm/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786277382073938-888-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskgH5Pjm/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786277382073938-888-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskgH5Pjm/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786277382073938-888-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260809 12:10:18.536473 4866 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260809 12:10:18.536810 4866 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260809 12:10:18.536974 4866 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260809 12:10:18.544497 4866 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260809 12:10:18.544654 4866 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260809 12:10:18.544826 4866 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.222.1
I20260809 12:10:18.554849 4866 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.222.20:39623
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskgH5Pjm/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786277382073938-888-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskgH5Pjm/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786277382073938-888-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskgH5Pjm/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786277382073938-888-0/krb5kdc/kudu_127.0.222.1.keytab
--principal=kudu/127.0.222.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.222.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskgH5Pjm/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786277382073938-888-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.222.1
--webserver_port=0
--tserver_master_addrs=127.0.222.62:42731
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.222.1
--log_dir=/tmp/dist-test-taskgH5Pjm/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786277382073938-888-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 09 Aug 2026 11:06:41 UTC on ccycloud.quasar-bwoogy.root.comops.site
build id 4575991
ASAN enabled
I20260809 12:10:18.556210 4866 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260809 12:10:18.557538 4866 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260809 12:10:18.571564 4871 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 12:10:18.571539 4872 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 12:10:18.571641 4874 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260809 12:10:18.572707 4866 server_base.cc:1034] running on GCE node
I20260809 12:10:18.573312 4866 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260809 12:10:18.574218 4866 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260809 12:10:18.575414 4866 hybrid_clock.cc:648] HybridClock initialized: now 1786277418575347 us; error 54 us; skew 500 ppm
I20260809 12:10:18.575748 4866 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 09 12:10:18 dist-test-slave-2w3w krb5kdc[4749](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786277418, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.222.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260809 12:10:18.579161 4866 init.cc:377] Logged in from keytab as kudu/127.0.222.1@KRBTEST.COM (short username kudu)
I20260809 12:10:18.580719 4866 webserver.cc:460] Webserver started at http://127.0.222.1:38969/ using document root <none> and password file <none>
I20260809 12:10:18.581429 4866 fs_manager.cc:359] Metadata directory not provided
I20260809 12:10:18.581568 4866 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260809 12:10:18.581929 4866 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260809 12:10:18.585197 4866 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskgH5Pjm/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786277382073938-888-0/minicluster-data/ts-0/data/instance:
uuid: "67e055bad3744bff9353da43b76d32c0"
format_stamp: "Formatted at 2026-08-09 12:10:18 on dist-test-slave-2w3w"
server_key: "ffd62aed1b4d3e7d612e71ed9ea19712"
server_key_iv: "767df3b6d991fef117a540e497bdcd33"
server_key_version: "encryptionkey@0"
I20260809 12:10:18.586079 4866 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskgH5Pjm/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786277382073938-888-0/minicluster-data/ts-0/wal/instance:
uuid: "67e055bad3744bff9353da43b76d32c0"
format_stamp: "Formatted at 2026-08-09 12:10:18 on dist-test-slave-2w3w"
server_key: "ffd62aed1b4d3e7d612e71ed9ea19712"
server_key_iv: "767df3b6d991fef117a540e497bdcd33"
server_key_version: "encryptionkey@0"
I20260809 12:10:18.591650 4866 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.007s sys 0.000s
I20260809 12:10:18.595152 4881 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260809 12:10:18.596480 4866 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260809 12:10:18.596663 4866 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskgH5Pjm/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786277382073938-888-0/minicluster-data/ts-0/data,/tmp/dist-test-taskgH5Pjm/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786277382073938-888-0/minicluster-data/ts-0/wal
uuid: "67e055bad3744bff9353da43b76d32c0"
format_stamp: "Formatted at 2026-08-09 12:10:18 on dist-test-slave-2w3w"
server_key: "ffd62aed1b4d3e7d612e71ed9ea19712"
server_key_iv: "767df3b6d991fef117a540e497bdcd33"
server_key_version: "encryptionkey@0"
I20260809 12:10:18.596916 4866 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskgH5Pjm/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786277382073938-888-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskgH5Pjm/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786277382073938-888-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskgH5Pjm/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786277382073938-888-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260809 12:10:18.613103 4866 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260809 12:10:18.617019 4866 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260809 12:10:18.617350 4866 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260809 12:10:18.618351 4866 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260809 12:10:18.619745 4866 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260809 12:10:18.619855 4866 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260809 12:10:18.620014 4866 ts_tablet_manager.cc:610] Registered 0 tablets
I20260809 12:10:18.620090 4866 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260809 12:10:18.655269 4866 rpc_server.cc:307] RPC server started. Bound to: 127.0.222.1:34295
I20260809 12:10:18.655309 5006 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.222.1:34295 every 8 connection(s)
I20260809 12:10:18.656600 4866 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskgH5Pjm/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786277382073938-888-0/minicluster-data/ts-0/data/info.pb
I20260809 12:10:18.663232 888 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskgH5Pjm/build/asan/bin/kudu as pid 4866
I20260809 12:10:18.663472 888 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskgH5Pjm/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786277382073938-888-0/minicluster-data/ts-0/wal/instance
I20260809 12:10:18.664121 888 external_mini_cluster.cc:1428] Setting key d5fc00c7316714574b045bc7b48bbd38
Aug 09 12:10:18 dist-test-slave-2w3w krb5kdc[4749](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786277418, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.222.1@KRBTEST.COM for kudu/127.0.222.62@KRBTEST.COM
I20260809 12:10:18.676959 5010 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 12:10:18.659258 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.222.1:45665 (local address 127.0.222.62:42731)
0809 12:10:18.659702 (+ 444us) server_negotiation.cc:206] Beginning negotiation
0809 12:10:18.659707 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0809 12:10:18.660655 (+ 948us) server_negotiation.cc:407] Connection header received
0809 12:10:18.662478 (+ 1823us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 12:10:18.662484 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 12:10:18.662580 (+ 96us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 12:10:18.662708 (+ 128us) server_negotiation.cc:226] Negotiated authn=SASL
0809 12:10:18.664438 (+ 1730us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 12:10:18.665184 (+ 746us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 12:10:18.666780 (+ 1596us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 12:10:18.667256 (+ 476us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 12:10:18.667401 (+ 145us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 12:10:18.670952 (+ 3551us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 12:10:18.671005 (+ 53us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 12:10:18.671010 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0809 12:10:18.671054 (+ 44us) server_negotiation.cc:922] calling sasl_server_start()
0809 12:10:18.673562 (+ 2508us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 12:10:18.674408 (+ 846us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 12:10:18.674415 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 12:10:18.674422 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0809 12:10:18.674558 (+ 136us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 12:10:18.675036 (+ 478us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 12:10:18.675043 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 12:10:18.675046 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0809 12:10:18.675327 (+ 281us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 12:10:18.675493 (+ 166us) server_negotiation.cc:1035] Waiting for connection context
0809 12:10:18.676316 (+ 823us) server_negotiation.cc:299] Negotiation successful
0809 12:10:18.676554 (+ 238us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":336,"thread_start_us":220,"threads_started":1}
I20260809 12:10:18.677946 5012 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 12:10:18.659772 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.222.62:42731 (local address 127.0.222.1:45665)
0809 12:10:18.660487 (+ 715us) negotiation.cc:107] Waiting for socket to connect
0809 12:10:18.660525 (+ 38us) client_negotiation.cc:174] Beginning negotiation
0809 12:10:18.662208 (+ 1683us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 12:10:18.663025 (+ 817us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 12:10:18.663038 (+ 13us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 12:10:18.663663 (+ 625us) client_negotiation.cc:189] Negotiated authn=SASL
0809 12:10:18.664251 (+ 588us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 12:10:18.664276 (+ 25us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 12:10:18.665382 (+ 1106us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 12:10:18.665387 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 12:10:18.666622 (+ 1235us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 12:10:18.666634 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 12:10:18.667513 (+ 879us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 12:10:18.667521 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 12:10:18.667757 (+ 236us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 12:10:18.668459 (+ 702us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0809 12:10:18.668498 (+ 39us) client_negotiation.cc:656] calling sasl_client_start()
0809 12:10:18.670721 (+ 2223us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0809 12:10:18.673708 (+ 2987us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0809 12:10:18.673718 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0809 12:10:18.673736 (+ 18us) client_negotiation.cc:760] calling sasl_client_step()
0809 12:10:18.674283 (+ 547us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0809 12:10:18.674678 (+ 395us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0809 12:10:18.674684 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0809 12:10:18.674687 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0809 12:10:18.674895 (+ 208us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0809 12:10:18.675466 (+ 571us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0809 12:10:18.675473 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0809 12:10:18.675914 (+ 441us) client_negotiation.cc:769] Sending connection context
0809 12:10:18.676191 (+ 277us) client_negotiation.cc:240] Negotiation successful
0809 12:10:18.676759 (+ 568us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":487,"thread_start_us":288,"threads_started":1}
I20260809 12:10:18.679646 5007 heartbeater.cc:347] Connected to a master server at 127.0.222.62:42731
I20260809 12:10:18.679970 5007 heartbeater.cc:464] Registering TS with master...
I20260809 12:10:18.680716 5007 heartbeater.cc:511] Master 127.0.222.62:42731 requested a full tablet report, sending...
I20260809 12:10:18.683240 4800 ts_manager.cc:194] Registered new tserver with Master: 67e055bad3744bff9353da43b76d32c0 (127.0.222.1:34295)
I20260809 12:10:18.685664 4800 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.222.1@KRBTEST.COM'} at 127.0.222.1:45665
WARNING: no policy specified for kudu/127.0.222.2@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.222.2@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.222.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskgH5Pjm/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786277382073938-888-0/krb5kdc/kudu_127.0.222.2.keytab.
Entry for principal kudu/127.0.222.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskgH5Pjm/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786277382073938-888-0/krb5kdc/kudu_127.0.222.2.keytab.
I20260809 12:10:18.719271 888 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskgH5Pjm/build/asan/bin/kudu
/tmp/dist-test-taskgH5Pjm/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskgH5Pjm/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786277382073938-888-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskgH5Pjm/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786277382073938-888-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskgH5Pjm/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786277382073938-888-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskgH5Pjm/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786277382073938-888-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.222.2:0
--local_ip_for_outbound_sockets=127.0.222.2
--webserver_interface=127.0.222.2
--webserver_port=0
--tserver_master_addrs=127.0.222.62:42731
--keytab_file=/tmp/dist-test-taskgH5Pjm/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786277382073938-888-0/krb5kdc/kudu_127.0.222.2.keytab
--principal=kudu/127.0.222.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.222.20:39623
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskgH5Pjm/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786277382073938-888-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskgH5Pjm/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786277382073938-888-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskgH5Pjm/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786277382073938-888-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260809 12:10:19.052066 5015 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260809 12:10:19.052497 5015 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260809 12:10:19.052675 5015 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260809 12:10:19.062870 5015 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260809 12:10:19.063027 5015 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260809 12:10:19.063218 5015 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.222.2
I20260809 12:10:19.073163 5015 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.222.20:39623
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskgH5Pjm/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786277382073938-888-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskgH5Pjm/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786277382073938-888-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskgH5Pjm/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786277382073938-888-0/krb5kdc/kudu_127.0.222.2.keytab
--principal=kudu/127.0.222.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.222.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskgH5Pjm/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786277382073938-888-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.222.2
--webserver_port=0
--tserver_master_addrs=127.0.222.62:42731
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.222.2
--log_dir=/tmp/dist-test-taskgH5Pjm/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786277382073938-888-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 09 Aug 2026 11:06:41 UTC on ccycloud.quasar-bwoogy.root.comops.site
build id 4575991
ASAN enabled
I20260809 12:10:19.074469 5015 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260809 12:10:19.076001 5015 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260809 12:10:19.089089 5020 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 12:10:19.089036 5021 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 12:10:19.088997 5023 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260809 12:10:19.090191 5015 server_base.cc:1034] running on GCE node
I20260809 12:10:19.090809 5015 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260809 12:10:19.091789 5015 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260809 12:10:19.093025 5015 hybrid_clock.cc:648] HybridClock initialized: now 1786277419092974 us; error 45 us; skew 500 ppm
I20260809 12:10:19.093343 5015 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 09 12:10:19 dist-test-slave-2w3w krb5kdc[4749](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786277419, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.222.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260809 12:10:19.096531 5015 init.cc:377] Logged in from keytab as kudu/127.0.222.2@KRBTEST.COM (short username kudu)
I20260809 12:10:19.098007 5015 webserver.cc:460] Webserver started at http://127.0.222.2:40173/ using document root <none> and password file <none>
I20260809 12:10:19.098701 5015 fs_manager.cc:359] Metadata directory not provided
I20260809 12:10:19.098853 5015 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260809 12:10:19.099221 5015 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260809 12:10:19.102299 5015 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskgH5Pjm/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786277382073938-888-0/minicluster-data/ts-1/data/instance:
uuid: "5cb013a4453346fa9db87e47fa28dd5d"
format_stamp: "Formatted at 2026-08-09 12:10:19 on dist-test-slave-2w3w"
server_key: "b31bba2872b3a64d53559a63efbe0952"
server_key_iv: "8b9a11f0381a22b511b1252c9aa6cdd5"
server_key_version: "encryptionkey@0"
I20260809 12:10:19.103335 5015 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskgH5Pjm/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786277382073938-888-0/minicluster-data/ts-1/wal/instance:
uuid: "5cb013a4453346fa9db87e47fa28dd5d"
format_stamp: "Formatted at 2026-08-09 12:10:19 on dist-test-slave-2w3w"
server_key: "b31bba2872b3a64d53559a63efbe0952"
server_key_iv: "8b9a11f0381a22b511b1252c9aa6cdd5"
server_key_version: "encryptionkey@0"
I20260809 12:10:19.108608 5015 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.000s
I20260809 12:10:19.112007 5030 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260809 12:10:19.113399 5015 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.001s
I20260809 12:10:19.113534 5015 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskgH5Pjm/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786277382073938-888-0/minicluster-data/ts-1/data,/tmp/dist-test-taskgH5Pjm/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786277382073938-888-0/minicluster-data/ts-1/wal
uuid: "5cb013a4453346fa9db87e47fa28dd5d"
format_stamp: "Formatted at 2026-08-09 12:10:19 on dist-test-slave-2w3w"
server_key: "b31bba2872b3a64d53559a63efbe0952"
server_key_iv: "8b9a11f0381a22b511b1252c9aa6cdd5"
server_key_version: "encryptionkey@0"
I20260809 12:10:19.113704 5015 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskgH5Pjm/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786277382073938-888-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskgH5Pjm/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786277382073938-888-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskgH5Pjm/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786277382073938-888-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260809 12:10:19.125550 5015 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260809 12:10:19.129462 5015 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260809 12:10:19.129702 5015 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260809 12:10:19.130571 5015 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260809 12:10:19.132037 5015 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260809 12:10:19.132143 5015 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260809 12:10:19.132292 5015 ts_tablet_manager.cc:610] Registered 0 tablets
I20260809 12:10:19.132365 5015 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260809 12:10:19.168326 5015 rpc_server.cc:307] RPC server started. Bound to: 127.0.222.2:42671
I20260809 12:10:19.168375 5153 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.222.2:42671 every 8 connection(s)
I20260809 12:10:19.169672 5015 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskgH5Pjm/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786277382073938-888-0/minicluster-data/ts-1/data/info.pb
I20260809 12:10:19.177034 888 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskgH5Pjm/build/asan/bin/kudu as pid 5015
I20260809 12:10:19.177274 888 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskgH5Pjm/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786277382073938-888-0/minicluster-data/ts-1/wal/instance
I20260809 12:10:19.177718 888 external_mini_cluster.cc:1428] Setting key 9931900258998c67797fb049c5942378
Aug 09 12:10:19 dist-test-slave-2w3w krb5kdc[4749](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786277419, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.222.2@KRBTEST.COM for kudu/127.0.222.62@KRBTEST.COM
I20260809 12:10:19.193364 5010 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 12:10:19.172388 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.222.2:34143 (local address 127.0.222.62:42731)
0809 12:10:19.172584 (+ 196us) server_negotiation.cc:206] Beginning negotiation
0809 12:10:19.172589 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0809 12:10:19.173779 (+ 1190us) server_negotiation.cc:407] Connection header received
0809 12:10:19.175781 (+ 2002us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 12:10:19.175786 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 12:10:19.175882 (+ 96us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 12:10:19.176004 (+ 122us) server_negotiation.cc:226] Negotiated authn=SASL
0809 12:10:19.177828 (+ 1824us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 12:10:19.178749 (+ 921us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 12:10:19.181218 (+ 2469us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 12:10:19.181708 (+ 490us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 12:10:19.181810 (+ 102us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 12:10:19.186338 (+ 4528us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 12:10:19.186347 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 12:10:19.186355 (+ 8us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0809 12:10:19.186405 (+ 50us) server_negotiation.cc:922] calling sasl_server_start()
0809 12:10:19.189716 (+ 3311us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 12:10:19.190798 (+ 1082us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 12:10:19.190802 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 12:10:19.190804 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0809 12:10:19.190940 (+ 136us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 12:10:19.191565 (+ 625us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 12:10:19.191569 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 12:10:19.191571 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0809 12:10:19.191869 (+ 298us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 12:10:19.191989 (+ 120us) server_negotiation.cc:1035] Waiting for connection context
0809 12:10:19.192920 (+ 931us) server_negotiation.cc:299] Negotiation successful
0809 12:10:19.193101 (+ 181us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":62}
I20260809 12:10:19.195037 5157 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 12:10:19.172802 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.222.62:42731 (local address 127.0.222.2:34143)
0809 12:10:19.173553 (+ 751us) negotiation.cc:107] Waiting for socket to connect
0809 12:10:19.173637 (+ 84us) client_negotiation.cc:174] Beginning negotiation
0809 12:10:19.175454 (+ 1817us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 12:10:19.176256 (+ 802us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 12:10:19.176265 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 12:10:19.176923 (+ 658us) client_negotiation.cc:189] Negotiated authn=SASL
0809 12:10:19.177647 (+ 724us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 12:10:19.177665 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 12:10:19.178948 (+ 1283us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 12:10:19.179001 (+ 53us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 12:10:19.181042 (+ 2041us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 12:10:19.181076 (+ 34us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 12:10:19.181967 (+ 891us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 12:10:19.181977 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 12:10:19.182324 (+ 347us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 12:10:19.183136 (+ 812us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0809 12:10:19.183171 (+ 35us) client_negotiation.cc:656] calling sasl_client_start()
0809 12:10:19.186089 (+ 2918us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0809 12:10:19.189886 (+ 3797us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0809 12:10:19.189898 (+ 12us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0809 12:10:19.189920 (+ 22us) client_negotiation.cc:760] calling sasl_client_step()
0809 12:10:19.190625 (+ 705us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0809 12:10:19.191120 (+ 495us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0809 12:10:19.191127 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0809 12:10:19.191134 (+ 7us) client_negotiation.cc:760] calling sasl_client_step()
0809 12:10:19.191394 (+ 260us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0809 12:10:19.192014 (+ 620us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0809 12:10:19.192024 (+ 10us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0809 12:10:19.192500 (+ 476us) client_negotiation.cc:769] Sending connection context
0809 12:10:19.192818 (+ 318us) client_negotiation.cc:240] Negotiation successful
0809 12:10:19.193476 (+ 658us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":499,"thread_start_us":290,"threads_started":1}
I20260809 12:10:19.196621 5155 heartbeater.cc:347] Connected to a master server at 127.0.222.62:42731
I20260809 12:10:19.196964 5155 heartbeater.cc:464] Registering TS with master...
I20260809 12:10:19.197690 5155 heartbeater.cc:511] Master 127.0.222.62:42731 requested a full tablet report, sending...
I20260809 12:10:19.199411 4800 ts_manager.cc:194] Registered new tserver with Master: 5cb013a4453346fa9db87e47fa28dd5d (127.0.222.2:42671)
I20260809 12:10:19.200224 4800 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.222.2@KRBTEST.COM'} at 127.0.222.2:34143
WARNING: no policy specified for kudu/127.0.222.3@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.222.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.222.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskgH5Pjm/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786277382073938-888-0/krb5kdc/kudu_127.0.222.3.keytab.
Entry for principal kudu/127.0.222.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskgH5Pjm/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786277382073938-888-0/krb5kdc/kudu_127.0.222.3.keytab.
I20260809 12:10:19.234457 888 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskgH5Pjm/build/asan/bin/kudu
/tmp/dist-test-taskgH5Pjm/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskgH5Pjm/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786277382073938-888-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskgH5Pjm/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786277382073938-888-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskgH5Pjm/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786277382073938-888-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskgH5Pjm/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786277382073938-888-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.222.3:0
--local_ip_for_outbound_sockets=127.0.222.3
--webserver_interface=127.0.222.3
--webserver_port=0
--tserver_master_addrs=127.0.222.62:42731
--keytab_file=/tmp/dist-test-taskgH5Pjm/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786277382073938-888-0/krb5kdc/kudu_127.0.222.3.keytab
--principal=kudu/127.0.222.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.222.20:39623
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskgH5Pjm/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786277382073938-888-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskgH5Pjm/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786277382073938-888-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskgH5Pjm/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786277382073938-888-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260809 12:10:19.562115 5161 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260809 12:10:19.562412 5161 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260809 12:10:19.562506 5161 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260809 12:10:19.569782 5161 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260809 12:10:19.569900 5161 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260809 12:10:19.570053 5161 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.222.3
I20260809 12:10:19.579869 5161 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.222.20:39623
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskgH5Pjm/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786277382073938-888-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskgH5Pjm/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786277382073938-888-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskgH5Pjm/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786277382073938-888-0/krb5kdc/kudu_127.0.222.3.keytab
--principal=kudu/127.0.222.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.222.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskgH5Pjm/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786277382073938-888-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.222.3
--webserver_port=0
--tserver_master_addrs=127.0.222.62:42731
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.222.3
--log_dir=/tmp/dist-test-taskgH5Pjm/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786277382073938-888-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 09 Aug 2026 11:06:41 UTC on ccycloud.quasar-bwoogy.root.comops.site
build id 4575991
ASAN enabled
I20260809 12:10:19.581166 5161 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260809 12:10:19.582532 5161 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260809 12:10:19.596606 5169 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 12:10:19.596675 5166 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 12:10:19.596647 5167 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260809 12:10:19.597456 5161 server_base.cc:1034] running on GCE node
I20260809 12:10:19.598057 5161 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260809 12:10:19.599056 5161 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260809 12:10:19.600287 5161 hybrid_clock.cc:648] HybridClock initialized: now 1786277419600239 us; error 46 us; skew 500 ppm
I20260809 12:10:19.600698 5161 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 09 12:10:19 dist-test-slave-2w3w krb5kdc[4749](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786277419, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.222.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260809 12:10:19.604061 5161 init.cc:377] Logged in from keytab as kudu/127.0.222.3@KRBTEST.COM (short username kudu)
I20260809 12:10:19.605471 5161 webserver.cc:460] Webserver started at http://127.0.222.3:41853/ using document root <none> and password file <none>
I20260809 12:10:19.606169 5161 fs_manager.cc:359] Metadata directory not provided
I20260809 12:10:19.606320 5161 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260809 12:10:19.606643 5161 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260809 12:10:19.609650 5161 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskgH5Pjm/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786277382073938-888-0/minicluster-data/ts-2/data/instance:
uuid: "6ef2a686cdce4591875f5f5b56d46fb9"
format_stamp: "Formatted at 2026-08-09 12:10:19 on dist-test-slave-2w3w"
server_key: "0bbbd0d2c37eb0141ac334aaa82446f5"
server_key_iv: "029102d2f8575cc0b2a702b97450896e"
server_key_version: "encryptionkey@0"
I20260809 12:10:19.610471 5161 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskgH5Pjm/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786277382073938-888-0/minicluster-data/ts-2/wal/instance:
uuid: "6ef2a686cdce4591875f5f5b56d46fb9"
format_stamp: "Formatted at 2026-08-09 12:10:19 on dist-test-slave-2w3w"
server_key: "0bbbd0d2c37eb0141ac334aaa82446f5"
server_key_iv: "029102d2f8575cc0b2a702b97450896e"
server_key_version: "encryptionkey@0"
I20260809 12:10:19.615603 5161 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.008s sys 0.000s
I20260809 12:10:19.619406 5176 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260809 12:10:19.620707 5161 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260809 12:10:19.620893 5161 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskgH5Pjm/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786277382073938-888-0/minicluster-data/ts-2/data,/tmp/dist-test-taskgH5Pjm/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786277382073938-888-0/minicluster-data/ts-2/wal
uuid: "6ef2a686cdce4591875f5f5b56d46fb9"
format_stamp: "Formatted at 2026-08-09 12:10:19 on dist-test-slave-2w3w"
server_key: "0bbbd0d2c37eb0141ac334aaa82446f5"
server_key_iv: "029102d2f8575cc0b2a702b97450896e"
server_key_version: "encryptionkey@0"
I20260809 12:10:19.621106 5161 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskgH5Pjm/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786277382073938-888-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskgH5Pjm/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786277382073938-888-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskgH5Pjm/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786277382073938-888-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260809 12:10:19.633893 5161 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260809 12:10:19.638355 5161 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260809 12:10:19.638669 5161 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260809 12:10:19.639698 5161 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260809 12:10:19.641000 5161 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260809 12:10:19.641106 5161 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260809 12:10:19.641252 5161 ts_tablet_manager.cc:610] Registered 0 tablets
I20260809 12:10:19.641330 5161 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260809 12:10:19.675423 5161 rpc_server.cc:307] RPC server started. Bound to: 127.0.222.3:45061
I20260809 12:10:19.675515 5295 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.222.3:45061 every 8 connection(s)
I20260809 12:10:19.676944 5161 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskgH5Pjm/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786277382073938-888-0/minicluster-data/ts-2/data/info.pb
I20260809 12:10:19.686909 888 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskgH5Pjm/build/asan/bin/kudu as pid 5161
I20260809 12:10:19.687179 888 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskgH5Pjm/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786277382073938-888-0/minicluster-data/ts-2/wal/instance
I20260809 12:10:19.687636 888 external_mini_cluster.cc:1428] Setting key 2191faf8e9549a3e30e91e80820e6cdf
I20260809 12:10:19.688970 5007 heartbeater.cc:503] Master 127.0.222.62:42731 was elected leader, sending a full tablet report...
Aug 09 12:10:19 dist-test-slave-2w3w krb5kdc[4749](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786277419, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.222.3@KRBTEST.COM for kudu/127.0.222.62@KRBTEST.COM
I20260809 12:10:19.698097 5010 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 12:10:19.679881 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.222.3:36617 (local address 127.0.222.62:42731)
0809 12:10:19.680075 (+ 194us) server_negotiation.cc:206] Beginning negotiation
0809 12:10:19.680080 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0809 12:10:19.681276 (+ 1196us) server_negotiation.cc:407] Connection header received
0809 12:10:19.683184 (+ 1908us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 12:10:19.683190 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 12:10:19.683320 (+ 130us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 12:10:19.683436 (+ 116us) server_negotiation.cc:226] Negotiated authn=SASL
0809 12:10:19.685346 (+ 1910us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 12:10:19.686133 (+ 787us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 12:10:19.687976 (+ 1843us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 12:10:19.688574 (+ 598us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 12:10:19.688720 (+ 146us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 12:10:19.692707 (+ 3987us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 12:10:19.692712 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 12:10:19.692715 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0809 12:10:19.692748 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0809 12:10:19.695303 (+ 2555us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 12:10:19.696022 (+ 719us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 12:10:19.696028 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 12:10:19.696030 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0809 12:10:19.696208 (+ 178us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 12:10:19.696638 (+ 430us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 12:10:19.696642 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 12:10:19.696644 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0809 12:10:19.696913 (+ 269us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 12:10:19.697026 (+ 113us) server_negotiation.cc:1035] Waiting for connection context
0809 12:10:19.697677 (+ 651us) server_negotiation.cc:299] Negotiation successful
0809 12:10:19.697842 (+ 165us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":69}
I20260809 12:10:19.699263 5298 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 12:10:19.680342 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.222.62:42731 (local address 127.0.222.3:36617)
0809 12:10:19.681098 (+ 756us) negotiation.cc:107] Waiting for socket to connect
0809 12:10:19.681146 (+ 48us) client_negotiation.cc:174] Beginning negotiation
0809 12:10:19.682838 (+ 1692us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 12:10:19.683692 (+ 854us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 12:10:19.683701 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 12:10:19.684497 (+ 796us) client_negotiation.cc:189] Negotiated authn=SASL
0809 12:10:19.685169 (+ 672us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 12:10:19.685189 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 12:10:19.686305 (+ 1116us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 12:10:19.686309 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 12:10:19.687832 (+ 1523us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 12:10:19.687845 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 12:10:19.688912 (+ 1067us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 12:10:19.688921 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 12:10:19.689214 (+ 293us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 12:10:19.690040 (+ 826us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0809 12:10:19.690071 (+ 31us) client_negotiation.cc:656] calling sasl_client_start()
0809 12:10:19.692500 (+ 2429us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0809 12:10:19.695444 (+ 2944us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0809 12:10:19.695451 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0809 12:10:19.695464 (+ 13us) client_negotiation.cc:760] calling sasl_client_step()
0809 12:10:19.695892 (+ 428us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0809 12:10:19.696338 (+ 446us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0809 12:10:19.696342 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0809 12:10:19.696344 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0809 12:10:19.696533 (+ 189us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0809 12:10:19.697018 (+ 485us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0809 12:10:19.697024 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0809 12:10:19.697348 (+ 324us) client_negotiation.cc:769] Sending connection context
0809 12:10:19.697569 (+ 221us) client_negotiation.cc:240] Negotiation successful
0809 12:10:19.697989 (+ 420us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":513,"thread_start_us":311,"threads_started":1}
I20260809 12:10:19.700798 5296 heartbeater.cc:347] Connected to a master server at 127.0.222.62:42731
I20260809 12:10:19.701189 5296 heartbeater.cc:464] Registering TS with master...
I20260809 12:10:19.702064 5296 heartbeater.cc:511] Master 127.0.222.62:42731 requested a full tablet report, sending...
I20260809 12:10:19.703824 4799 ts_manager.cc:194] Registered new tserver with Master: 6ef2a686cdce4591875f5f5b56d46fb9 (127.0.222.3:45061)
I20260809 12:10:19.704710 4799 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.222.3@KRBTEST.COM'} at 127.0.222.3:36617
I20260809 12:10:19.716174 888 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
Aug 09 12:10:19 dist-test-slave-2w3w krb5kdc[4749](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786277419, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
Aug 09 12:10:19 dist-test-slave-2w3w krb5kdc[4749](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786277419, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.222.1@KRBTEST.COM
I20260809 12:10:19.772889 5303 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 12:10:19.757300 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60380 (local address 127.0.222.1:34295)
0809 12:10:19.757961 (+ 661us) server_negotiation.cc:206] Beginning negotiation
0809 12:10:19.757969 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0809 12:10:19.757996 (+ 27us) server_negotiation.cc:407] Connection header received
0809 12:10:19.758059 (+ 63us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 12:10:19.758066 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 12:10:19.758289 (+ 223us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 12:10:19.758419 (+ 130us) server_negotiation.cc:226] Negotiated authn=SASL
0809 12:10:19.759979 (+ 1560us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 12:10:19.761046 (+ 1067us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 12:10:19.762332 (+ 1286us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 12:10:19.763429 (+ 1097us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 12:10:19.763534 (+ 105us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 12:10:19.767462 (+ 3928us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 12:10:19.767477 (+ 15us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 12:10:19.767501 (+ 24us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0809 12:10:19.767541 (+ 40us) server_negotiation.cc:922] calling sasl_server_start()
0809 12:10:19.770338 (+ 2797us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 12:10:19.770921 (+ 583us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 12:10:19.770928 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 12:10:19.770930 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0809 12:10:19.771125 (+ 195us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 12:10:19.771458 (+ 333us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 12:10:19.771461 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 12:10:19.771463 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0809 12:10:19.771790 (+ 327us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 12:10:19.772022 (+ 232us) server_negotiation.cc:1035] Waiting for connection context
0809 12:10:19.772266 (+ 244us) server_negotiation.cc:299] Negotiation successful
0809 12:10:19.772475 (+ 209us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":473,"thread_start_us":285,"threads_started":1}
Aug 09 12:10:19 dist-test-slave-2w3w krb5kdc[4749](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786277419, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.222.62@KRBTEST.COM
I20260809 12:10:19.791131 5010 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 12:10:19.777362 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49848 (local address 127.0.222.62:42731)
0809 12:10:19.777591 (+ 229us) server_negotiation.cc:206] Beginning negotiation
0809 12:10:19.777596 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0809 12:10:19.777669 (+ 73us) server_negotiation.cc:407] Connection header received
0809 12:10:19.777795 (+ 126us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 12:10:19.777799 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 12:10:19.777859 (+ 60us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 12:10:19.777939 (+ 80us) server_negotiation.cc:226] Negotiated authn=SASL
0809 12:10:19.779153 (+ 1214us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 12:10:19.779883 (+ 730us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 12:10:19.781066 (+ 1183us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 12:10:19.781507 (+ 441us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 12:10:19.781603 (+ 96us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 12:10:19.785831 (+ 4228us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 12:10:19.785836 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 12:10:19.785839 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0809 12:10:19.785873 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0809 12:10:19.788377 (+ 2504us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 12:10:19.789027 (+ 650us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 12:10:19.789034 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 12:10:19.789038 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0809 12:10:19.789184 (+ 146us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 12:10:19.789577 (+ 393us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 12:10:19.789582 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 12:10:19.789585 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0809 12:10:19.789949 (+ 364us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 12:10:19.790221 (+ 272us) server_negotiation.cc:1035] Waiting for connection context
0809 12:10:19.790528 (+ 307us) server_negotiation.cc:299] Negotiation successful
0809 12:10:19.790757 (+ 229us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":79}
W20260809 12:10:19.791697 4799 server_base.cc:1116] Unauthorized access attempt to method kudu.master.MasterService.TSHeartbeat from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:49848
I20260809 12:10:19.792789 888 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskgH5Pjm/build/asan/bin/kudu pid 4866
I20260809 12:10:19.805707 5303 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 12:10:19.793794 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60388 (local address 127.0.222.1:34295)
0809 12:10:19.793963 (+ 169us) server_negotiation.cc:206] Beginning negotiation
0809 12:10:19.793968 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0809 12:10:19.794196 (+ 228us) server_negotiation.cc:407] Connection header received
0809 12:10:19.794322 (+ 126us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 12:10:19.794326 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 12:10:19.794389 (+ 63us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 12:10:19.794469 (+ 80us) server_negotiation.cc:226] Negotiated authn=SASL
0809 12:10:19.795794 (+ 1325us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 12:10:19.796524 (+ 730us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 12:10:19.797662 (+ 1138us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 12:10:19.798114 (+ 452us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 12:10:19.798213 (+ 99us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 12:10:19.800563 (+ 2350us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 12:10:19.800567 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 12:10:19.800571 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0809 12:10:19.800601 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0809 12:10:19.803440 (+ 2839us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 12:10:19.804128 (+ 688us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 12:10:19.804132 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 12:10:19.804134 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0809 12:10:19.804258 (+ 124us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 12:10:19.804579 (+ 321us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 12:10:19.804581 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 12:10:19.804583 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0809 12:10:19.804834 (+ 251us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 12:10:19.805018 (+ 184us) server_negotiation.cc:1035] Waiting for connection context
0809 12:10:19.805248 (+ 230us) server_negotiation.cc:299] Negotiation successful
0809 12:10:19.805413 (+ 165us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":60}
I20260809 12:10:19.806473 5000 generic_service.cc:196] Checking for leaks (request via RPC)
I20260809 12:10:19.890913 888 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskgH5Pjm/build/asan/bin/kudu with pid 4866
I20260809 12:10:19.911705 888 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskgH5Pjm/build/asan/bin/kudu pid 5015
Aug 09 12:10:19 dist-test-slave-2w3w krb5kdc[4749](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786277419, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.222.2@KRBTEST.COM
I20260809 12:10:19.928359 5309 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 12:10:19.912622 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57762 (local address 127.0.222.2:42671)
0809 12:10:19.913196 (+ 574us) server_negotiation.cc:206] Beginning negotiation
0809 12:10:19.913204 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0809 12:10:19.913231 (+ 27us) server_negotiation.cc:407] Connection header received
0809 12:10:19.913295 (+ 64us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 12:10:19.913301 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 12:10:19.913518 (+ 217us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 12:10:19.913647 (+ 129us) server_negotiation.cc:226] Negotiated authn=SASL
0809 12:10:19.914817 (+ 1170us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 12:10:19.915863 (+ 1046us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 12:10:19.917217 (+ 1354us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 12:10:19.917848 (+ 631us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 12:10:19.917945 (+ 97us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 12:10:19.922366 (+ 4421us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 12:10:19.922379 (+ 13us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 12:10:19.922394 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0809 12:10:19.922436 (+ 42us) server_negotiation.cc:922] calling sasl_server_start()
0809 12:10:19.925470 (+ 3034us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 12:10:19.926218 (+ 748us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 12:10:19.926225 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 12:10:19.926228 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0809 12:10:19.926367 (+ 139us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 12:10:19.926766 (+ 399us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 12:10:19.926769 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 12:10:19.926771 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0809 12:10:19.927229 (+ 458us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 12:10:19.927468 (+ 239us) server_negotiation.cc:1035] Waiting for connection context
0809 12:10:19.927752 (+ 284us) server_negotiation.cc:299] Negotiation successful
0809 12:10:19.927950 (+ 198us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":413,"thread_start_us":294,"threads_started":1}
I20260809 12:10:19.929703 5148 generic_service.cc:196] Checking for leaks (request via RPC)
I20260809 12:10:20.014474 888 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskgH5Pjm/build/asan/bin/kudu with pid 5015
I20260809 12:10:20.034191 888 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskgH5Pjm/build/asan/bin/kudu pid 5161
Aug 09 12:10:20 dist-test-slave-2w3w krb5kdc[4749](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786277419, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.222.3@KRBTEST.COM
I20260809 12:10:20.052639 5312 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 12:10:20.035432 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56962 (local address 127.0.222.3:45061)
0809 12:10:20.036161 (+ 729us) server_negotiation.cc:206] Beginning negotiation
0809 12:10:20.036168 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0809 12:10:20.036196 (+ 28us) server_negotiation.cc:407] Connection header received
0809 12:10:20.036258 (+ 62us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 12:10:20.036264 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 12:10:20.036478 (+ 214us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 12:10:20.036609 (+ 131us) server_negotiation.cc:226] Negotiated authn=SASL
0809 12:10:20.038278 (+ 1669us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 12:10:20.039384 (+ 1106us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 12:10:20.041273 (+ 1889us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 12:10:20.041816 (+ 543us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 12:10:20.041936 (+ 120us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 12:10:20.046593 (+ 4657us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 12:10:20.046605 (+ 12us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 12:10:20.046620 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0809 12:10:20.046655 (+ 35us) server_negotiation.cc:922] calling sasl_server_start()
0809 12:10:20.049989 (+ 3334us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 12:10:20.050627 (+ 638us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 12:10:20.050635 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 12:10:20.050637 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0809 12:10:20.050756 (+ 119us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 12:10:20.051162 (+ 406us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 12:10:20.051166 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 12:10:20.051168 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0809 12:10:20.051553 (+ 385us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 12:10:20.051798 (+ 245us) server_negotiation.cc:1035] Waiting for connection context
0809 12:10:20.052039 (+ 241us) server_negotiation.cc:299] Negotiation successful
0809 12:10:20.052240 (+ 201us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":575,"thread_start_us":424,"threads_started":1}
I20260809 12:10:20.053941 5291 generic_service.cc:196] Checking for leaks (request via RPC)
I20260809 12:10:20.139936 888 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskgH5Pjm/build/asan/bin/kudu with pid 5161
I20260809 12:10:20.160856 888 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskgH5Pjm/build/asan/bin/kudu pid 4764
I20260809 12:10:20.162120 4831 generic_service.cc:196] Checking for leaks (request via RPC)
I20260809 12:10:20.268735 888 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskgH5Pjm/build/asan/bin/kudu with pid 4764
2026-08-09T12:10:20Z chronyd exiting
[ OK ] SecurityITest.TestAuthorizedSuperuser (4064 ms)
[ RUN ] SecurityITest.TestDisableWebUI
Loading random data
Initializing database '/tmp/dist-test-taskgH5Pjm/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786277382073938-888-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 09 12:10:20 dist-test-slave-2w3w krb5kdc[5317](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 09 12:10:20 dist-test-slave-2w3w krb5kdc[5317](info): set up 2 sockets
Aug 09 12:10:20 dist-test-slave-2w3w krb5kdc[5317](info): commencing operation
krb5kdc: starting...
W20260809 12:10:21.416906 888 mini_kdc.cc:121] Time spent starting KDC: real 1.101s user 0.000s sys 0.025s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskgH5Pjm/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786277382073938-888-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskgH5Pjm/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786277382073938-888-0/krb5kdc/test-user.keytab.
Aug 09 12:10:21 dist-test-slave-2w3w krb5kdc[5317](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786277421, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-09T12:10:21Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-09T12:10:21Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.222.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.222.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.222.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskgH5Pjm/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786277382073938-888-0/krb5kdc/kudu_127.0.222.62.keytab.
Entry for principal kudu/127.0.222.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskgH5Pjm/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786277382073938-888-0/krb5kdc/kudu_127.0.222.62.keytab.
I20260809 12:10:21.658200 888 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskgH5Pjm/build/asan/bin/kudu
/tmp/dist-test-taskgH5Pjm/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskgH5Pjm/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786277382073938-888-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskgH5Pjm/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786277382073938-888-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskgH5Pjm/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786277382073938-888-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskgH5Pjm/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786277382073938-888-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.222.62:44975
--webserver_interface=127.0.222.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskgH5Pjm/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786277382073938-888-0/krb5kdc/kudu_127.0.222.62.keytab
--principal=kudu/127.0.222.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.222.20:45671
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.222.62:44975
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--webserver_enabled=0 with env {KRB5CCNAME=/tmp/dist-test-taskgH5Pjm/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786277382073938-888-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskgH5Pjm/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786277382073938-888-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskgH5Pjm/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786277382073938-888-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260809 12:10:21.995467 5330 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260809 12:10:21.995761 5330 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260809 12:10:21.995946 5330 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260809 12:10:22.003999 5330 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260809 12:10:22.004117 5330 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260809 12:10:22.004200 5330 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260809 12:10:22.004307 5330 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260809 12:10:22.004398 5330 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260809 12:10:22.014837 5330 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.222.20:45671
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskgH5Pjm/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786277382073938-888-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskgH5Pjm/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786277382073938-888-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.222.62:44975
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskgH5Pjm/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786277382073938-888-0/krb5kdc/kudu_127.0.222.62.keytab
--principal=kudu/127.0.222.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.222.62:44975
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--webserver_enabled=false
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskgH5Pjm/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786277382073938-888-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.222.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskgH5Pjm/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786277382073938-888-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 09 Aug 2026 11:06:41 UTC on ccycloud.quasar-bwoogy.root.comops.site
build id 4575991
ASAN enabled
I20260809 12:10:22.016251 5330 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260809 12:10:22.017588 5330 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260809 12:10:22.029202 5335 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 12:10:22.029332 5336 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 12:10:22.029335 5339 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260809 12:10:22.030563 5330 server_base.cc:1034] running on GCE node
I20260809 12:10:22.031391 5330 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260809 12:10:22.032739 5330 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260809 12:10:22.033942 5330 hybrid_clock.cc:648] HybridClock initialized: now 1786277422033907 us; error 26 us; skew 500 ppm
I20260809 12:10:22.034261 5330 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 09 12:10:22 dist-test-slave-2w3w krb5kdc[5317](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786277422, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.222.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260809 12:10:22.041352 5330 init.cc:377] Logged in from keytab as kudu/127.0.222.62@KRBTEST.COM (short username kudu)
I20260809 12:10:22.042646 5330 fs_manager.cc:359] Metadata directory not provided
I20260809 12:10:22.042766 5330 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260809 12:10:22.043211 5330 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260809 12:10:22.046367 5330 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskgH5Pjm/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786277382073938-888-0/minicluster-data/master-0/data/instance:
uuid: "cc90aedc79944628b63ccf00259c6ede"
format_stamp: "Formatted at 2026-08-09 12:10:22 on dist-test-slave-2w3w"
server_key: "64b212a48dcd4f008bccbf38fd119fd8"
server_key_iv: "bdb602aaedf67dd827eee3352a755a72"
server_key_version: "encryptionkey@0"
I20260809 12:10:22.047277 5330 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskgH5Pjm/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786277382073938-888-0/minicluster-data/master-0/wal/instance:
uuid: "cc90aedc79944628b63ccf00259c6ede"
format_stamp: "Formatted at 2026-08-09 12:10:22 on dist-test-slave-2w3w"
server_key: "64b212a48dcd4f008bccbf38fd119fd8"
server_key_iv: "bdb602aaedf67dd827eee3352a755a72"
server_key_version: "encryptionkey@0"
I20260809 12:10:22.052523 5330 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.004s sys 0.000s
I20260809 12:10:22.056242 5345 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260809 12:10:22.057546 5330 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260809 12:10:22.057734 5330 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskgH5Pjm/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786277382073938-888-0/minicluster-data/master-0/data,/tmp/dist-test-taskgH5Pjm/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786277382073938-888-0/minicluster-data/master-0/wal
uuid: "cc90aedc79944628b63ccf00259c6ede"
format_stamp: "Formatted at 2026-08-09 12:10:22 on dist-test-slave-2w3w"
server_key: "64b212a48dcd4f008bccbf38fd119fd8"
server_key_iv: "bdb602aaedf67dd827eee3352a755a72"
server_key_version: "encryptionkey@0"
I20260809 12:10:22.057976 5330 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskgH5Pjm/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786277382073938-888-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskgH5Pjm/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786277382073938-888-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskgH5Pjm/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786277382073938-888-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260809 12:10:22.092751 5330 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260809 12:10:22.096937 5330 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260809 12:10:22.097256 5330 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260809 12:10:22.116684 5330 rpc_server.cc:307] RPC server started. Bound to: 127.0.222.62:44975
I20260809 12:10:22.116659 5397 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.222.62:44975 every 8 connection(s)
I20260809 12:10:22.117637 5330 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskgH5Pjm/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786277382073938-888-0/minicluster-data/master-0/data/info.pb
I20260809 12:10:22.121407 5398 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260809 12:10:22.122046 888 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskgH5Pjm/build/asan/bin/kudu as pid 5330
I20260809 12:10:22.122284 888 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskgH5Pjm/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786277382073938-888-0/minicluster-data/master-0/wal/instance
I20260809 12:10:22.123138 888 external_mini_cluster.cc:1428] Setting key 4e98388ea7e7652aa1e69512d73bb5f2
I20260809 12:10:22.136281 5398 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P cc90aedc79944628b63ccf00259c6ede: Bootstrap starting.
Aug 09 12:10:22 dist-test-slave-2w3w krb5kdc[5317](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786277421, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.222.62@KRBTEST.COM
I20260809 12:10:22.140331 5398 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P cc90aedc79944628b63ccf00259c6ede: Neither blocks nor log segments found. Creating new log.
I20260809 12:10:22.141710 5398 log.cc:824] T 00000000000000000000000000000000 P cc90aedc79944628b63ccf00259c6ede: Log is configured to *not* fsync() on all Append() calls
I20260809 12:10:22.144784 5398 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P cc90aedc79944628b63ccf00259c6ede: No bootstrap required, opened a new log
I20260809 12:10:22.147531 5401 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 12:10:22.125333 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38668 (local address 127.0.222.62:44975)
0809 12:10:22.126174 (+ 841us) server_negotiation.cc:206] Beginning negotiation
0809 12:10:22.126184 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0809 12:10:22.126230 (+ 46us) server_negotiation.cc:407] Connection header received
0809 12:10:22.128093 (+ 1863us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 12:10:22.128115 (+ 22us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 12:10:22.128493 (+ 378us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 12:10:22.128827 (+ 334us) server_negotiation.cc:226] Negotiated authn=SASL
0809 12:10:22.130717 (+ 1890us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 12:10:22.132077 (+ 1360us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 12:10:22.133949 (+ 1872us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 12:10:22.134617 (+ 668us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 12:10:22.134777 (+ 160us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 12:10:22.139789 (+ 5012us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 12:10:22.139797 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 12:10:22.139813 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0809 12:10:22.139856 (+ 43us) server_negotiation.cc:922] calling sasl_server_start()
0809 12:10:22.143121 (+ 3265us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 12:10:22.143858 (+ 737us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 12:10:22.143865 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 12:10:22.143872 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0809 12:10:22.144058 (+ 186us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 12:10:22.144482 (+ 424us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 12:10:22.144487 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 12:10:22.144492 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0809 12:10:22.145126 (+ 634us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 12:10:22.145478 (+ 352us) server_negotiation.cc:1035] Waiting for connection context
0809 12:10:22.145927 (+ 449us) server_negotiation.cc:299] Negotiation successful
0809 12:10:22.146366 (+ 439us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":614,"thread_start_us":399,"threads_started":1}
I20260809 12:10:22.152966 5398 raft_consensus.cc:348] T 00000000000000000000000000000000 P cc90aedc79944628b63ccf00259c6ede [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "cc90aedc79944628b63ccf00259c6ede" member_type: VOTER last_known_addr { host: "127.0.222.62" port: 44975 } }
I20260809 12:10:22.153400 5398 raft_consensus.cc:374] T 00000000000000000000000000000000 P cc90aedc79944628b63ccf00259c6ede [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260809 12:10:22.153520 5398 raft_consensus.cc:729] T 00000000000000000000000000000000 P cc90aedc79944628b63ccf00259c6ede [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: cc90aedc79944628b63ccf00259c6ede, State: Initialized, Role: FOLLOWER
I20260809 12:10:22.154094 5398 consensus_queue.cc:260] T 00000000000000000000000000000000 P cc90aedc79944628b63ccf00259c6ede [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "cc90aedc79944628b63ccf00259c6ede" member_type: VOTER last_known_addr { host: "127.0.222.62" port: 44975 } }
I20260809 12:10:22.154343 5398 raft_consensus.cc:388] T 00000000000000000000000000000000 P cc90aedc79944628b63ccf00259c6ede [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260809 12:10:22.154510 5398 raft_consensus.cc:482] T 00000000000000000000000000000000 P cc90aedc79944628b63ccf00259c6ede [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260809 12:10:22.154692 5398 raft_consensus.cc:3037] T 00000000000000000000000000000000 P cc90aedc79944628b63ccf00259c6ede [term 0 FOLLOWER]: Advancing to term 1
I20260809 12:10:22.157480 5398 raft_consensus.cc:504] T 00000000000000000000000000000000 P cc90aedc79944628b63ccf00259c6ede [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "cc90aedc79944628b63ccf00259c6ede" member_type: VOTER last_known_addr { host: "127.0.222.62" port: 44975 } }
I20260809 12:10:22.157991 5398 leader_election.cc:302] T 00000000000000000000000000000000 P cc90aedc79944628b63ccf00259c6ede [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: cc90aedc79944628b63ccf00259c6ede; no voters:
I20260809 12:10:22.158594 5398 leader_election.cc:288] T 00000000000000000000000000000000 P cc90aedc79944628b63ccf00259c6ede [CANDIDATE]: Term 1 election: Requested vote from peers
I20260809 12:10:22.158732 5403 raft_consensus.cc:2781] T 00000000000000000000000000000000 P cc90aedc79944628b63ccf00259c6ede [term 1 FOLLOWER]: Leader election won for term 1
I20260809 12:10:22.159489 5403 raft_consensus.cc:686] T 00000000000000000000000000000000 P cc90aedc79944628b63ccf00259c6ede [term 1 LEADER]: Becoming Leader. State: Replica: cc90aedc79944628b63ccf00259c6ede, State: Running, Role: LEADER
I20260809 12:10:22.160028 5403 consensus_queue.cc:237] T 00000000000000000000000000000000 P cc90aedc79944628b63ccf00259c6ede [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "cc90aedc79944628b63ccf00259c6ede" member_type: VOTER last_known_addr { host: "127.0.222.62" port: 44975 } }
I20260809 12:10:22.160849 5398 sys_catalog.cc:564] T 00000000000000000000000000000000 P cc90aedc79944628b63ccf00259c6ede [sys.catalog]: configured and running, proceeding with master startup.
I20260809 12:10:22.162976 5404 sys_catalog.cc:455] T 00000000000000000000000000000000 P cc90aedc79944628b63ccf00259c6ede [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "cc90aedc79944628b63ccf00259c6ede" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "cc90aedc79944628b63ccf00259c6ede" member_type: VOTER last_known_addr { host: "127.0.222.62" port: 44975 } } }
I20260809 12:10:22.163064 5405 sys_catalog.cc:455] T 00000000000000000000000000000000 P cc90aedc79944628b63ccf00259c6ede [sys.catalog]: SysCatalogTable state changed. Reason: New leader cc90aedc79944628b63ccf00259c6ede. Latest consensus state: current_term: 1 leader_uuid: "cc90aedc79944628b63ccf00259c6ede" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "cc90aedc79944628b63ccf00259c6ede" member_type: VOTER last_known_addr { host: "127.0.222.62" port: 44975 } } }
I20260809 12:10:22.163437 5404 sys_catalog.cc:458] T 00000000000000000000000000000000 P cc90aedc79944628b63ccf00259c6ede [sys.catalog]: This master's current role is: LEADER
I20260809 12:10:22.163511 5405 sys_catalog.cc:458] T 00000000000000000000000000000000 P cc90aedc79944628b63ccf00259c6ede [sys.catalog]: This master's current role is: LEADER
I20260809 12:10:22.166024 5410 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260809 12:10:22.171466 5410 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260809 12:10:22.179867 5410 catalog_manager.cc:1347] Generated new cluster ID: b0b805582e0f4e91b90d381c72cf6ca4
I20260809 12:10:22.179986 5410 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260809 12:10:22.210119 5410 catalog_manager.cc:1370] Generated new certificate authority record
I20260809 12:10:22.211629 5410 catalog_manager.cc:1504] Loading token signing keys...
I20260809 12:10:22.217905 5410 catalog_manager.cc:5868] T 00000000000000000000000000000000 P cc90aedc79944628b63ccf00259c6ede: Generated new TSK 0
I20260809 12:10:22.218762 5410 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.0.222.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.222.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.222.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskgH5Pjm/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786277382073938-888-0/krb5kdc/kudu_127.0.222.1.keytab.
Entry for principal kudu/127.0.222.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskgH5Pjm/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786277382073938-888-0/krb5kdc/kudu_127.0.222.1.keytab.
I20260809 12:10:22.274802 888 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskgH5Pjm/build/asan/bin/kudu
/tmp/dist-test-taskgH5Pjm/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskgH5Pjm/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786277382073938-888-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskgH5Pjm/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786277382073938-888-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskgH5Pjm/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786277382073938-888-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskgH5Pjm/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786277382073938-888-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.222.1:0
--local_ip_for_outbound_sockets=127.0.222.1
--webserver_interface=127.0.222.1
--webserver_port=0
--tserver_master_addrs=127.0.222.62:44975
--keytab_file=/tmp/dist-test-taskgH5Pjm/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786277382073938-888-0/krb5kdc/kudu_127.0.222.1.keytab
--principal=kudu/127.0.222.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.222.20:45671
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--webserver_enabled=0 with env {KRB5CCNAME=/tmp/dist-test-taskgH5Pjm/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786277382073938-888-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskgH5Pjm/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786277382073938-888-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskgH5Pjm/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786277382073938-888-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260809 12:10:22.618337 5427 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260809 12:10:22.618675 5427 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260809 12:10:22.618901 5427 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260809 12:10:22.626163 5427 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260809 12:10:22.626325 5427 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260809 12:10:22.626526 5427 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.222.1
I20260809 12:10:22.636413 5427 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.222.20:45671
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskgH5Pjm/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786277382073938-888-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskgH5Pjm/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786277382073938-888-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskgH5Pjm/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786277382073938-888-0/krb5kdc/kudu_127.0.222.1.keytab
--principal=kudu/127.0.222.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.222.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--webserver_enabled=false
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskgH5Pjm/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786277382073938-888-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.222.1
--webserver_port=0
--tserver_master_addrs=127.0.222.62:44975
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.222.1
--log_dir=/tmp/dist-test-taskgH5Pjm/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786277382073938-888-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 09 Aug 2026 11:06:41 UTC on ccycloud.quasar-bwoogy.root.comops.site
build id 4575991
ASAN enabled
I20260809 12:10:22.637835 5427 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260809 12:10:22.639240 5427 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260809 12:10:22.652645 5434 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 12:10:22.652582 5433 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 12:10:22.652685 5437 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260809 12:10:22.653751 5427 server_base.cc:1034] running on GCE node
I20260809 12:10:22.654392 5427 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260809 12:10:22.655426 5427 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260809 12:10:22.656665 5427 hybrid_clock.cc:648] HybridClock initialized: now 1786277422656622 us; error 39 us; skew 500 ppm
I20260809 12:10:22.657006 5427 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 09 12:10:22 dist-test-slave-2w3w krb5kdc[5317](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786277422, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.222.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260809 12:10:22.660279 5427 init.cc:377] Logged in from keytab as kudu/127.0.222.1@KRBTEST.COM (short username kudu)
I20260809 12:10:22.661609 5427 fs_manager.cc:359] Metadata directory not provided
I20260809 12:10:22.661832 5427 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260809 12:10:22.662165 5427 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260809 12:10:22.665349 5427 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskgH5Pjm/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786277382073938-888-0/minicluster-data/ts-0/data/instance:
uuid: "aa783af47cba48afa9f1f262e83568f4"
format_stamp: "Formatted at 2026-08-09 12:10:22 on dist-test-slave-2w3w"
server_key: "2e805eaf1a5e4e28b6120988899261b1"
server_key_iv: "94cff4e41a4882a7787f9068e869aac3"
server_key_version: "encryptionkey@0"
I20260809 12:10:22.666235 5427 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskgH5Pjm/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786277382073938-888-0/minicluster-data/ts-0/wal/instance:
uuid: "aa783af47cba48afa9f1f262e83568f4"
format_stamp: "Formatted at 2026-08-09 12:10:22 on dist-test-slave-2w3w"
server_key: "2e805eaf1a5e4e28b6120988899261b1"
server_key_iv: "94cff4e41a4882a7787f9068e869aac3"
server_key_version: "encryptionkey@0"
I20260809 12:10:22.671741 5427 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.002s sys 0.005s
I20260809 12:10:22.675659 5443 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260809 12:10:22.677073 5427 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260809 12:10:22.677276 5427 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskgH5Pjm/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786277382073938-888-0/minicluster-data/ts-0/data,/tmp/dist-test-taskgH5Pjm/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786277382073938-888-0/minicluster-data/ts-0/wal
uuid: "aa783af47cba48afa9f1f262e83568f4"
format_stamp: "Formatted at 2026-08-09 12:10:22 on dist-test-slave-2w3w"
server_key: "2e805eaf1a5e4e28b6120988899261b1"
server_key_iv: "94cff4e41a4882a7787f9068e869aac3"
server_key_version: "encryptionkey@0"
I20260809 12:10:22.677517 5427 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskgH5Pjm/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786277382073938-888-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskgH5Pjm/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786277382073938-888-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskgH5Pjm/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786277382073938-888-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260809 12:10:22.712903 5427 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260809 12:10:22.717209 5427 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260809 12:10:22.717512 5427 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260809 12:10:22.718559 5427 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260809 12:10:22.720003 5427 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260809 12:10:22.720137 5427 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260809 12:10:22.720288 5427 ts_tablet_manager.cc:610] Registered 0 tablets
I20260809 12:10:22.720384 5427 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260809 12:10:22.757134 5427 rpc_server.cc:307] RPC server started. Bound to: 127.0.222.1:43045
I20260809 12:10:22.757225 5565 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.222.1:43045 every 8 connection(s)
I20260809 12:10:22.758107 5427 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskgH5Pjm/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786277382073938-888-0/minicluster-data/ts-0/data/info.pb
I20260809 12:10:22.760180 888 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskgH5Pjm/build/asan/bin/kudu as pid 5427
I20260809 12:10:22.760454 888 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskgH5Pjm/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786277382073938-888-0/minicluster-data/ts-0/wal/instance
I20260809 12:10:22.761328 888 external_mini_cluster.cc:1428] Setting key 04aa7485307464029c3823a2a3b84b9b
Aug 09 12:10:22 dist-test-slave-2w3w krb5kdc[5317](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786277422, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.222.1@KRBTEST.COM for kudu/127.0.222.62@KRBTEST.COM
I20260809 12:10:22.782094 5569 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 12:10:22.762316 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.222.1:48603 (local address 127.0.222.62:44975)
0809 12:10:22.767148 (+ 4832us) server_negotiation.cc:206] Beginning negotiation
0809 12:10:22.767154 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0809 12:10:22.767178 (+ 24us) server_negotiation.cc:407] Connection header received
0809 12:10:22.767253 (+ 75us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 12:10:22.767258 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 12:10:22.767328 (+ 70us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 12:10:22.767447 (+ 119us) server_negotiation.cc:226] Negotiated authn=SASL
0809 12:10:22.769097 (+ 1650us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 12:10:22.769856 (+ 759us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 12:10:22.771580 (+ 1724us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 12:10:22.772003 (+ 423us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 12:10:22.772143 (+ 140us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 12:10:22.776318 (+ 4175us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 12:10:22.776323 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 12:10:22.776327 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0809 12:10:22.776359 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0809 12:10:22.778883 (+ 2524us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 12:10:22.779667 (+ 784us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 12:10:22.779673 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 12:10:22.779678 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0809 12:10:22.779802 (+ 124us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 12:10:22.780260 (+ 458us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 12:10:22.780264 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 12:10:22.780266 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0809 12:10:22.780527 (+ 261us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 12:10:22.780704 (+ 177us) server_negotiation.cc:1035] Waiting for connection context
0809 12:10:22.781462 (+ 758us) server_negotiation.cc:299] Negotiation successful
0809 12:10:22.781697 (+ 235us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":4686,"thread_start_us":4541,"threads_started":1}
I20260809 12:10:22.783224 5568 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 12:10:22.761666 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.222.62:44975 (local address 127.0.222.1:48603)
0809 12:10:22.762629 (+ 963us) negotiation.cc:107] Waiting for socket to connect
0809 12:10:22.762692 (+ 63us) client_negotiation.cc:174] Beginning negotiation
0809 12:10:22.764689 (+ 1997us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 12:10:22.767697 (+ 3008us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 12:10:22.767706 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 12:10:22.768328 (+ 622us) client_negotiation.cc:189] Negotiated authn=SASL
0809 12:10:22.768908 (+ 580us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 12:10:22.768925 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 12:10:22.770047 (+ 1122us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 12:10:22.770052 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 12:10:22.771417 (+ 1365us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 12:10:22.771431 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 12:10:22.772258 (+ 827us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 12:10:22.772266 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 12:10:22.772494 (+ 228us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 12:10:22.773278 (+ 784us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0809 12:10:22.773318 (+ 40us) client_negotiation.cc:656] calling sasl_client_start()
0809 12:10:22.776083 (+ 2765us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0809 12:10:22.779067 (+ 2984us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0809 12:10:22.779076 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0809 12:10:22.779093 (+ 17us) client_negotiation.cc:760] calling sasl_client_step()
0809 12:10:22.779533 (+ 440us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0809 12:10:22.779919 (+ 386us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0809 12:10:22.779924 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0809 12:10:22.779928 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0809 12:10:22.780123 (+ 195us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0809 12:10:22.780682 (+ 559us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0809 12:10:22.780690 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0809 12:10:22.781065 (+ 375us) client_negotiation.cc:769] Sending connection context
0809 12:10:22.781347 (+ 282us) client_negotiation.cc:240] Negotiation successful
0809 12:10:22.781887 (+ 540us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":647,"thread_start_us":389,"threads_started":1}
I20260809 12:10:22.784924 5566 heartbeater.cc:347] Connected to a master server at 127.0.222.62:44975
I20260809 12:10:22.785251 5566 heartbeater.cc:464] Registering TS with master...
I20260809 12:10:22.785912 5566 heartbeater.cc:511] Master 127.0.222.62:44975 requested a full tablet report, sending...
WARNING: no policy specified for kudu/127.0.222.2@KRBTEST.COM; defaulting to no policy
I20260809 12:10:22.788189 5362 ts_manager.cc:194] Registered new tserver with Master: aa783af47cba48afa9f1f262e83568f4 (127.0.222.1:43045)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.222.2@KRBTEST.COM" created.
I20260809 12:10:22.790117 5362 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.222.1@KRBTEST.COM'} at 127.0.222.1:48603
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.222.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskgH5Pjm/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786277382073938-888-0/krb5kdc/kudu_127.0.222.2.keytab.
Entry for principal kudu/127.0.222.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskgH5Pjm/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786277382073938-888-0/krb5kdc/kudu_127.0.222.2.keytab.
I20260809 12:10:22.818938 888 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskgH5Pjm/build/asan/bin/kudu
/tmp/dist-test-taskgH5Pjm/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskgH5Pjm/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786277382073938-888-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskgH5Pjm/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786277382073938-888-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskgH5Pjm/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786277382073938-888-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskgH5Pjm/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786277382073938-888-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.222.2:0
--local_ip_for_outbound_sockets=127.0.222.2
--webserver_interface=127.0.222.2
--webserver_port=0
--tserver_master_addrs=127.0.222.62:44975
--keytab_file=/tmp/dist-test-taskgH5Pjm/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786277382073938-888-0/krb5kdc/kudu_127.0.222.2.keytab
--principal=kudu/127.0.222.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.222.20:45671
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--webserver_enabled=0 with env {KRB5CCNAME=/tmp/dist-test-taskgH5Pjm/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786277382073938-888-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskgH5Pjm/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786277382073938-888-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskgH5Pjm/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786277382073938-888-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260809 12:10:23.152132 5572 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260809 12:10:23.152405 5572 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260809 12:10:23.152591 5572 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260809 12:10:23.159789 5572 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260809 12:10:23.159900 5572 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260809 12:10:23.160074 5572 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.222.2
I20260809 12:10:23.170156 5572 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.222.20:45671
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskgH5Pjm/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786277382073938-888-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskgH5Pjm/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786277382073938-888-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskgH5Pjm/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786277382073938-888-0/krb5kdc/kudu_127.0.222.2.keytab
--principal=kudu/127.0.222.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.222.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--webserver_enabled=false
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskgH5Pjm/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786277382073938-888-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.222.2
--webserver_port=0
--tserver_master_addrs=127.0.222.62:44975
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.222.2
--log_dir=/tmp/dist-test-taskgH5Pjm/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786277382073938-888-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 09 Aug 2026 11:06:41 UTC on ccycloud.quasar-bwoogy.root.comops.site
build id 4575991
ASAN enabled
I20260809 12:10:23.171628 5572 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260809 12:10:23.172982 5572 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260809 12:10:23.185917 5578 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 12:10:23.185891 5579 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 12:10:23.186123 5581 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260809 12:10:23.187278 5572 server_base.cc:1034] running on GCE node
I20260809 12:10:23.187904 5572 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260809 12:10:23.188966 5572 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260809 12:10:23.190207 5572 hybrid_clock.cc:648] HybridClock initialized: now 1786277423190167 us; error 54 us; skew 500 ppm
I20260809 12:10:23.190593 5572 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 09 12:10:23 dist-test-slave-2w3w krb5kdc[5317](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786277423, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.222.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260809 12:10:23.194279 5572 init.cc:377] Logged in from keytab as kudu/127.0.222.2@KRBTEST.COM (short username kudu)
I20260809 12:10:23.195473 5572 fs_manager.cc:359] Metadata directory not provided
I20260809 12:10:23.195575 5572 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260809 12:10:23.195845 5572 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260809 12:10:23.198752 5572 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskgH5Pjm/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786277382073938-888-0/minicluster-data/ts-1/data/instance:
uuid: "a1175d22014141e1bad56a71341169b8"
format_stamp: "Formatted at 2026-08-09 12:10:23 on dist-test-slave-2w3w"
server_key: "d5fd6285f360bd37cf0761ee02c5722a"
server_key_iv: "f9a446e9a17ae8e8f6f8455fd6854c0b"
server_key_version: "encryptionkey@0"
I20260809 12:10:23.199625 5572 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskgH5Pjm/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786277382073938-888-0/minicluster-data/ts-1/wal/instance:
uuid: "a1175d22014141e1bad56a71341169b8"
format_stamp: "Formatted at 2026-08-09 12:10:23 on dist-test-slave-2w3w"
server_key: "d5fd6285f360bd37cf0761ee02c5722a"
server_key_iv: "f9a446e9a17ae8e8f6f8455fd6854c0b"
server_key_version: "encryptionkey@0"
I20260809 12:10:23.204795 5572 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.000s
I20260809 12:10:23.208289 5587 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260809 12:10:23.209616 5572 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260809 12:10:23.209750 5572 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskgH5Pjm/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786277382073938-888-0/minicluster-data/ts-1/data,/tmp/dist-test-taskgH5Pjm/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786277382073938-888-0/minicluster-data/ts-1/wal
uuid: "a1175d22014141e1bad56a71341169b8"
format_stamp: "Formatted at 2026-08-09 12:10:23 on dist-test-slave-2w3w"
server_key: "d5fd6285f360bd37cf0761ee02c5722a"
server_key_iv: "f9a446e9a17ae8e8f6f8455fd6854c0b"
server_key_version: "encryptionkey@0"
I20260809 12:10:23.209934 5572 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskgH5Pjm/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786277382073938-888-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskgH5Pjm/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786277382073938-888-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskgH5Pjm/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786277382073938-888-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260809 12:10:23.239039 5572 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260809 12:10:23.243378 5572 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260809 12:10:23.243623 5572 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260809 12:10:23.244920 5572 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260809 12:10:23.246884 5572 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260809 12:10:23.247052 5572 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.001s sys 0.000s
I20260809 12:10:23.247190 5572 ts_tablet_manager.cc:610] Registered 0 tablets
I20260809 12:10:23.247277 5572 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260809 12:10:23.280385 5572 rpc_server.cc:307] RPC server started. Bound to: 127.0.222.2:44491
I20260809 12:10:23.280426 5710 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.222.2:44491 every 8 connection(s)
I20260809 12:10:23.281345 5572 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskgH5Pjm/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786277382073938-888-0/minicluster-data/ts-1/data/info.pb
I20260809 12:10:23.282230 888 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskgH5Pjm/build/asan/bin/kudu as pid 5572
I20260809 12:10:23.282472 888 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskgH5Pjm/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786277382073938-888-0/minicluster-data/ts-1/wal/instance
I20260809 12:10:23.283092 888 external_mini_cluster.cc:1428] Setting key ffd748afd94a971de52d4bc428ef5800
Aug 09 12:10:23 dist-test-slave-2w3w krb5kdc[5317](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786277423, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.222.2@KRBTEST.COM for kudu/127.0.222.62@KRBTEST.COM
I20260809 12:10:23.305907 5713 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 12:10:23.284213 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.222.2:56045 (local address 127.0.222.62:44975)
0809 12:10:23.284725 (+ 512us) server_negotiation.cc:206] Beginning negotiation
0809 12:10:23.284731 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0809 12:10:23.286054 (+ 1323us) server_negotiation.cc:407] Connection header received
0809 12:10:23.289081 (+ 3027us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 12:10:23.289085 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 12:10:23.289152 (+ 67us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 12:10:23.289240 (+ 88us) server_negotiation.cc:226] Negotiated authn=SASL
0809 12:10:23.291440 (+ 2200us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 12:10:23.292184 (+ 744us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 12:10:23.294711 (+ 2527us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 12:10:23.295256 (+ 545us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 12:10:23.295400 (+ 144us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 12:10:23.299690 (+ 4290us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 12:10:23.299695 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 12:10:23.299699 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0809 12:10:23.299734 (+ 35us) server_negotiation.cc:922] calling sasl_server_start()
0809 12:10:23.302491 (+ 2757us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 12:10:23.303409 (+ 918us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 12:10:23.303421 (+ 12us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 12:10:23.303424 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0809 12:10:23.303552 (+ 128us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 12:10:23.304103 (+ 551us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 12:10:23.304107 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 12:10:23.304109 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0809 12:10:23.304389 (+ 280us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 12:10:23.304552 (+ 163us) server_negotiation.cc:1035] Waiting for connection context
0809 12:10:23.305263 (+ 711us) server_negotiation.cc:299] Negotiation successful
0809 12:10:23.305510 (+ 247us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":388,"thread_start_us":260,"threads_started":1}
I20260809 12:10:23.306942 5714 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 12:10:23.284953 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.222.62:44975 (local address 127.0.222.2:56045)
0809 12:10:23.285864 (+ 911us) negotiation.cc:107] Waiting for socket to connect
0809 12:10:23.285927 (+ 63us) client_negotiation.cc:174] Beginning negotiation
0809 12:10:23.288753 (+ 2826us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 12:10:23.289517 (+ 764us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 12:10:23.289530 (+ 13us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 12:10:23.290353 (+ 823us) client_negotiation.cc:189] Negotiated authn=SASL
0809 12:10:23.291229 (+ 876us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 12:10:23.291258 (+ 29us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 12:10:23.292355 (+ 1097us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 12:10:23.292363 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 12:10:23.294550 (+ 2187us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 12:10:23.294569 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 12:10:23.295522 (+ 953us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 12:10:23.295530 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 12:10:23.295753 (+ 223us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 12:10:23.296495 (+ 742us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0809 12:10:23.296535 (+ 40us) client_negotiation.cc:656] calling sasl_client_start()
0809 12:10:23.299429 (+ 2894us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0809 12:10:23.302663 (+ 3234us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0809 12:10:23.302672 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0809 12:10:23.302691 (+ 19us) client_negotiation.cc:760] calling sasl_client_step()
0809 12:10:23.303262 (+ 571us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0809 12:10:23.303672 (+ 410us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0809 12:10:23.303677 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0809 12:10:23.303681 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0809 12:10:23.303961 (+ 280us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0809 12:10:23.304526 (+ 565us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0809 12:10:23.304534 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0809 12:10:23.304894 (+ 360us) client_negotiation.cc:769] Sending connection context
0809 12:10:23.305175 (+ 281us) client_negotiation.cc:240] Negotiation successful
0809 12:10:23.305734 (+ 559us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":621,"thread_start_us":377,"threads_started":1}
WARNING: no policy specified for kudu/127.0.222.3@KRBTEST.COM; defaulting to no policy
I20260809 12:10:23.308578 5711 heartbeater.cc:347] Connected to a master server at 127.0.222.62:44975
I20260809 12:10:23.308887 5711 heartbeater.cc:464] Registering TS with master...
I20260809 12:10:23.309553 5711 heartbeater.cc:511] Master 127.0.222.62:44975 requested a full tablet report, sending...
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.222.3@KRBTEST.COM" created.
I20260809 12:10:23.311239 5362 ts_manager.cc:194] Registered new tserver with Master: a1175d22014141e1bad56a71341169b8 (127.0.222.2:44491)
I20260809 12:10:23.312528 5362 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.222.2@KRBTEST.COM'} at 127.0.222.2:56045
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.222.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskgH5Pjm/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786277382073938-888-0/krb5kdc/kudu_127.0.222.3.keytab.
Entry for principal kudu/127.0.222.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskgH5Pjm/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786277382073938-888-0/krb5kdc/kudu_127.0.222.3.keytab.
I20260809 12:10:23.338418 888 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskgH5Pjm/build/asan/bin/kudu
/tmp/dist-test-taskgH5Pjm/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskgH5Pjm/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786277382073938-888-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskgH5Pjm/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786277382073938-888-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskgH5Pjm/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786277382073938-888-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskgH5Pjm/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786277382073938-888-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.222.3:0
--local_ip_for_outbound_sockets=127.0.222.3
--webserver_interface=127.0.222.3
--webserver_port=0
--tserver_master_addrs=127.0.222.62:44975
--keytab_file=/tmp/dist-test-taskgH5Pjm/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786277382073938-888-0/krb5kdc/kudu_127.0.222.3.keytab
--principal=kudu/127.0.222.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.222.20:45671
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--webserver_enabled=0 with env {KRB5CCNAME=/tmp/dist-test-taskgH5Pjm/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786277382073938-888-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskgH5Pjm/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786277382073938-888-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskgH5Pjm/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786277382073938-888-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260809 12:10:23.677134 5718 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260809 12:10:23.677536 5718 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260809 12:10:23.677718 5718 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260809 12:10:23.685289 5718 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260809 12:10:23.685456 5718 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260809 12:10:23.685633 5718 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.222.3
I20260809 12:10:23.695379 5718 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.222.20:45671
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskgH5Pjm/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786277382073938-888-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskgH5Pjm/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786277382073938-888-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskgH5Pjm/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786277382073938-888-0/krb5kdc/kudu_127.0.222.3.keytab
--principal=kudu/127.0.222.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.222.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--webserver_enabled=false
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskgH5Pjm/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786277382073938-888-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.222.3
--webserver_port=0
--tserver_master_addrs=127.0.222.62:44975
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.222.3
--log_dir=/tmp/dist-test-taskgH5Pjm/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786277382073938-888-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 09 Aug 2026 11:06:41 UTC on ccycloud.quasar-bwoogy.root.comops.site
build id 4575991
ASAN enabled
I20260809 12:10:23.696689 5718 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260809 12:10:23.697975 5718 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260809 12:10:23.711256 5726 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 12:10:23.711223 5724 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260809 12:10:23.711757 5718 server_base.cc:1034] running on GCE node
W20260809 12:10:23.711290 5723 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260809 12:10:23.712750 5718 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260809 12:10:23.713850 5718 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260809 12:10:23.715059 5718 hybrid_clock.cc:648] HybridClock initialized: now 1786277423714998 us; error 62 us; skew 500 ppm
I20260809 12:10:23.715380 5718 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 09 12:10:23 dist-test-slave-2w3w krb5kdc[5317](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786277423, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.222.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260809 12:10:23.718580 5718 init.cc:377] Logged in from keytab as kudu/127.0.222.3@KRBTEST.COM (short username kudu)
I20260809 12:10:23.719859 5718 fs_manager.cc:359] Metadata directory not provided
I20260809 12:10:23.719990 5718 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260809 12:10:23.720315 5718 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260809 12:10:23.723337 5718 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskgH5Pjm/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786277382073938-888-0/minicluster-data/ts-2/data/instance:
uuid: "cdc0af2ee8f9477994b1555d954b66d9"
format_stamp: "Formatted at 2026-08-09 12:10:23 on dist-test-slave-2w3w"
server_key: "8a4e0873b914aaa0ba3a40dd3a366877"
server_key_iv: "0cfe422dfcd9ddd9c7cb3e8d8ab8225f"
server_key_version: "encryptionkey@0"
I20260809 12:10:23.724166 5718 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskgH5Pjm/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786277382073938-888-0/minicluster-data/ts-2/wal/instance:
uuid: "cdc0af2ee8f9477994b1555d954b66d9"
format_stamp: "Formatted at 2026-08-09 12:10:23 on dist-test-slave-2w3w"
server_key: "8a4e0873b914aaa0ba3a40dd3a366877"
server_key_iv: "0cfe422dfcd9ddd9c7cb3e8d8ab8225f"
server_key_version: "encryptionkey@0"
I20260809 12:10:23.729295 5718 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.000s
I20260809 12:10:23.732808 5733 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260809 12:10:23.734189 5718 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260809 12:10:23.734380 5718 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskgH5Pjm/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786277382073938-888-0/minicluster-data/ts-2/data,/tmp/dist-test-taskgH5Pjm/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786277382073938-888-0/minicluster-data/ts-2/wal
uuid: "cdc0af2ee8f9477994b1555d954b66d9"
format_stamp: "Formatted at 2026-08-09 12:10:23 on dist-test-slave-2w3w"
server_key: "8a4e0873b914aaa0ba3a40dd3a366877"
server_key_iv: "0cfe422dfcd9ddd9c7cb3e8d8ab8225f"
server_key_version: "encryptionkey@0"
I20260809 12:10:23.734608 5718 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskgH5Pjm/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786277382073938-888-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskgH5Pjm/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786277382073938-888-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskgH5Pjm/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786277382073938-888-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260809 12:10:23.750748 5718 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260809 12:10:23.755172 5718 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260809 12:10:23.755465 5718 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260809 12:10:23.756490 5718 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260809 12:10:23.758256 5718 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260809 12:10:23.758363 5718 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260809 12:10:23.758460 5718 ts_tablet_manager.cc:610] Registered 0 tablets
I20260809 12:10:23.758538 5718 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260809 12:10:23.793356 5566 heartbeater.cc:503] Master 127.0.222.62:44975 was elected leader, sending a full tablet report...
I20260809 12:10:23.796190 5718 rpc_server.cc:307] RPC server started. Bound to: 127.0.222.3:44817
I20260809 12:10:23.796276 5856 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.222.3:44817 every 8 connection(s)
I20260809 12:10:23.797075 5718 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskgH5Pjm/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786277382073938-888-0/minicluster-data/ts-2/data/info.pb
I20260809 12:10:23.802004 888 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskgH5Pjm/build/asan/bin/kudu as pid 5718
I20260809 12:10:23.802215 888 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskgH5Pjm/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786277382073938-888-0/minicluster-data/ts-2/wal/instance
I20260809 12:10:23.802634 888 external_mini_cluster.cc:1428] Setting key a0642259933e808a90106af7101c425d
Aug 09 12:10:23 dist-test-slave-2w3w krb5kdc[5317](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786277423, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.222.3@KRBTEST.COM for kudu/127.0.222.62@KRBTEST.COM
I20260809 12:10:23.817498 5713 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 12:10:23.799473 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.222.3:35829 (local address 127.0.222.62:44975)
0809 12:10:23.799650 (+ 177us) server_negotiation.cc:206] Beginning negotiation
0809 12:10:23.799655 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0809 12:10:23.800887 (+ 1232us) server_negotiation.cc:407] Connection header received
0809 12:10:23.802781 (+ 1894us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 12:10:23.802786 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 12:10:23.802876 (+ 90us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 12:10:23.803101 (+ 225us) server_negotiation.cc:226] Negotiated authn=SASL
0809 12:10:23.804880 (+ 1779us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 12:10:23.805797 (+ 917us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 12:10:23.807391 (+ 1594us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 12:10:23.807837 (+ 446us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 12:10:23.807950 (+ 113us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 12:10:23.811355 (+ 3405us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 12:10:23.811360 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 12:10:23.811363 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0809 12:10:23.811395 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0809 12:10:23.814291 (+ 2896us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 12:10:23.815092 (+ 801us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 12:10:23.815096 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 12:10:23.815098 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0809 12:10:23.815214 (+ 116us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 12:10:23.815583 (+ 369us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 12:10:23.815586 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 12:10:23.815588 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0809 12:10:23.815870 (+ 282us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 12:10:23.816045 (+ 175us) server_negotiation.cc:1035] Waiting for connection context
0809 12:10:23.816943 (+ 898us) server_negotiation.cc:299] Negotiation successful
0809 12:10:23.817159 (+ 216us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":67}
I20260809 12:10:23.818578 5859 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 12:10:23.800026 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.222.62:44975 (local address 127.0.222.3:35829)
0809 12:10:23.800720 (+ 694us) negotiation.cc:107] Waiting for socket to connect
0809 12:10:23.800757 (+ 37us) client_negotiation.cc:174] Beginning negotiation
0809 12:10:23.802510 (+ 1753us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 12:10:23.803322 (+ 812us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 12:10:23.803334 (+ 12us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 12:10:23.803910 (+ 576us) client_negotiation.cc:189] Negotiated authn=SASL
0809 12:10:23.804663 (+ 753us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 12:10:23.804690 (+ 27us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 12:10:23.805949 (+ 1259us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 12:10:23.805953 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 12:10:23.807247 (+ 1294us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 12:10:23.807260 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 12:10:23.808046 (+ 786us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 12:10:23.808051 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 12:10:23.808232 (+ 181us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 12:10:23.808855 (+ 623us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0809 12:10:23.808877 (+ 22us) client_negotiation.cc:656] calling sasl_client_start()
0809 12:10:23.811159 (+ 2282us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0809 12:10:23.814414 (+ 3255us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0809 12:10:23.814420 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0809 12:10:23.814432 (+ 12us) client_negotiation.cc:760] calling sasl_client_step()
0809 12:10:23.814947 (+ 515us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0809 12:10:23.815310 (+ 363us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0809 12:10:23.815313 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0809 12:10:23.815315 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0809 12:10:23.815482 (+ 167us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0809 12:10:23.816036 (+ 554us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0809 12:10:23.816044 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0809 12:10:23.816444 (+ 400us) client_negotiation.cc:769] Sending connection context
0809 12:10:23.816759 (+ 315us) client_negotiation.cc:240] Negotiation successful
0809 12:10:23.817315 (+ 556us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":479,"thread_start_us":293,"threads_started":1}
I20260809 12:10:23.820106 5857 heartbeater.cc:347] Connected to a master server at 127.0.222.62:44975
I20260809 12:10:23.820477 5857 heartbeater.cc:464] Registering TS with master...
I20260809 12:10:23.821234 5857 heartbeater.cc:511] Master 127.0.222.62:44975 requested a full tablet report, sending...
I20260809 12:10:23.822854 5362 ts_manager.cc:194] Registered new tserver with Master: cdc0af2ee8f9477994b1555d954b66d9 (127.0.222.3:44817)
I20260809 12:10:23.823701 5362 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.222.3@KRBTEST.COM'} at 127.0.222.3:35829
I20260809 12:10:23.830482 888 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260809 12:10:23.848426 5713 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 12:10:23.835097 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38680 (local address 127.0.222.62:44975)
0809 12:10:23.835315 (+ 218us) server_negotiation.cc:206] Beginning negotiation
0809 12:10:23.835321 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0809 12:10:23.835987 (+ 666us) server_negotiation.cc:407] Connection header received
0809 12:10:23.836127 (+ 140us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 12:10:23.836131 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 12:10:23.836194 (+ 63us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 12:10:23.836274 (+ 80us) server_negotiation.cc:226] Negotiated authn=SASL
0809 12:10:23.837776 (+ 1502us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 12:10:23.838520 (+ 744us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 12:10:23.839978 (+ 1458us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 12:10:23.840406 (+ 428us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 12:10:23.840501 (+ 95us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 12:10:23.842727 (+ 2226us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 12:10:23.842732 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 12:10:23.842735 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0809 12:10:23.842768 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0809 12:10:23.845948 (+ 3180us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 12:10:23.846565 (+ 617us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 12:10:23.846569 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 12:10:23.846571 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0809 12:10:23.846710 (+ 139us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 12:10:23.847085 (+ 375us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 12:10:23.847089 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 12:10:23.847091 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0809 12:10:23.847412 (+ 321us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 12:10:23.847605 (+ 193us) server_negotiation.cc:1035] Waiting for connection context
0809 12:10:23.847903 (+ 298us) server_negotiation.cc:299] Negotiation successful
0809 12:10:23.848063 (+ 160us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":64}
I20260809 12:10:23.852630 5362 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:38680:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260809 12:10:23.859611 5362 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260809 12:10:23.893893 5871 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 12:10:23.872107 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.222.1:43045 (local address 127.0.0.1:45346)
0809 12:10:23.884993 (+ 12886us) negotiation.cc:107] Waiting for socket to connect
0809 12:10:23.885012 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0809 12:10:23.885211 (+ 199us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 12:10:23.886067 (+ 856us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 12:10:23.886076 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 12:10:23.886109 (+ 33us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0809 12:10:23.886568 (+ 459us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 12:10:23.886582 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 12:10:23.888856 (+ 2274us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 12:10:23.888862 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 12:10:23.891373 (+ 2511us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 12:10:23.891388 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 12:10:23.892758 (+ 1370us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 12:10:23.892765 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 12:10:23.892961 (+ 196us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 12:10:23.893032 (+ 71us) client_negotiation.cc:769] Sending connection context
0809 12:10:23.893182 (+ 150us) client_negotiation.cc:240] Negotiation successful
0809 12:10:23.893365 (+ 183us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":12751,"mutex_wait_us":3,"spinlock_wait_cycles":14592,"thread_start_us":13056,"threads_started":1}
I20260809 12:10:23.894488 5872 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 12:10:23.871964 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.222.2:44491 (local address 127.0.0.1:45948)
0809 12:10:23.884932 (+ 12968us) negotiation.cc:107] Waiting for socket to connect
0809 12:10:23.884982 (+ 50us) client_negotiation.cc:174] Beginning negotiation
0809 12:10:23.885212 (+ 230us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 12:10:23.886880 (+ 1668us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 12:10:23.886883 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 12:10:23.886928 (+ 45us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0809 12:10:23.887551 (+ 623us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 12:10:23.887568 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 12:10:23.889585 (+ 2017us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 12:10:23.889590 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 12:10:23.891516 (+ 1926us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 12:10:23.891534 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 12:10:23.893621 (+ 2087us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 12:10:23.893628 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 12:10:23.893816 (+ 188us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 12:10:23.893853 (+ 37us) client_negotiation.cc:769] Sending connection context
0809 12:10:23.893947 (+ 94us) client_negotiation.cc:240] Negotiation successful
0809 12:10:23.894078 (+ 131us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":12616,"mutex_wait_us":2,"spinlock_wait_cycles":4480,"thread_start_us":12729,"threads_started":1}
I20260809 12:10:23.894591 5873 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 12:10:23.872166 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45948 (local address 127.0.222.2:44491)
0809 12:10:23.884610 (+ 12444us) server_negotiation.cc:206] Beginning negotiation
0809 12:10:23.884621 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0809 12:10:23.885585 (+ 964us) server_negotiation.cc:407] Connection header received
0809 12:10:23.885692 (+ 107us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 12:10:23.885698 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 12:10:23.886093 (+ 395us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 12:10:23.886292 (+ 199us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0809 12:10:23.887755 (+ 1463us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 12:10:23.889247 (+ 1492us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 12:10:23.891689 (+ 2442us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 12:10:23.893118 (+ 1429us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 12:10:23.893243 (+ 125us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 12:10:23.893461 (+ 218us) server_negotiation.cc:1035] Waiting for connection context
0809 12:10:23.894048 (+ 587us) server_negotiation.cc:299] Negotiation successful
0809 12:10:23.894255 (+ 207us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":12236,"thread_start_us":12094,"threads_started":1}
I20260809 12:10:23.895131 5874 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 12:10:23.872378 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45346 (local address 127.0.222.1:43045)
0809 12:10:23.885105 (+ 12727us) server_negotiation.cc:206] Beginning negotiation
0809 12:10:23.885113 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0809 12:10:23.885145 (+ 32us) server_negotiation.cc:407] Connection header received
0809 12:10:23.885511 (+ 366us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 12:10:23.885520 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 12:10:23.885800 (+ 280us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 12:10:23.886787 (+ 987us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0809 12:10:23.887123 (+ 336us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 12:10:23.888646 (+ 1523us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 12:10:23.891552 (+ 2906us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 12:10:23.892539 (+ 987us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 12:10:23.892697 (+ 158us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 12:10:23.892949 (+ 252us) server_negotiation.cc:1035] Waiting for connection context
0809 12:10:23.894545 (+ 1596us) server_negotiation.cc:299] Negotiation successful
0809 12:10:23.894766 (+ 221us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":12538,"thread_start_us":12993,"threads_started":1}
I20260809 12:10:23.896427 5876 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 12:10:23.886773 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.222.3:44817 (local address 127.0.0.1:59876)
0809 12:10:23.887473 (+ 700us) negotiation.cc:107] Waiting for socket to connect
0809 12:10:23.887498 (+ 25us) client_negotiation.cc:174] Beginning negotiation
0809 12:10:23.887639 (+ 141us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 12:10:23.888217 (+ 578us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 12:10:23.888222 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 12:10:23.888247 (+ 25us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0809 12:10:23.888588 (+ 341us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 12:10:23.888602 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 12:10:23.890552 (+ 1950us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 12:10:23.890559 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 12:10:23.893225 (+ 2666us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 12:10:23.893239 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 12:10:23.895607 (+ 2368us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 12:10:23.895613 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 12:10:23.895788 (+ 175us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 12:10:23.895819 (+ 31us) client_negotiation.cc:769] Sending connection context
0809 12:10:23.895936 (+ 117us) client_negotiation.cc:240] Negotiation successful
0809 12:10:23.896067 (+ 131us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":540,"mutex_wait_us":36,"thread_start_us":380,"threads_started":1}
I20260809 12:10:23.896477 5875 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 12:10:23.885985 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59876 (local address 127.0.222.3:44817)
0809 12:10:23.886811 (+ 826us) server_negotiation.cc:206] Beginning negotiation
0809 12:10:23.886817 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0809 12:10:23.887626 (+ 809us) server_negotiation.cc:407] Connection header received
0809 12:10:23.887797 (+ 171us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 12:10:23.887804 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 12:10:23.888013 (+ 209us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 12:10:23.888211 (+ 198us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0809 12:10:23.888781 (+ 570us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 12:10:23.890394 (+ 1613us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 12:10:23.893407 (+ 3013us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 12:10:23.895392 (+ 1985us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 12:10:23.895521 (+ 129us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 12:10:23.895735 (+ 214us) server_negotiation.cc:1035] Waiting for connection context
0809 12:10:23.895985 (+ 250us) server_negotiation.cc:299] Negotiation successful
0809 12:10:23.896147 (+ 162us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":591,"thread_start_us":428,"threads_started":1}
I20260809 12:10:23.898617 5638 tablet_service.cc:1467] Processing CreateTablet for tablet 4256d5c19a3a47ae8bea0721ab33169a (DEFAULT_TABLE table=test-table [id=74677c77ee7d4419bad4e46f6f97ce6f]), partition=RANGE (key) PARTITION UNBOUNDED
I20260809 12:10:23.898864 5496 tablet_service.cc:1467] Processing CreateTablet for tablet 4256d5c19a3a47ae8bea0721ab33169a (DEFAULT_TABLE table=test-table [id=74677c77ee7d4419bad4e46f6f97ce6f]), partition=RANGE (key) PARTITION UNBOUNDED
I20260809 12:10:23.899945 5638 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 4256d5c19a3a47ae8bea0721ab33169a. 1 dirs total, 0 dirs full, 0 dirs failed
I20260809 12:10:23.900247 5496 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 4256d5c19a3a47ae8bea0721ab33169a. 1 dirs total, 0 dirs full, 0 dirs failed
I20260809 12:10:23.900693 5789 tablet_service.cc:1467] Processing CreateTablet for tablet 4256d5c19a3a47ae8bea0721ab33169a (DEFAULT_TABLE table=test-table [id=74677c77ee7d4419bad4e46f6f97ce6f]), partition=RANGE (key) PARTITION UNBOUNDED
I20260809 12:10:23.902071 5789 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 4256d5c19a3a47ae8bea0721ab33169a. 1 dirs total, 0 dirs full, 0 dirs failed
I20260809 12:10:23.912011 5877 tablet_bootstrap.cc:492] T 4256d5c19a3a47ae8bea0721ab33169a P a1175d22014141e1bad56a71341169b8: Bootstrap starting.
I20260809 12:10:23.915282 5878 tablet_bootstrap.cc:492] T 4256d5c19a3a47ae8bea0721ab33169a P aa783af47cba48afa9f1f262e83568f4: Bootstrap starting.
I20260809 12:10:23.916265 5877 tablet_bootstrap.cc:654] T 4256d5c19a3a47ae8bea0721ab33169a P a1175d22014141e1bad56a71341169b8: Neither blocks nor log segments found. Creating new log.
I20260809 12:10:23.917658 5877 log.cc:824] T 4256d5c19a3a47ae8bea0721ab33169a P a1175d22014141e1bad56a71341169b8: Log is configured to *not* fsync() on all Append() calls
I20260809 12:10:23.918442 5879 tablet_bootstrap.cc:492] T 4256d5c19a3a47ae8bea0721ab33169a P cdc0af2ee8f9477994b1555d954b66d9: Bootstrap starting.
I20260809 12:10:23.919402 5878 tablet_bootstrap.cc:654] T 4256d5c19a3a47ae8bea0721ab33169a P aa783af47cba48afa9f1f262e83568f4: Neither blocks nor log segments found. Creating new log.
I20260809 12:10:23.921100 5878 log.cc:824] T 4256d5c19a3a47ae8bea0721ab33169a P aa783af47cba48afa9f1f262e83568f4: Log is configured to *not* fsync() on all Append() calls
I20260809 12:10:23.921505 5879 tablet_bootstrap.cc:654] T 4256d5c19a3a47ae8bea0721ab33169a P cdc0af2ee8f9477994b1555d954b66d9: Neither blocks nor log segments found. Creating new log.
I20260809 12:10:23.922761 5879 log.cc:824] T 4256d5c19a3a47ae8bea0721ab33169a P cdc0af2ee8f9477994b1555d954b66d9: Log is configured to *not* fsync() on all Append() calls
I20260809 12:10:23.924882 5878 tablet_bootstrap.cc:492] T 4256d5c19a3a47ae8bea0721ab33169a P aa783af47cba48afa9f1f262e83568f4: No bootstrap required, opened a new log
I20260809 12:10:23.925232 5878 ts_tablet_manager.cc:1397] T 4256d5c19a3a47ae8bea0721ab33169a P aa783af47cba48afa9f1f262e83568f4: Time spent bootstrapping tablet: real 0.010s user 0.006s sys 0.004s
I20260809 12:10:23.925660 5877 tablet_bootstrap.cc:492] T 4256d5c19a3a47ae8bea0721ab33169a P a1175d22014141e1bad56a71341169b8: No bootstrap required, opened a new log
I20260809 12:10:23.925928 5879 tablet_bootstrap.cc:492] T 4256d5c19a3a47ae8bea0721ab33169a P cdc0af2ee8f9477994b1555d954b66d9: No bootstrap required, opened a new log
I20260809 12:10:23.925940 5877 ts_tablet_manager.cc:1397] T 4256d5c19a3a47ae8bea0721ab33169a P a1175d22014141e1bad56a71341169b8: Time spent bootstrapping tablet: real 0.014s user 0.009s sys 0.004s
I20260809 12:10:23.926219 5879 ts_tablet_manager.cc:1397] T 4256d5c19a3a47ae8bea0721ab33169a P cdc0af2ee8f9477994b1555d954b66d9: Time spent bootstrapping tablet: real 0.008s user 0.003s sys 0.003s
I20260809 12:10:23.933005 5878 raft_consensus.cc:348] T 4256d5c19a3a47ae8bea0721ab33169a P aa783af47cba48afa9f1f262e83568f4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a1175d22014141e1bad56a71341169b8" member_type: VOTER last_known_addr { host: "127.0.222.2" port: 44491 } } peers { permanent_uuid: "aa783af47cba48afa9f1f262e83568f4" member_type: VOTER last_known_addr { host: "127.0.222.1" port: 43045 } } peers { permanent_uuid: "cdc0af2ee8f9477994b1555d954b66d9" member_type: VOTER last_known_addr { host: "127.0.222.3" port: 44817 } }
I20260809 12:10:23.933487 5878 raft_consensus.cc:374] T 4256d5c19a3a47ae8bea0721ab33169a P aa783af47cba48afa9f1f262e83568f4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260809 12:10:23.933635 5878 raft_consensus.cc:729] T 4256d5c19a3a47ae8bea0721ab33169a P aa783af47cba48afa9f1f262e83568f4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: aa783af47cba48afa9f1f262e83568f4, State: Initialized, Role: FOLLOWER
I20260809 12:10:23.934288 5878 consensus_queue.cc:260] T 4256d5c19a3a47ae8bea0721ab33169a P aa783af47cba48afa9f1f262e83568f4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a1175d22014141e1bad56a71341169b8" member_type: VOTER last_known_addr { host: "127.0.222.2" port: 44491 } } peers { permanent_uuid: "aa783af47cba48afa9f1f262e83568f4" member_type: VOTER last_known_addr { host: "127.0.222.1" port: 43045 } } peers { permanent_uuid: "cdc0af2ee8f9477994b1555d954b66d9" member_type: VOTER last_known_addr { host: "127.0.222.3" port: 44817 } }
I20260809 12:10:23.936290 5878 ts_tablet_manager.cc:1428] T 4256d5c19a3a47ae8bea0721ab33169a P aa783af47cba48afa9f1f262e83568f4: Time spent starting tablet: real 0.011s user 0.008s sys 0.001s
I20260809 12:10:23.937052 5879 raft_consensus.cc:348] T 4256d5c19a3a47ae8bea0721ab33169a P cdc0af2ee8f9477994b1555d954b66d9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a1175d22014141e1bad56a71341169b8" member_type: VOTER last_known_addr { host: "127.0.222.2" port: 44491 } } peers { permanent_uuid: "aa783af47cba48afa9f1f262e83568f4" member_type: VOTER last_known_addr { host: "127.0.222.1" port: 43045 } } peers { permanent_uuid: "cdc0af2ee8f9477994b1555d954b66d9" member_type: VOTER last_known_addr { host: "127.0.222.3" port: 44817 } }
I20260809 12:10:23.937052 5877 raft_consensus.cc:348] T 4256d5c19a3a47ae8bea0721ab33169a P a1175d22014141e1bad56a71341169b8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a1175d22014141e1bad56a71341169b8" member_type: VOTER last_known_addr { host: "127.0.222.2" port: 44491 } } peers { permanent_uuid: "aa783af47cba48afa9f1f262e83568f4" member_type: VOTER last_known_addr { host: "127.0.222.1" port: 43045 } } peers { permanent_uuid: "cdc0af2ee8f9477994b1555d954b66d9" member_type: VOTER last_known_addr { host: "127.0.222.3" port: 44817 } }
I20260809 12:10:23.937500 5877 raft_consensus.cc:374] T 4256d5c19a3a47ae8bea0721ab33169a P a1175d22014141e1bad56a71341169b8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260809 12:10:23.937505 5879 raft_consensus.cc:374] T 4256d5c19a3a47ae8bea0721ab33169a P cdc0af2ee8f9477994b1555d954b66d9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260809 12:10:23.937608 5879 raft_consensus.cc:729] T 4256d5c19a3a47ae8bea0721ab33169a P cdc0af2ee8f9477994b1555d954b66d9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: cdc0af2ee8f9477994b1555d954b66d9, State: Initialized, Role: FOLLOWER
I20260809 12:10:23.937618 5877 raft_consensus.cc:729] T 4256d5c19a3a47ae8bea0721ab33169a P a1175d22014141e1bad56a71341169b8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a1175d22014141e1bad56a71341169b8, State: Initialized, Role: FOLLOWER
I20260809 12:10:23.938282 5879 consensus_queue.cc:260] T 4256d5c19a3a47ae8bea0721ab33169a P cdc0af2ee8f9477994b1555d954b66d9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a1175d22014141e1bad56a71341169b8" member_type: VOTER last_known_addr { host: "127.0.222.2" port: 44491 } } peers { permanent_uuid: "aa783af47cba48afa9f1f262e83568f4" member_type: VOTER last_known_addr { host: "127.0.222.1" port: 43045 } } peers { permanent_uuid: "cdc0af2ee8f9477994b1555d954b66d9" member_type: VOTER last_known_addr { host: "127.0.222.3" port: 44817 } }
I20260809 12:10:23.938268 5877 consensus_queue.cc:260] T 4256d5c19a3a47ae8bea0721ab33169a P a1175d22014141e1bad56a71341169b8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a1175d22014141e1bad56a71341169b8" member_type: VOTER last_known_addr { host: "127.0.222.2" port: 44491 } } peers { permanent_uuid: "aa783af47cba48afa9f1f262e83568f4" member_type: VOTER last_known_addr { host: "127.0.222.1" port: 43045 } } peers { permanent_uuid: "cdc0af2ee8f9477994b1555d954b66d9" member_type: VOTER last_known_addr { host: "127.0.222.3" port: 44817 } }
I20260809 12:10:23.939536 5857 heartbeater.cc:503] Master 127.0.222.62:44975 was elected leader, sending a full tablet report...
I20260809 12:10:23.940014 5711 heartbeater.cc:503] Master 127.0.222.62:44975 was elected leader, sending a full tablet report...
I20260809 12:10:23.942035 5877 ts_tablet_manager.cc:1428] T 4256d5c19a3a47ae8bea0721ab33169a P a1175d22014141e1bad56a71341169b8: Time spent starting tablet: real 0.016s user 0.007s sys 0.007s
I20260809 12:10:23.942771 5879 ts_tablet_manager.cc:1428] T 4256d5c19a3a47ae8bea0721ab33169a P cdc0af2ee8f9477994b1555d954b66d9: Time spent starting tablet: real 0.016s user 0.015s sys 0.001s
W20260809 12:10:24.010469 5567 tablet.cc:2369] T 4256d5c19a3a47ae8bea0721ab33169a P aa783af47cba48afa9f1f262e83568f4: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260809 12:10:24.023913 5885 raft_consensus.cc:482] T 4256d5c19a3a47ae8bea0721ab33169a P cdc0af2ee8f9477994b1555d954b66d9 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260809 12:10:24.024197 5885 raft_consensus.cc:504] T 4256d5c19a3a47ae8bea0721ab33169a P cdc0af2ee8f9477994b1555d954b66d9 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a1175d22014141e1bad56a71341169b8" member_type: VOTER last_known_addr { host: "127.0.222.2" port: 44491 } } peers { permanent_uuid: "aa783af47cba48afa9f1f262e83568f4" member_type: VOTER last_known_addr { host: "127.0.222.1" port: 43045 } } peers { permanent_uuid: "cdc0af2ee8f9477994b1555d954b66d9" member_type: VOTER last_known_addr { host: "127.0.222.3" port: 44817 } }
I20260809 12:10:24.025878 5885 leader_election.cc:288] T 4256d5c19a3a47ae8bea0721ab33169a P cdc0af2ee8f9477994b1555d954b66d9 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers a1175d22014141e1bad56a71341169b8 (127.0.222.2:44491), aa783af47cba48afa9f1f262e83568f4 (127.0.222.1:43045)
I20260809 12:10:24.033085 5859 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 12:10:24.026282 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.222.2:44491 (local address 127.0.222.3:47085)
0809 12:10:24.026605 (+ 323us) negotiation.cc:107] Waiting for socket to connect
0809 12:10:24.026629 (+ 24us) client_negotiation.cc:174] Beginning negotiation
0809 12:10:24.026830 (+ 201us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 12:10:24.027241 (+ 411us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 12:10:24.027246 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 12:10:24.027272 (+ 26us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0809 12:10:24.027515 (+ 243us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 12:10:24.027525 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 12:10:24.028860 (+ 1335us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 12:10:24.028863 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 12:10:24.030842 (+ 1979us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 12:10:24.030858 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 12:10:24.032332 (+ 1474us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 12:10:24.032336 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 12:10:24.032469 (+ 133us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 12:10:24.032505 (+ 36us) client_negotiation.cc:769] Sending connection context
0809 12:10:24.032581 (+ 76us) client_negotiation.cc:240] Negotiation successful
0809 12:10:24.032737 (+ 156us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":184}
I20260809 12:10:24.033051 5873 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 12:10:24.026348 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.222.3:47085 (local address 127.0.222.2:44491)
0809 12:10:24.026564 (+ 216us) server_negotiation.cc:206] Beginning negotiation
0809 12:10:24.026571 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0809 12:10:24.026702 (+ 131us) server_negotiation.cc:407] Connection header received
0809 12:10:24.027016 (+ 314us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 12:10:24.027021 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 12:10:24.027085 (+ 64us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 12:10:24.027215 (+ 130us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0809 12:10:24.027709 (+ 494us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 12:10:24.028716 (+ 1007us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 12:10:24.031072 (+ 2356us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 12:10:24.032185 (+ 1113us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 12:10:24.032329 (+ 144us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 12:10:24.032448 (+ 119us) server_negotiation.cc:1035] Waiting for connection context
0809 12:10:24.032644 (+ 196us) server_negotiation.cc:299] Negotiation successful
0809 12:10:24.032788 (+ 144us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":67}
W20260809 12:10:24.033392 5712 tablet.cc:2369] T 4256d5c19a3a47ae8bea0721ab33169a P a1175d22014141e1bad56a71341169b8: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260809 12:10:24.033759 5874 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 12:10:24.026472 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.222.3:49967 (local address 127.0.222.1:43045)
0809 12:10:24.026695 (+ 223us) server_negotiation.cc:206] Beginning negotiation
0809 12:10:24.026701 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0809 12:10:24.027422 (+ 721us) server_negotiation.cc:407] Connection header received
0809 12:10:24.027650 (+ 228us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 12:10:24.027657 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 12:10:24.027739 (+ 82us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 12:10:24.027839 (+ 100us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0809 12:10:24.028344 (+ 505us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 12:10:24.029234 (+ 890us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 12:10:24.031463 (+ 2229us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 12:10:24.032712 (+ 1249us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 12:10:24.032846 (+ 134us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 12:10:24.032947 (+ 101us) server_negotiation.cc:1035] Waiting for connection context
0809 12:10:24.033369 (+ 422us) server_negotiation.cc:299] Negotiation successful
0809 12:10:24.033512 (+ 143us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":66}
I20260809 12:10:24.034459 5886 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 12:10:24.026289 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.222.1:43045 (local address 127.0.222.3:49967)
0809 12:10:24.027323 (+ 1034us) negotiation.cc:107] Waiting for socket to connect
0809 12:10:24.027338 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0809 12:10:24.027453 (+ 115us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 12:10:24.027881 (+ 428us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 12:10:24.027885 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 12:10:24.027904 (+ 19us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0809 12:10:24.028186 (+ 282us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 12:10:24.028194 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 12:10:24.029398 (+ 1204us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 12:10:24.029408 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 12:10:24.031313 (+ 1905us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 12:10:24.031323 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 12:10:24.032920 (+ 1597us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 12:10:24.032931 (+ 11us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 12:10:24.033125 (+ 194us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 12:10:24.033191 (+ 66us) client_negotiation.cc:769] Sending connection context
0809 12:10:24.033941 (+ 750us) client_negotiation.cc:240] Negotiation successful
0809 12:10:24.034083 (+ 142us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":914,"mutex_wait_us":223,"thread_start_us":458,"threads_started":1}
I20260809 12:10:24.034463 5659 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4256d5c19a3a47ae8bea0721ab33169a" candidate_uuid: "cdc0af2ee8f9477994b1555d954b66d9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a1175d22014141e1bad56a71341169b8" is_pre_election: true
I20260809 12:10:24.035121 5659 raft_consensus.cc:2454] T 4256d5c19a3a47ae8bea0721ab33169a P a1175d22014141e1bad56a71341169b8 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate cdc0af2ee8f9477994b1555d954b66d9 in term 0.
I20260809 12:10:24.035604 5517 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4256d5c19a3a47ae8bea0721ab33169a" candidate_uuid: "cdc0af2ee8f9477994b1555d954b66d9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "aa783af47cba48afa9f1f262e83568f4" is_pre_election: true
I20260809 12:10:24.035915 5736 leader_election.cc:302] T 4256d5c19a3a47ae8bea0721ab33169a P cdc0af2ee8f9477994b1555d954b66d9 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: a1175d22014141e1bad56a71341169b8, cdc0af2ee8f9477994b1555d954b66d9; no voters:
I20260809 12:10:24.036113 5517 raft_consensus.cc:2454] T 4256d5c19a3a47ae8bea0721ab33169a P aa783af47cba48afa9f1f262e83568f4 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate cdc0af2ee8f9477994b1555d954b66d9 in term 0.
I20260809 12:10:24.036442 5885 raft_consensus.cc:2781] T 4256d5c19a3a47ae8bea0721ab33169a P cdc0af2ee8f9477994b1555d954b66d9 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260809 12:10:24.036609 5885 raft_consensus.cc:482] T 4256d5c19a3a47ae8bea0721ab33169a P cdc0af2ee8f9477994b1555d954b66d9 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260809 12:10:24.036746 5885 raft_consensus.cc:3037] T 4256d5c19a3a47ae8bea0721ab33169a P cdc0af2ee8f9477994b1555d954b66d9 [term 0 FOLLOWER]: Advancing to term 1
I20260809 12:10:24.039283 5885 raft_consensus.cc:504] T 4256d5c19a3a47ae8bea0721ab33169a P cdc0af2ee8f9477994b1555d954b66d9 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a1175d22014141e1bad56a71341169b8" member_type: VOTER last_known_addr { host: "127.0.222.2" port: 44491 } } peers { permanent_uuid: "aa783af47cba48afa9f1f262e83568f4" member_type: VOTER last_known_addr { host: "127.0.222.1" port: 43045 } } peers { permanent_uuid: "cdc0af2ee8f9477994b1555d954b66d9" member_type: VOTER last_known_addr { host: "127.0.222.3" port: 44817 } }
I20260809 12:10:24.039978 5885 leader_election.cc:288] T 4256d5c19a3a47ae8bea0721ab33169a P cdc0af2ee8f9477994b1555d954b66d9 [CANDIDATE]: Term 1 election: Requested vote from peers a1175d22014141e1bad56a71341169b8 (127.0.222.2:44491), aa783af47cba48afa9f1f262e83568f4 (127.0.222.1:43045)
I20260809 12:10:24.040292 5659 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4256d5c19a3a47ae8bea0721ab33169a" candidate_uuid: "cdc0af2ee8f9477994b1555d954b66d9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a1175d22014141e1bad56a71341169b8"
I20260809 12:10:24.040365 5517 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4256d5c19a3a47ae8bea0721ab33169a" candidate_uuid: "cdc0af2ee8f9477994b1555d954b66d9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "aa783af47cba48afa9f1f262e83568f4"
I20260809 12:10:24.040489 5659 raft_consensus.cc:3037] T 4256d5c19a3a47ae8bea0721ab33169a P a1175d22014141e1bad56a71341169b8 [term 0 FOLLOWER]: Advancing to term 1
I20260809 12:10:24.040572 5517 raft_consensus.cc:3037] T 4256d5c19a3a47ae8bea0721ab33169a P aa783af47cba48afa9f1f262e83568f4 [term 0 FOLLOWER]: Advancing to term 1
I20260809 12:10:24.042682 5659 raft_consensus.cc:2454] T 4256d5c19a3a47ae8bea0721ab33169a P a1175d22014141e1bad56a71341169b8 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate cdc0af2ee8f9477994b1555d954b66d9 in term 1.
I20260809 12:10:24.042846 5517 raft_consensus.cc:2454] T 4256d5c19a3a47ae8bea0721ab33169a P aa783af47cba48afa9f1f262e83568f4 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate cdc0af2ee8f9477994b1555d954b66d9 in term 1.
I20260809 12:10:24.043109 5736 leader_election.cc:302] T 4256d5c19a3a47ae8bea0721ab33169a P cdc0af2ee8f9477994b1555d954b66d9 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: a1175d22014141e1bad56a71341169b8, cdc0af2ee8f9477994b1555d954b66d9; no voters:
I20260809 12:10:24.043418 5885 raft_consensus.cc:2781] T 4256d5c19a3a47ae8bea0721ab33169a P cdc0af2ee8f9477994b1555d954b66d9 [term 1 FOLLOWER]: Leader election won for term 1
I20260809 12:10:24.043900 5885 raft_consensus.cc:686] T 4256d5c19a3a47ae8bea0721ab33169a P cdc0af2ee8f9477994b1555d954b66d9 [term 1 LEADER]: Becoming Leader. State: Replica: cdc0af2ee8f9477994b1555d954b66d9, State: Running, Role: LEADER
I20260809 12:10:24.044459 5885 consensus_queue.cc:237] T 4256d5c19a3a47ae8bea0721ab33169a P cdc0af2ee8f9477994b1555d954b66d9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a1175d22014141e1bad56a71341169b8" member_type: VOTER last_known_addr { host: "127.0.222.2" port: 44491 } } peers { permanent_uuid: "aa783af47cba48afa9f1f262e83568f4" member_type: VOTER last_known_addr { host: "127.0.222.1" port: 43045 } } peers { permanent_uuid: "cdc0af2ee8f9477994b1555d954b66d9" member_type: VOTER last_known_addr { host: "127.0.222.3" port: 44817 } }
W20260809 12:10:24.048710 5858 tablet.cc:2369] T 4256d5c19a3a47ae8bea0721ab33169a P cdc0af2ee8f9477994b1555d954b66d9: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260809 12:10:24.050526 5360 catalog_manager.cc:5495] T 4256d5c19a3a47ae8bea0721ab33169a P cdc0af2ee8f9477994b1555d954b66d9 reported cstate change: term changed from 0 to 1, leader changed from <none> to cdc0af2ee8f9477994b1555d954b66d9 (127.0.222.3). New cstate: current_term: 1 leader_uuid: "cdc0af2ee8f9477994b1555d954b66d9" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a1175d22014141e1bad56a71341169b8" member_type: VOTER last_known_addr { host: "127.0.222.2" port: 44491 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "aa783af47cba48afa9f1f262e83568f4" member_type: VOTER last_known_addr { host: "127.0.222.1" port: 43045 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "cdc0af2ee8f9477994b1555d954b66d9" member_type: VOTER last_known_addr { host: "127.0.222.3" port: 44817 } health_report { overall_health: HEALTHY } } }
I20260809 12:10:24.089432 5875 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 12:10:24.084034 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59892 (local address 127.0.222.3:44817)
0809 12:10:24.084227 (+ 193us) server_negotiation.cc:206] Beginning negotiation
0809 12:10:24.084232 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0809 12:10:24.084343 (+ 111us) server_negotiation.cc:407] Connection header received
0809 12:10:24.084526 (+ 183us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 12:10:24.084530 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 12:10:24.084611 (+ 81us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 12:10:24.084738 (+ 127us) server_negotiation.cc:226] Negotiated authn=TOKEN
0809 12:10:24.085106 (+ 368us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 12:10:24.085883 (+ 777us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 12:10:24.087409 (+ 1526us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 12:10:24.087824 (+ 415us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 12:10:24.087925 (+ 101us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 12:10:24.088385 (+ 460us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0809 12:10:24.088929 (+ 544us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0809 12:10:24.089055 (+ 126us) server_negotiation.cc:1035] Waiting for connection context
0809 12:10:24.089158 (+ 103us) server_negotiation.cc:299] Negotiation successful
0809 12:10:24.089230 (+ 72us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":60}
I20260809 12:10:24.098120 5659 raft_consensus.cc:1261] T 4256d5c19a3a47ae8bea0721ab33169a P a1175d22014141e1bad56a71341169b8 [term 1 FOLLOWER]: Refusing update from remote peer cdc0af2ee8f9477994b1555d954b66d9: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260809 12:10:24.098120 5517 raft_consensus.cc:1261] T 4256d5c19a3a47ae8bea0721ab33169a P aa783af47cba48afa9f1f262e83568f4 [term 1 FOLLOWER]: Refusing update from remote peer cdc0af2ee8f9477994b1555d954b66d9: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260809 12:10:24.099241 5887 consensus_queue.cc:1035] T 4256d5c19a3a47ae8bea0721ab33169a P cdc0af2ee8f9477994b1555d954b66d9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "aa783af47cba48afa9f1f262e83568f4" member_type: VOTER last_known_addr { host: "127.0.222.1" port: 43045 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260809 12:10:24.099625 5885 consensus_queue.cc:1035] T 4256d5c19a3a47ae8bea0721ab33169a P cdc0af2ee8f9477994b1555d954b66d9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a1175d22014141e1bad56a71341169b8" member_type: VOTER last_known_addr { host: "127.0.222.2" port: 44491 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260809 12:10:24.112836 5890 mvcc.cc:205] Tried to move back new op lower bound from 7316592329095360512 to 7316592328891604992. Current Snapshot: MvccSnapshot[applied={T|T < 7316592329095360512}]
I20260809 12:10:24.128348 5360 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:38680:
table { table_name: "test-table" } modify_external_catalogs: true
I20260809 12:10:24.128679 5360 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:38680:
table { table_name: "test-table" } modify_external_catalogs: true
I20260809 12:10:24.132848 5360 catalog_manager.cc:5782] T 00000000000000000000000000000000 P cc90aedc79944628b63ccf00259c6ede: Sending DeleteTablet for 3 replicas of tablet 4256d5c19a3a47ae8bea0721ab33169a
I20260809 12:10:24.134119 5638 tablet_service.cc:1514] Processing DeleteTablet for tablet 4256d5c19a3a47ae8bea0721ab33169a with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-09 12:10:24 UTC) from {username='kudu', principal='kudu/127.0.222.62@KRBTEST.COM'} at 127.0.0.1:45948
I20260809 12:10:24.134214 5496 tablet_service.cc:1514] Processing DeleteTablet for tablet 4256d5c19a3a47ae8bea0721ab33169a with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-09 12:10:24 UTC) from {username='kudu', principal='kudu/127.0.222.62@KRBTEST.COM'} at 127.0.0.1:45346
I20260809 12:10:24.134450 5789 tablet_service.cc:1514] Processing DeleteTablet for tablet 4256d5c19a3a47ae8bea0721ab33169a with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-09 12:10:24 UTC) from {username='kudu', principal='kudu/127.0.222.62@KRBTEST.COM'} at 127.0.0.1:59876
I20260809 12:10:24.136727 888 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskgH5Pjm/build/asan/bin/kudu pid 5427
I20260809 12:10:24.144567 5904 tablet_replica.cc:330] T 4256d5c19a3a47ae8bea0721ab33169a P cdc0af2ee8f9477994b1555d954b66d9: stopping tablet replica
I20260809 12:10:24.145242 5904 raft_consensus.cc:2229] T 4256d5c19a3a47ae8bea0721ab33169a P cdc0af2ee8f9477994b1555d954b66d9 [term 1 LEADER]: Raft consensus shutting down.
I20260809 12:10:24.146114 5904 raft_consensus.cc:2258] T 4256d5c19a3a47ae8bea0721ab33169a P cdc0af2ee8f9477994b1555d954b66d9 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260809 12:10:24.148568 5904 ts_tablet_manager.cc:1905] T 4256d5c19a3a47ae8bea0721ab33169a P cdc0af2ee8f9477994b1555d954b66d9: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260809 12:10:24.149223 5901 tablet_replica.cc:330] T 4256d5c19a3a47ae8bea0721ab33169a P aa783af47cba48afa9f1f262e83568f4: stopping tablet replica
I20260809 12:10:24.149856 5903 tablet_replica.cc:330] T 4256d5c19a3a47ae8bea0721ab33169a P a1175d22014141e1bad56a71341169b8: stopping tablet replica
I20260809 12:10:24.150393 5901 raft_consensus.cc:2229] T 4256d5c19a3a47ae8bea0721ab33169a P aa783af47cba48afa9f1f262e83568f4 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260809 12:10:24.150906 5903 raft_consensus.cc:2229] T 4256d5c19a3a47ae8bea0721ab33169a P a1175d22014141e1bad56a71341169b8 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260809 12:10:24.151497 5903 raft_consensus.cc:2258] T 4256d5c19a3a47ae8bea0721ab33169a P a1175d22014141e1bad56a71341169b8 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260809 12:10:24.151991 5901 raft_consensus.cc:2258] T 4256d5c19a3a47ae8bea0721ab33169a P aa783af47cba48afa9f1f262e83568f4 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260809 12:10:24.154786 5903 ts_tablet_manager.cc:1905] T 4256d5c19a3a47ae8bea0721ab33169a P a1175d22014141e1bad56a71341169b8: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260809 12:10:24.156180 5901 ts_tablet_manager.cc:1905] T 4256d5c19a3a47ae8bea0721ab33169a P aa783af47cba48afa9f1f262e83568f4: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260809 12:10:24.157442 5904 ts_tablet_manager.cc:1918] T 4256d5c19a3a47ae8bea0721ab33169a P cdc0af2ee8f9477994b1555d954b66d9: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260809 12:10:24.157658 5904 log.cc:1196] T 4256d5c19a3a47ae8bea0721ab33169a P cdc0af2ee8f9477994b1555d954b66d9: Deleting WAL directory at /tmp/dist-test-taskgH5Pjm/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786277382073938-888-0/minicluster-data/ts-2/wal/wals/4256d5c19a3a47ae8bea0721ab33169a
Aug 09 12:10:24 dist-test-slave-2w3w krb5kdc[5317](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786277421, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.222.1@KRBTEST.COM
I20260809 12:10:24.158310 5904 ts_tablet_manager.cc:1939] T 4256d5c19a3a47ae8bea0721ab33169a P cdc0af2ee8f9477994b1555d954b66d9: Deleting consensus metadata
I20260809 12:10:24.160322 5348 catalog_manager.cc:4841] TS cdc0af2ee8f9477994b1555d954b66d9 (127.0.222.3:44817): tablet 4256d5c19a3a47ae8bea0721ab33169a (table test-table [id=74677c77ee7d4419bad4e46f6f97ce6f]) successfully deleted
I20260809 12:10:24.165984 5903 ts_tablet_manager.cc:1918] T 4256d5c19a3a47ae8bea0721ab33169a P a1175d22014141e1bad56a71341169b8: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260809 12:10:24.166355 5903 log.cc:1196] T 4256d5c19a3a47ae8bea0721ab33169a P a1175d22014141e1bad56a71341169b8: Deleting WAL directory at /tmp/dist-test-taskgH5Pjm/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786277382073938-888-0/minicluster-data/ts-1/wal/wals/4256d5c19a3a47ae8bea0721ab33169a
I20260809 12:10:24.166400 5901 ts_tablet_manager.cc:1918] T 4256d5c19a3a47ae8bea0721ab33169a P aa783af47cba48afa9f1f262e83568f4: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260809 12:10:24.166635 5901 log.cc:1196] T 4256d5c19a3a47ae8bea0721ab33169a P aa783af47cba48afa9f1f262e83568f4: Deleting WAL directory at /tmp/dist-test-taskgH5Pjm/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786277382073938-888-0/minicluster-data/ts-0/wal/wals/4256d5c19a3a47ae8bea0721ab33169a
I20260809 12:10:24.167156 5903 ts_tablet_manager.cc:1939] T 4256d5c19a3a47ae8bea0721ab33169a P a1175d22014141e1bad56a71341169b8: Deleting consensus metadata
I20260809 12:10:24.167280 5901 ts_tablet_manager.cc:1939] T 4256d5c19a3a47ae8bea0721ab33169a P aa783af47cba48afa9f1f262e83568f4: Deleting consensus metadata
I20260809 12:10:24.168118 5874 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 12:10:24.138435 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45350 (local address 127.0.222.1:43045)
0809 12:10:24.138677 (+ 242us) server_negotiation.cc:206] Beginning negotiation
0809 12:10:24.138686 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0809 12:10:24.143647 (+ 4961us) server_negotiation.cc:407] Connection header received
0809 12:10:24.143888 (+ 241us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 12:10:24.143897 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 12:10:24.144046 (+ 149us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 12:10:24.144193 (+ 147us) server_negotiation.cc:226] Negotiated authn=SASL
0809 12:10:24.149074 (+ 4881us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 12:10:24.150247 (+ 1173us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 12:10:24.152168 (+ 1921us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 12:10:24.152890 (+ 722us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 12:10:24.152984 (+ 94us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 12:10:24.159350 (+ 6366us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 12:10:24.159365 (+ 15us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 12:10:24.159384 (+ 19us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0809 12:10:24.159426 (+ 42us) server_negotiation.cc:922] calling sasl_server_start()
0809 12:10:24.163475 (+ 4049us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 12:10:24.164476 (+ 1001us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 12:10:24.164485 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 12:10:24.164488 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0809 12:10:24.164667 (+ 179us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 12:10:24.165165 (+ 498us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 12:10:24.165172 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 12:10:24.165178 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0809 12:10:24.165656 (+ 478us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 12:10:24.165975 (+ 319us) server_negotiation.cc:1035] Waiting for connection context
0809 12:10:24.167479 (+ 1504us) server_negotiation.cc:299] Negotiation successful
0809 12:10:24.167708 (+ 229us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":77}
I20260809 12:10:24.168946 5561 generic_service.cc:196] Checking for leaks (request via RPC)
I20260809 12:10:24.169276 5348 catalog_manager.cc:4841] TS a1175d22014141e1bad56a71341169b8 (127.0.222.2:44491): tablet 4256d5c19a3a47ae8bea0721ab33169a (table test-table [id=74677c77ee7d4419bad4e46f6f97ce6f]) successfully deleted
I20260809 12:10:24.169279 5347 catalog_manager.cc:4841] TS aa783af47cba48afa9f1f262e83568f4 (127.0.222.1:43045): tablet 4256d5c19a3a47ae8bea0721ab33169a (table test-table [id=74677c77ee7d4419bad4e46f6f97ce6f]) successfully deleted
I20260809 12:10:24.261201 888 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskgH5Pjm/build/asan/bin/kudu with pid 5427
I20260809 12:10:24.282555 888 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskgH5Pjm/build/asan/bin/kudu pid 5572
Aug 09 12:10:24 dist-test-slave-2w3w krb5kdc[5317](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786277421, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.222.2@KRBTEST.COM
I20260809 12:10:24.298589 5873 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 12:10:24.283393 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45950 (local address 127.0.222.2:44491)
0809 12:10:24.283570 (+ 177us) server_negotiation.cc:206] Beginning negotiation
0809 12:10:24.283574 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0809 12:10:24.283809 (+ 235us) server_negotiation.cc:407] Connection header received
0809 12:10:24.284025 (+ 216us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 12:10:24.284037 (+ 12us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 12:10:24.284134 (+ 97us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 12:10:24.284244 (+ 110us) server_negotiation.cc:226] Negotiated authn=SASL
0809 12:10:24.285607 (+ 1363us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 12:10:24.286374 (+ 767us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 12:10:24.287797 (+ 1423us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 12:10:24.288253 (+ 456us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 12:10:24.288400 (+ 147us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 12:10:24.292532 (+ 4132us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 12:10:24.292545 (+ 13us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 12:10:24.292559 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0809 12:10:24.292596 (+ 37us) server_negotiation.cc:922] calling sasl_server_start()
0809 12:10:24.295708 (+ 3112us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 12:10:24.296379 (+ 671us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 12:10:24.296395 (+ 16us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 12:10:24.296397 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0809 12:10:24.296517 (+ 120us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 12:10:24.296918 (+ 401us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 12:10:24.296924 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 12:10:24.296927 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0809 12:10:24.297316 (+ 389us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 12:10:24.297635 (+ 319us) server_negotiation.cc:1035] Waiting for connection context
0809 12:10:24.297911 (+ 276us) server_negotiation.cc:299] Negotiation successful
0809 12:10:24.298163 (+ 252us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":62}
I20260809 12:10:24.299412 5705 generic_service.cc:196] Checking for leaks (request via RPC)
I20260809 12:10:24.389534 888 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskgH5Pjm/build/asan/bin/kudu with pid 5572
I20260809 12:10:24.409749 888 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskgH5Pjm/build/asan/bin/kudu pid 5718
Aug 09 12:10:24 dist-test-slave-2w3w krb5kdc[5317](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786277421, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.222.3@KRBTEST.COM
I20260809 12:10:24.427591 5875 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 12:10:24.410586 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59898 (local address 127.0.222.3:44817)
0809 12:10:24.410792 (+ 206us) server_negotiation.cc:206] Beginning negotiation
0809 12:10:24.410798 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0809 12:10:24.410823 (+ 25us) server_negotiation.cc:407] Connection header received
0809 12:10:24.410945 (+ 122us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 12:10:24.410952 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 12:10:24.411086 (+ 134us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 12:10:24.411209 (+ 123us) server_negotiation.cc:226] Negotiated authn=SASL
0809 12:10:24.412498 (+ 1289us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 12:10:24.413731 (+ 1233us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 12:10:24.415376 (+ 1645us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 12:10:24.416092 (+ 716us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 12:10:24.416240 (+ 148us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 12:10:24.420362 (+ 4122us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 12:10:24.420380 (+ 18us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 12:10:24.420400 (+ 20us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0809 12:10:24.420449 (+ 49us) server_negotiation.cc:922] calling sasl_server_start()
0809 12:10:24.424548 (+ 4099us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 12:10:24.425190 (+ 642us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 12:10:24.425199 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 12:10:24.425203 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0809 12:10:24.425394 (+ 191us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 12:10:24.425759 (+ 365us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 12:10:24.425764 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 12:10:24.425768 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0809 12:10:24.426264 (+ 496us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 12:10:24.426602 (+ 338us) server_negotiation.cc:1035] Waiting for connection context
0809 12:10:24.426889 (+ 287us) server_negotiation.cc:299] Negotiation successful
0809 12:10:24.427151 (+ 262us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":84}
I20260809 12:10:24.428478 5852 generic_service.cc:196] Checking for leaks (request via RPC)
I20260809 12:10:24.551151 888 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskgH5Pjm/build/asan/bin/kudu with pid 5718
I20260809 12:10:24.573562 888 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskgH5Pjm/build/asan/bin/kudu pid 5330
I20260809 12:10:24.574714 5393 generic_service.cc:196] Checking for leaks (request via RPC)
I20260809 12:10:24.689219 888 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskgH5Pjm/build/asan/bin/kudu with pid 5330
2026-08-09T12:10:24Z chronyd exiting
[ OK ] SecurityITest.TestDisableWebUI (4419 ms)
[ RUN ] SecurityITest.TestDisableAuthenticationEncryption
2026-08-09T12:10:24Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-09T12:10:24Z Disabled control of system clock
I20260809 12:10:24.768107 888 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskgH5Pjm/build/asan/bin/kudu
/tmp/dist-test-taskgH5Pjm/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskgH5Pjm/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786277382073938-888-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskgH5Pjm/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786277382073938-888-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskgH5Pjm/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786277382073938-888-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskgH5Pjm/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786277382073938-888-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.222.62:38945
--webserver_interface=127.0.222.62
--webserver_port=0
--builtin_ntp_servers=127.0.222.20:38877
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.222.62:38945
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--rpc_authentication=disabled
--rpc_encryption=disabled with env {}
W20260809 12:10:25.106184 5916 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260809 12:10:25.106487 5916 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260809 12:10:25.106660 5916 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260809 12:10:25.114158 5916 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260809 12:10:25.114272 5916 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260809 12:10:25.114362 5916 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260809 12:10:25.114476 5916 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260809 12:10:25.114557 5916 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260809 12:10:25.124373 5916 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.222.20:38877
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskgH5Pjm/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786277382073938-888-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskgH5Pjm/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786277382073938-888-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.222.62:38945
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.222.62:38945
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskgH5Pjm/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786277382073938-888-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.222.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskgH5Pjm/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786277382073938-888-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 09 Aug 2026 11:06:41 UTC on ccycloud.quasar-bwoogy.root.comops.site
build id 4575991
ASAN enabled
I20260809 12:10:25.125768 5916 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260809 12:10:25.127753 5916 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260809 12:10:25.139787 5922 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 12:10:25.139833 5923 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 12:10:25.139889 5925 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260809 12:10:25.141018 5916 server_base.cc:1034] running on GCE node
I20260809 12:10:25.141780 5916 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260809 12:10:25.143222 5916 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260809 12:10:25.144470 5916 hybrid_clock.cc:648] HybridClock initialized: now 1786277425144432 us; error 48 us; skew 500 ppm
I20260809 12:10:25.144821 5916 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260809 12:10:25.147217 5916 webserver.cc:460] Webserver started at http://127.0.222.62:42335/ using document root <none> and password file <none>
I20260809 12:10:25.147974 5916 fs_manager.cc:359] Metadata directory not provided
I20260809 12:10:25.148108 5916 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260809 12:10:25.148465 5916 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260809 12:10:25.151445 5916 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskgH5Pjm/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786277382073938-888-0/minicluster-data/master-0/data/instance:
uuid: "39893086d5c946c08450f232d49c51cf"
format_stamp: "Formatted at 2026-08-09 12:10:25 on dist-test-slave-2w3w"
server_key: "981c91a1f360b6daf2ef4d71fbc69b16"
server_key_iv: "9b23e959033757f0ff488496c6f01c79"
server_key_version: "encryptionkey@0"
I20260809 12:10:25.152244 5916 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskgH5Pjm/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786277382073938-888-0/minicluster-data/master-0/wal/instance:
uuid: "39893086d5c946c08450f232d49c51cf"
format_stamp: "Formatted at 2026-08-09 12:10:25 on dist-test-slave-2w3w"
server_key: "981c91a1f360b6daf2ef4d71fbc69b16"
server_key_iv: "9b23e959033757f0ff488496c6f01c79"
server_key_version: "encryptionkey@0"
I20260809 12:10:25.157262 5916 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.000s
I20260809 12:10:25.160791 5931 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260809 12:10:25.162153 5916 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260809 12:10:25.162348 5916 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskgH5Pjm/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786277382073938-888-0/minicluster-data/master-0/data,/tmp/dist-test-taskgH5Pjm/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786277382073938-888-0/minicluster-data/master-0/wal
uuid: "39893086d5c946c08450f232d49c51cf"
format_stamp: "Formatted at 2026-08-09 12:10:25 on dist-test-slave-2w3w"
server_key: "981c91a1f360b6daf2ef4d71fbc69b16"
server_key_iv: "9b23e959033757f0ff488496c6f01c79"
server_key_version: "encryptionkey@0"
I20260809 12:10:25.162585 5916 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskgH5Pjm/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786277382073938-888-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskgH5Pjm/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786277382073938-888-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskgH5Pjm/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786277382073938-888-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260809 12:10:25.168366 5916 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260809 12:10:25.169116 5916 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260809 12:10:25.169428 5916 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260809 12:10:25.187542 5916 rpc_server.cc:307] RPC server started. Bound to: 127.0.222.62:38945
I20260809 12:10:25.187582 5983 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.222.62:38945 every 8 connection(s)
I20260809 12:10:25.189056 5916 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskgH5Pjm/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786277382073938-888-0/minicluster-data/master-0/data/info.pb
I20260809 12:10:25.192938 5984 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260809 12:10:25.199075 888 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskgH5Pjm/build/asan/bin/kudu as pid 5916
I20260809 12:10:25.199293 888 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskgH5Pjm/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786277382073938-888-0/minicluster-data/master-0/wal/instance
I20260809 12:10:25.199976 888 external_mini_cluster.cc:1428] Setting key b236bb8bd94a9cf0d8c5675bd1ecb13c
I20260809 12:10:25.204566 5984 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 39893086d5c946c08450f232d49c51cf: Bootstrap starting.
I20260809 12:10:25.208076 5984 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 39893086d5c946c08450f232d49c51cf: Neither blocks nor log segments found. Creating new log.
I20260809 12:10:25.208879 5987 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 12:10:25.202164 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46692 (local address 127.0.222.62:38945)
0809 12:10:25.203307 (+ 1143us) server_negotiation.cc:206] Beginning negotiation
0809 12:10:25.203320 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0809 12:10:25.203368 (+ 48us) server_negotiation.cc:407] Connection header received
0809 12:10:25.205116 (+ 1748us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 12:10:25.205140 (+ 24us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 12:10:25.205561 (+ 421us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 12:10:25.206007 (+ 446us) server_negotiation.cc:226] Negotiated authn=SASL
0809 12:10:25.207175 (+ 1168us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 12:10:25.207181 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 12:10:25.207193 (+ 12us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0809 12:10:25.207207 (+ 14us) server_negotiation.cc:922] calling sasl_server_start()
0809 12:10:25.207260 (+ 53us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0809 12:10:25.207274 (+ 14us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 12:10:25.207467 (+ 193us) server_negotiation.cc:1035] Waiting for connection context
0809 12:10:25.207618 (+ 151us) server_negotiation.cc:299] Negotiation successful
0809 12:10:25.207886 (+ 268us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":597,"thread_start_us":382,"threads_started":1}
I20260809 12:10:25.209338 5984 log.cc:824] T 00000000000000000000000000000000 P 39893086d5c946c08450f232d49c51cf: Log is configured to *not* fsync() on all Append() calls
I20260809 12:10:25.212695 5984 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 39893086d5c946c08450f232d49c51cf: No bootstrap required, opened a new log
I20260809 12:10:25.220248 5984 raft_consensus.cc:348] T 00000000000000000000000000000000 P 39893086d5c946c08450f232d49c51cf [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "39893086d5c946c08450f232d49c51cf" member_type: VOTER last_known_addr { host: "127.0.222.62" port: 38945 } }
I20260809 12:10:25.220609 5984 raft_consensus.cc:374] T 00000000000000000000000000000000 P 39893086d5c946c08450f232d49c51cf [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260809 12:10:25.220727 5984 raft_consensus.cc:729] T 00000000000000000000000000000000 P 39893086d5c946c08450f232d49c51cf [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 39893086d5c946c08450f232d49c51cf, State: Initialized, Role: FOLLOWER
I20260809 12:10:25.221448 5984 consensus_queue.cc:260] T 00000000000000000000000000000000 P 39893086d5c946c08450f232d49c51cf [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "39893086d5c946c08450f232d49c51cf" member_type: VOTER last_known_addr { host: "127.0.222.62" port: 38945 } }
I20260809 12:10:25.221772 5984 raft_consensus.cc:388] T 00000000000000000000000000000000 P 39893086d5c946c08450f232d49c51cf [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260809 12:10:25.221939 5984 raft_consensus.cc:482] T 00000000000000000000000000000000 P 39893086d5c946c08450f232d49c51cf [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260809 12:10:25.222138 5984 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 39893086d5c946c08450f232d49c51cf [term 0 FOLLOWER]: Advancing to term 1
I20260809 12:10:25.224485 5984 raft_consensus.cc:504] T 00000000000000000000000000000000 P 39893086d5c946c08450f232d49c51cf [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "39893086d5c946c08450f232d49c51cf" member_type: VOTER last_known_addr { host: "127.0.222.62" port: 38945 } }
I20260809 12:10:25.225021 5984 leader_election.cc:302] T 00000000000000000000000000000000 P 39893086d5c946c08450f232d49c51cf [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 39893086d5c946c08450f232d49c51cf; no voters:
I20260809 12:10:25.225630 5984 leader_election.cc:288] T 00000000000000000000000000000000 P 39893086d5c946c08450f232d49c51cf [CANDIDATE]: Term 1 election: Requested vote from peers
I20260809 12:10:25.225780 5989 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 39893086d5c946c08450f232d49c51cf [term 1 FOLLOWER]: Leader election won for term 1
I20260809 12:10:25.226512 5989 raft_consensus.cc:686] T 00000000000000000000000000000000 P 39893086d5c946c08450f232d49c51cf [term 1 LEADER]: Becoming Leader. State: Replica: 39893086d5c946c08450f232d49c51cf, State: Running, Role: LEADER
I20260809 12:10:25.227201 5989 consensus_queue.cc:237] T 00000000000000000000000000000000 P 39893086d5c946c08450f232d49c51cf [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "39893086d5c946c08450f232d49c51cf" member_type: VOTER last_known_addr { host: "127.0.222.62" port: 38945 } }
I20260809 12:10:25.227964 5984 sys_catalog.cc:564] T 00000000000000000000000000000000 P 39893086d5c946c08450f232d49c51cf [sys.catalog]: configured and running, proceeding with master startup.
I20260809 12:10:25.230307 5990 sys_catalog.cc:455] T 00000000000000000000000000000000 P 39893086d5c946c08450f232d49c51cf [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "39893086d5c946c08450f232d49c51cf" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "39893086d5c946c08450f232d49c51cf" member_type: VOTER last_known_addr { host: "127.0.222.62" port: 38945 } } }
I20260809 12:10:25.230305 5991 sys_catalog.cc:455] T 00000000000000000000000000000000 P 39893086d5c946c08450f232d49c51cf [sys.catalog]: SysCatalogTable state changed. Reason: New leader 39893086d5c946c08450f232d49c51cf. Latest consensus state: current_term: 1 leader_uuid: "39893086d5c946c08450f232d49c51cf" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "39893086d5c946c08450f232d49c51cf" member_type: VOTER last_known_addr { host: "127.0.222.62" port: 38945 } } }
I20260809 12:10:25.230759 5990 sys_catalog.cc:458] T 00000000000000000000000000000000 P 39893086d5c946c08450f232d49c51cf [sys.catalog]: This master's current role is: LEADER
I20260809 12:10:25.230895 5991 sys_catalog.cc:458] T 00000000000000000000000000000000 P 39893086d5c946c08450f232d49c51cf [sys.catalog]: This master's current role is: LEADER
I20260809 12:10:25.232542 5996 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260809 12:10:25.238261 5996 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260809 12:10:25.248262 5996 catalog_manager.cc:1347] Generated new cluster ID: 71a9a15398f34e0a8f318a4d1a612d72
I20260809 12:10:25.248384 5996 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260809 12:10:25.266484 5996 catalog_manager.cc:1370] Generated new certificate authority record
I20260809 12:10:25.267028 5996 catalog_manager.cc:1504] Loading token signing keys...
I20260809 12:10:25.282987 5996 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 39893086d5c946c08450f232d49c51cf: Generated new TSK 0
I20260809 12:10:25.283810 5996 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260809 12:10:25.298192 888 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskgH5Pjm/build/asan/bin/kudu
/tmp/dist-test-taskgH5Pjm/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskgH5Pjm/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786277382073938-888-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskgH5Pjm/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786277382073938-888-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskgH5Pjm/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786277382073938-888-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskgH5Pjm/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786277382073938-888-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.222.1:0
--local_ip_for_outbound_sockets=127.0.222.1
--webserver_interface=127.0.222.1
--webserver_port=0
--tserver_master_addrs=127.0.222.62:38945
--builtin_ntp_servers=127.0.222.20:38877
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled
--rpc_encryption=disabled with env {}
W20260809 12:10:25.644929 6010 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260809 12:10:25.645201 6010 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260809 12:10:25.645289 6010 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260809 12:10:25.652472 6010 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260809 12:10:25.652570 6010 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260809 12:10:25.652707 6010 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.222.1
I20260809 12:10:25.661757 6010 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.222.20:38877
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskgH5Pjm/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786277382073938-888-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskgH5Pjm/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786277382073938-888-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.222.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskgH5Pjm/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786277382073938-888-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.222.1
--webserver_port=0
--tserver_master_addrs=127.0.222.62:38945
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.222.1
--log_dir=/tmp/dist-test-taskgH5Pjm/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786277382073938-888-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 09 Aug 2026 11:06:41 UTC on ccycloud.quasar-bwoogy.root.comops.site
build id 4575991
ASAN enabled
I20260809 12:10:25.663079 6010 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260809 12:10:25.664865 6010 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260809 12:10:25.678828 6016 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 12:10:25.678881 6017 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 12:10:25.678917 6019 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260809 12:10:25.679679 6010 server_base.cc:1034] running on GCE node
I20260809 12:10:25.680338 6010 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260809 12:10:25.681321 6010 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260809 12:10:25.682571 6010 hybrid_clock.cc:648] HybridClock initialized: now 1786277425682528 us; error 72 us; skew 500 ppm
I20260809 12:10:25.682914 6010 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260809 12:10:25.685159 6010 webserver.cc:460] Webserver started at http://127.0.222.1:37851/ using document root <none> and password file <none>
I20260809 12:10:25.685849 6010 fs_manager.cc:359] Metadata directory not provided
I20260809 12:10:25.685992 6010 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260809 12:10:25.686317 6010 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260809 12:10:25.689383 6010 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskgH5Pjm/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786277382073938-888-0/minicluster-data/ts-0/data/instance:
uuid: "6dbe0cd9d5eb454e8ab7c80039960221"
format_stamp: "Formatted at 2026-08-09 12:10:25 on dist-test-slave-2w3w"
server_key: "9fc0fa956234d0fc7c02b3b2559bf7fb"
server_key_iv: "6d638e22c6904c8f400a81441bf29362"
server_key_version: "encryptionkey@0"
I20260809 12:10:25.690224 6010 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskgH5Pjm/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786277382073938-888-0/minicluster-data/ts-0/wal/instance:
uuid: "6dbe0cd9d5eb454e8ab7c80039960221"
format_stamp: "Formatted at 2026-08-09 12:10:25 on dist-test-slave-2w3w"
server_key: "9fc0fa956234d0fc7c02b3b2559bf7fb"
server_key_iv: "6d638e22c6904c8f400a81441bf29362"
server_key_version: "encryptionkey@0"
I20260809 12:10:25.695456 6010 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.004s sys 0.003s
I20260809 12:10:25.698869 6025 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260809 12:10:25.700215 6010 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260809 12:10:25.700405 6010 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskgH5Pjm/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786277382073938-888-0/minicluster-data/ts-0/data,/tmp/dist-test-taskgH5Pjm/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786277382073938-888-0/minicluster-data/ts-0/wal
uuid: "6dbe0cd9d5eb454e8ab7c80039960221"
format_stamp: "Formatted at 2026-08-09 12:10:25 on dist-test-slave-2w3w"
server_key: "9fc0fa956234d0fc7c02b3b2559bf7fb"
server_key_iv: "6d638e22c6904c8f400a81441bf29362"
server_key_version: "encryptionkey@0"
I20260809 12:10:25.700644 6010 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskgH5Pjm/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786277382073938-888-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskgH5Pjm/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786277382073938-888-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskgH5Pjm/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786277382073938-888-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260809 12:10:25.706359 6010 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260809 12:10:25.707221 6010 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260809 12:10:25.707504 6010 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260809 12:10:25.708550 6010 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260809 12:10:25.709959 6010 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260809 12:10:25.710091 6010 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260809 12:10:25.710242 6010 ts_tablet_manager.cc:610] Registered 0 tablets
I20260809 12:10:25.710332 6010 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260809 12:10:25.749800 6010 rpc_server.cc:307] RPC server started. Bound to: 127.0.222.1:44611
I20260809 12:10:25.749893 6146 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.222.1:44611 every 8 connection(s)
I20260809 12:10:25.751191 6010 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskgH5Pjm/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786277382073938-888-0/minicluster-data/ts-0/data/info.pb
I20260809 12:10:25.752208 888 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskgH5Pjm/build/asan/bin/kudu as pid 6010
I20260809 12:10:25.752420 888 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskgH5Pjm/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786277382073938-888-0/minicluster-data/ts-0/wal/instance
I20260809 12:10:25.753250 888 external_mini_cluster.cc:1428] Setting key b5ead0bf481efad6562899987fb1ddd1
I20260809 12:10:25.756440 888 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskgH5Pjm/build/asan/bin/kudu
/tmp/dist-test-taskgH5Pjm/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskgH5Pjm/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786277382073938-888-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskgH5Pjm/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786277382073938-888-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskgH5Pjm/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786277382073938-888-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskgH5Pjm/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786277382073938-888-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.222.2:0
--local_ip_for_outbound_sockets=127.0.222.2
--webserver_interface=127.0.222.2
--webserver_port=0
--tserver_master_addrs=127.0.222.62:38945
--builtin_ntp_servers=127.0.222.20:38877
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled
--rpc_encryption=disabled with env {}
I20260809 12:10:25.760329 6150 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 12:10:25.754179 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.222.1:47941 (local address 127.0.222.62:38945)
0809 12:10:25.754671 (+ 492us) server_negotiation.cc:206] Beginning negotiation
0809 12:10:25.754675 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0809 12:10:25.756167 (+ 1492us) server_negotiation.cc:407] Connection header received
0809 12:10:25.757908 (+ 1741us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 12:10:25.757917 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 12:10:25.757985 (+ 68us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 12:10:25.758109 (+ 124us) server_negotiation.cc:226] Negotiated authn=SASL
0809 12:10:25.759449 (+ 1340us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 12:10:25.759457 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 12:10:25.759465 (+ 8us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0809 12:10:25.759488 (+ 23us) server_negotiation.cc:922] calling sasl_server_start()
0809 12:10:25.759544 (+ 56us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0809 12:10:25.759565 (+ 21us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 12:10:25.759699 (+ 134us) server_negotiation.cc:1035] Waiting for connection context
0809 12:10:25.759868 (+ 169us) server_negotiation.cc:299] Negotiation successful
0809 12:10:25.759938 (+ 70us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":387,"thread_start_us":276,"threads_started":1}
I20260809 12:10:25.761207 6151 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 12:10:25.754825 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.222.62:38945 (local address 127.0.222.1:47941)
0809 12:10:25.755976 (+ 1151us) negotiation.cc:107] Waiting for socket to connect
0809 12:10:25.756032 (+ 56us) client_negotiation.cc:174] Beginning negotiation
0809 12:10:25.757641 (+ 1609us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 12:10:25.758304 (+ 663us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 12:10:25.758312 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 12:10:25.758396 (+ 84us) client_negotiation.cc:189] Negotiated authn=SASL
0809 12:10:25.759147 (+ 751us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0809 12:10:25.759150 (+ 3us) client_negotiation.cc:656] calling sasl_client_start()
0809 12:10:25.759186 (+ 36us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0809 12:10:25.759268 (+ 82us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0809 12:10:25.759671 (+ 403us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0809 12:10:25.759679 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0809 12:10:25.759685 (+ 6us) client_negotiation.cc:769] Sending connection context
0809 12:10:25.759900 (+ 215us) client_negotiation.cc:240] Negotiation successful
0809 12:10:25.760104 (+ 204us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":618,"thread_start_us":387,"threads_started":1}
I20260809 12:10:25.763479 6147 heartbeater.cc:347] Connected to a master server at 127.0.222.62:38945
I20260809 12:10:25.763779 6147 heartbeater.cc:464] Registering TS with master...
I20260809 12:10:25.764367 6147 heartbeater.cc:511] Master 127.0.222.62:38945 requested a full tablet report, sending...
I20260809 12:10:25.766635 5948 ts_manager.cc:194] Registered new tserver with Master: 6dbe0cd9d5eb454e8ab7c80039960221 (127.0.222.1:44611)
W20260809 12:10:26.091071 6152 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260809 12:10:26.091365 6152 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260809 12:10:26.091553 6152 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260809 12:10:26.098860 6152 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260809 12:10:26.099010 6152 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260809 12:10:26.099211 6152 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.222.2
I20260809 12:10:26.108598 6152 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.222.20:38877
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskgH5Pjm/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786277382073938-888-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskgH5Pjm/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786277382073938-888-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.222.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskgH5Pjm/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786277382073938-888-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.222.2
--webserver_port=0
--tserver_master_addrs=127.0.222.62:38945
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.222.2
--log_dir=/tmp/dist-test-taskgH5Pjm/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786277382073938-888-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 09 Aug 2026 11:06:41 UTC on ccycloud.quasar-bwoogy.root.comops.site
build id 4575991
ASAN enabled
I20260809 12:10:26.109915 6152 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260809 12:10:26.111805 6152 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260809 12:10:26.125964 6157 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 12:10:26.125998 6158 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 12:10:26.126322 6162 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260809 12:10:26.127017 6152 server_base.cc:1034] running on GCE node
I20260809 12:10:26.127813 6152 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260809 12:10:26.129004 6152 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260809 12:10:26.130234 6152 hybrid_clock.cc:648] HybridClock initialized: now 1786277426130162 us; error 54 us; skew 500 ppm
I20260809 12:10:26.130599 6152 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260809 12:10:26.133242 6152 webserver.cc:460] Webserver started at http://127.0.222.2:38889/ using document root <none> and password file <none>
I20260809 12:10:26.134104 6152 fs_manager.cc:359] Metadata directory not provided
I20260809 12:10:26.134235 6152 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260809 12:10:26.134598 6152 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260809 12:10:26.137847 6152 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskgH5Pjm/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786277382073938-888-0/minicluster-data/ts-1/data/instance:
uuid: "95d66d0c4b2f4416afdf59f353e0538e"
format_stamp: "Formatted at 2026-08-09 12:10:26 on dist-test-slave-2w3w"
server_key: "bbe6ca7140601f95877c810bc0b50af3"
server_key_iv: "9d1281e8c0175e93f91a77534002f18f"
server_key_version: "encryptionkey@0"
I20260809 12:10:26.138741 6152 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskgH5Pjm/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786277382073938-888-0/minicluster-data/ts-1/wal/instance:
uuid: "95d66d0c4b2f4416afdf59f353e0538e"
format_stamp: "Formatted at 2026-08-09 12:10:26 on dist-test-slave-2w3w"
server_key: "bbe6ca7140601f95877c810bc0b50af3"
server_key_iv: "9d1281e8c0175e93f91a77534002f18f"
server_key_version: "encryptionkey@0"
I20260809 12:10:26.144232 6152 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.001s sys 0.003s
I20260809 12:10:26.147801 6168 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260809 12:10:26.149277 6152 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260809 12:10:26.149467 6152 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskgH5Pjm/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786277382073938-888-0/minicluster-data/ts-1/data,/tmp/dist-test-taskgH5Pjm/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786277382073938-888-0/minicluster-data/ts-1/wal
uuid: "95d66d0c4b2f4416afdf59f353e0538e"
format_stamp: "Formatted at 2026-08-09 12:10:26 on dist-test-slave-2w3w"
server_key: "bbe6ca7140601f95877c810bc0b50af3"
server_key_iv: "9d1281e8c0175e93f91a77534002f18f"
server_key_version: "encryptionkey@0"
I20260809 12:10:26.149724 6152 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskgH5Pjm/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786277382073938-888-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskgH5Pjm/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786277382073938-888-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskgH5Pjm/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786277382073938-888-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260809 12:10:26.155439 6152 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260809 12:10:26.156247 6152 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260809 12:10:26.156525 6152 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260809 12:10:26.157446 6152 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260809 12:10:26.158820 6152 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260809 12:10:26.158946 6152 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260809 12:10:26.159169 6152 ts_tablet_manager.cc:610] Registered 0 tablets
I20260809 12:10:26.159271 6152 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260809 12:10:26.193588 6152 rpc_server.cc:307] RPC server started. Bound to: 127.0.222.2:46145
I20260809 12:10:26.193630 6287 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.222.2:46145 every 8 connection(s)
I20260809 12:10:26.195013 6152 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskgH5Pjm/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786277382073938-888-0/minicluster-data/ts-1/data/info.pb
I20260809 12:10:26.201892 888 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskgH5Pjm/build/asan/bin/kudu as pid 6152
I20260809 12:10:26.202073 888 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskgH5Pjm/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786277382073938-888-0/minicluster-data/ts-1/wal/instance
I20260809 12:10:26.202622 888 external_mini_cluster.cc:1428] Setting key 91cce05b6a4a35bfad56ab21ea9f20d9
I20260809 12:10:26.203949 6150 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 12:10:26.197788 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.222.2:59001 (local address 127.0.222.62:38945)
0809 12:10:26.198003 (+ 215us) server_negotiation.cc:206] Beginning negotiation
0809 12:10:26.198009 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0809 12:10:26.199672 (+ 1663us) server_negotiation.cc:407] Connection header received
0809 12:10:26.201381 (+ 1709us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 12:10:26.201386 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 12:10:26.201504 (+ 118us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 12:10:26.201623 (+ 119us) server_negotiation.cc:226] Negotiated authn=SASL
0809 12:10:26.203154 (+ 1531us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 12:10:26.203160 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 12:10:26.203165 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0809 12:10:26.203185 (+ 20us) server_negotiation.cc:922] calling sasl_server_start()
0809 12:10:26.203232 (+ 47us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0809 12:10:26.203248 (+ 16us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 12:10:26.203384 (+ 136us) server_negotiation.cc:1035] Waiting for connection context
0809 12:10:26.203623 (+ 239us) server_negotiation.cc:299] Negotiation successful
0809 12:10:26.203686 (+ 63us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":78}
I20260809 12:10:26.205085 6290 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 12:10:26.198377 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.222.62:38945 (local address 127.0.222.2:59001)
0809 12:10:26.199495 (+ 1118us) negotiation.cc:107] Waiting for socket to connect
0809 12:10:26.199542 (+ 47us) client_negotiation.cc:174] Beginning negotiation
0809 12:10:26.201094 (+ 1552us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 12:10:26.201851 (+ 757us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 12:10:26.201859 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 12:10:26.201948 (+ 89us) client_negotiation.cc:189] Negotiated authn=SASL
0809 12:10:26.202787 (+ 839us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0809 12:10:26.202789 (+ 2us) client_negotiation.cc:656] calling sasl_client_start()
0809 12:10:26.202838 (+ 49us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0809 12:10:26.202925 (+ 87us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0809 12:10:26.203359 (+ 434us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0809 12:10:26.203371 (+ 12us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0809 12:10:26.203380 (+ 9us) client_negotiation.cc:769] Sending connection context
0809 12:10:26.203693 (+ 313us) client_negotiation.cc:240] Negotiation successful
0809 12:10:26.203944 (+ 251us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":579,"thread_start_us":352,"threads_started":1}
I20260809 12:10:26.206187 888 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskgH5Pjm/build/asan/bin/kudu
/tmp/dist-test-taskgH5Pjm/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskgH5Pjm/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786277382073938-888-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskgH5Pjm/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786277382073938-888-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskgH5Pjm/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786277382073938-888-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskgH5Pjm/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786277382073938-888-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.222.3:0
--local_ip_for_outbound_sockets=127.0.222.3
--webserver_interface=127.0.222.3
--webserver_port=0
--tserver_master_addrs=127.0.222.62:38945
--builtin_ntp_servers=127.0.222.20:38877
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled
--rpc_encryption=disabled with env {}
I20260809 12:10:26.206617 6288 heartbeater.cc:347] Connected to a master server at 127.0.222.62:38945
I20260809 12:10:26.206941 6288 heartbeater.cc:464] Registering TS with master...
I20260809 12:10:26.207625 6288 heartbeater.cc:511] Master 127.0.222.62:38945 requested a full tablet report, sending...
I20260809 12:10:26.209035 5948 ts_manager.cc:194] Registered new tserver with Master: 95d66d0c4b2f4416afdf59f353e0538e (127.0.222.2:46145)
W20260809 12:10:26.539227 6292 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260809 12:10:26.539511 6292 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260809 12:10:26.539673 6292 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260809 12:10:26.546947 6292 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260809 12:10:26.547123 6292 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260809 12:10:26.547344 6292 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.222.3
I20260809 12:10:26.556906 6292 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.222.20:38877
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskgH5Pjm/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786277382073938-888-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskgH5Pjm/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786277382073938-888-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.222.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskgH5Pjm/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786277382073938-888-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.222.3
--webserver_port=0
--tserver_master_addrs=127.0.222.62:38945
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.222.3
--log_dir=/tmp/dist-test-taskgH5Pjm/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786277382073938-888-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 09 Aug 2026 11:06:41 UTC on ccycloud.quasar-bwoogy.root.comops.site
build id 4575991
ASAN enabled
I20260809 12:10:26.558254 6292 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260809 12:10:26.560194 6292 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260809 12:10:26.573669 6299 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 12:10:26.573725 6302 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 12:10:26.573637 6298 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260809 12:10:26.574688 6292 server_base.cc:1034] running on GCE node
I20260809 12:10:26.575364 6292 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260809 12:10:26.576315 6292 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260809 12:10:26.577597 6292 hybrid_clock.cc:648] HybridClock initialized: now 1786277426577547 us; error 58 us; skew 500 ppm
I20260809 12:10:26.577986 6292 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260809 12:10:26.580374 6292 webserver.cc:460] Webserver started at http://127.0.222.3:36093/ using document root <none> and password file <none>
I20260809 12:10:26.581085 6292 fs_manager.cc:359] Metadata directory not provided
I20260809 12:10:26.581221 6292 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260809 12:10:26.581564 6292 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260809 12:10:26.584640 6292 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskgH5Pjm/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786277382073938-888-0/minicluster-data/ts-2/data/instance:
uuid: "8a8a59f6be334a37b5fd635eb046cd88"
format_stamp: "Formatted at 2026-08-09 12:10:26 on dist-test-slave-2w3w"
server_key: "6ba00d0a4838a422eb1b14c95dd5d5ac"
server_key_iv: "a226f0371e7b37ab0964d83ba51bfd70"
server_key_version: "encryptionkey@0"
I20260809 12:10:26.585529 6292 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskgH5Pjm/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786277382073938-888-0/minicluster-data/ts-2/wal/instance:
uuid: "8a8a59f6be334a37b5fd635eb046cd88"
format_stamp: "Formatted at 2026-08-09 12:10:26 on dist-test-slave-2w3w"
server_key: "6ba00d0a4838a422eb1b14c95dd5d5ac"
server_key_iv: "a226f0371e7b37ab0964d83ba51bfd70"
server_key_version: "encryptionkey@0"
I20260809 12:10:26.591024 6292 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.007s sys 0.000s
I20260809 12:10:26.594521 6308 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260809 12:10:26.595914 6292 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260809 12:10:26.596107 6292 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskgH5Pjm/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786277382073938-888-0/minicluster-data/ts-2/data,/tmp/dist-test-taskgH5Pjm/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786277382073938-888-0/minicluster-data/ts-2/wal
uuid: "8a8a59f6be334a37b5fd635eb046cd88"
format_stamp: "Formatted at 2026-08-09 12:10:26 on dist-test-slave-2w3w"
server_key: "6ba00d0a4838a422eb1b14c95dd5d5ac"
server_key_iv: "a226f0371e7b37ab0964d83ba51bfd70"
server_key_version: "encryptionkey@0"
I20260809 12:10:26.596364 6292 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskgH5Pjm/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786277382073938-888-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskgH5Pjm/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786277382073938-888-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskgH5Pjm/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786277382073938-888-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260809 12:10:26.602082 6292 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260809 12:10:26.602923 6292 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260809 12:10:26.603232 6292 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260809 12:10:26.604303 6292 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260809 12:10:26.605856 6292 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260809 12:10:26.605947 6292 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260809 12:10:26.606031 6292 ts_tablet_manager.cc:610] Registered 0 tablets
I20260809 12:10:26.606086 6292 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260809 12:10:26.641109 6292 rpc_server.cc:307] RPC server started. Bound to: 127.0.222.3:34633
I20260809 12:10:26.641165 6429 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.222.3:34633 every 8 connection(s)
I20260809 12:10:26.642515 6292 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskgH5Pjm/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786277382073938-888-0/minicluster-data/ts-2/data/info.pb
I20260809 12:10:26.649839 888 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskgH5Pjm/build/asan/bin/kudu as pid 6292
I20260809 12:10:26.650033 888 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskgH5Pjm/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786277382073938-888-0/minicluster-data/ts-2/wal/instance
I20260809 12:10:26.650501 888 external_mini_cluster.cc:1428] Setting key 418a272062128e08c1313ee377ffff86
I20260809 12:10:26.651218 6150 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 12:10:26.645206 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.222.3:46565 (local address 127.0.222.62:38945)
0809 12:10:26.645446 (+ 240us) server_negotiation.cc:206] Beginning negotiation
0809 12:10:26.645450 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0809 12:10:26.646938 (+ 1488us) server_negotiation.cc:407] Connection header received
0809 12:10:26.648739 (+ 1801us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 12:10:26.648743 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 12:10:26.648807 (+ 64us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 12:10:26.648898 (+ 91us) server_negotiation.cc:226] Negotiated authn=SASL
0809 12:10:26.650397 (+ 1499us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 12:10:26.650403 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 12:10:26.650407 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0809 12:10:26.650434 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0809 12:10:26.650466 (+ 32us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0809 12:10:26.650479 (+ 13us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 12:10:26.650629 (+ 150us) server_negotiation.cc:1035] Waiting for connection context
0809 12:10:26.650825 (+ 196us) server_negotiation.cc:299] Negotiation successful
0809 12:10:26.650903 (+ 78us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":107}
I20260809 12:10:26.652307 6432 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 12:10:26.645725 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.222.62:38945 (local address 127.0.222.3:46565)
0809 12:10:26.646713 (+ 988us) negotiation.cc:107] Waiting for socket to connect
0809 12:10:26.646749 (+ 36us) client_negotiation.cc:174] Beginning negotiation
0809 12:10:26.648482 (+ 1733us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 12:10:26.649112 (+ 630us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 12:10:26.649120 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 12:10:26.649212 (+ 92us) client_negotiation.cc:189] Negotiated authn=SASL
0809 12:10:26.650029 (+ 817us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0809 12:10:26.650031 (+ 2us) client_negotiation.cc:656] calling sasl_client_start()
0809 12:10:26.650070 (+ 39us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0809 12:10:26.650204 (+ 134us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0809 12:10:26.650581 (+ 377us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0809 12:10:26.650591 (+ 10us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0809 12:10:26.650601 (+ 10us) client_negotiation.cc:769] Sending connection context
0809 12:10:26.650906 (+ 305us) client_negotiation.cc:240] Negotiation successful
0809 12:10:26.651193 (+ 287us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":544,"thread_start_us":331,"threads_started":1}
I20260809 12:10:26.653764 6430 heartbeater.cc:347] Connected to a master server at 127.0.222.62:38945
I20260809 12:10:26.654091 6430 heartbeater.cc:464] Registering TS with master...
I20260809 12:10:26.654731 6430 heartbeater.cc:511] Master 127.0.222.62:38945 requested a full tablet report, sending...
I20260809 12:10:26.656344 5948 ts_manager.cc:194] Registered new tserver with Master: 8a8a59f6be334a37b5fd635eb046cd88 (127.0.222.3:34633)
I20260809 12:10:26.666455 888 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260809 12:10:26.674412 6150 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 12:10:26.671695 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46698 (local address 127.0.222.62:38945)
0809 12:10:26.671964 (+ 269us) server_negotiation.cc:206] Beginning negotiation
0809 12:10:26.671976 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0809 12:10:26.672389 (+ 413us) server_negotiation.cc:407] Connection header received
0809 12:10:26.672536 (+ 147us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 12:10:26.672540 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 12:10:26.672598 (+ 58us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 12:10:26.672687 (+ 89us) server_negotiation.cc:226] Negotiated authn=SASL
0809 12:10:26.673566 (+ 879us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 12:10:26.673570 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 12:10:26.673574 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0809 12:10:26.673594 (+ 20us) server_negotiation.cc:922] calling sasl_server_start()
0809 12:10:26.673636 (+ 42us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0809 12:10:26.673649 (+ 13us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 12:10:26.673766 (+ 117us) server_negotiation.cc:1035] Waiting for connection context
0809 12:10:26.673962 (+ 196us) server_negotiation.cc:299] Negotiation successful
0809 12:10:26.674016 (+ 54us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":80}
I20260809 12:10:26.678661 5948 catalog_manager.cc:1908] Servicing CreateTable request from {username='slave'} at 127.0.0.1:46698:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260809 12:10:26.685509 5948 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260809 12:10:26.715837 6444 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 12:10:26.698736 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46030 (local address 127.0.222.1:44611)
0809 12:10:26.710275 (+ 11539us) server_negotiation.cc:206] Beginning negotiation
0809 12:10:26.710287 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0809 12:10:26.710331 (+ 44us) server_negotiation.cc:407] Connection header received
0809 12:10:26.710428 (+ 97us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 12:10:26.710441 (+ 13us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 12:10:26.710770 (+ 329us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 12:10:26.712838 (+ 2068us) server_negotiation.cc:226] Negotiated authn=SASL
0809 12:10:26.714160 (+ 1322us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 12:10:26.714181 (+ 21us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 12:10:26.714201 (+ 20us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0809 12:10:26.714236 (+ 35us) server_negotiation.cc:922] calling sasl_server_start()
0809 12:10:26.714304 (+ 68us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0809 12:10:26.714333 (+ 29us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 12:10:26.714534 (+ 201us) server_negotiation.cc:1035] Waiting for connection context
0809 12:10:26.715123 (+ 589us) server_negotiation.cc:299] Negotiation successful
0809 12:10:26.715239 (+ 116us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":11317,"thread_start_us":11142,"threads_started":1}
I20260809 12:10:26.716092 6445 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 12:10:26.699462 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55470 (local address 127.0.222.2:46145)
0809 12:10:26.712844 (+ 13382us) server_negotiation.cc:206] Beginning negotiation
0809 12:10:26.712856 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0809 12:10:26.712896 (+ 40us) server_negotiation.cc:407] Connection header received
0809 12:10:26.712981 (+ 85us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 12:10:26.712991 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 12:10:26.713267 (+ 276us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 12:10:26.713533 (+ 266us) server_negotiation.cc:226] Negotiated authn=SASL
0809 12:10:26.714509 (+ 976us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 12:10:26.714518 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 12:10:26.714530 (+ 12us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0809 12:10:26.714557 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0809 12:10:26.714618 (+ 61us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0809 12:10:26.714644 (+ 26us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 12:10:26.715574 (+ 930us) server_negotiation.cc:1035] Waiting for connection context
0809 12:10:26.715648 (+ 74us) server_negotiation.cc:299] Negotiation successful
0809 12:10:26.715743 (+ 95us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":13200,"thread_start_us":13061,"threads_started":1}
I20260809 12:10:26.716341 6443 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 12:10:26.698520 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.222.1:44611 (local address 127.0.0.1:46030)
0809 12:10:26.700236 (+ 1716us) negotiation.cc:107] Waiting for socket to connect
0809 12:10:26.700274 (+ 38us) client_negotiation.cc:174] Beginning negotiation
0809 12:10:26.700542 (+ 268us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 12:10:26.711109 (+ 10567us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 12:10:26.711121 (+ 12us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 12:10:26.711262 (+ 141us) client_negotiation.cc:189] Negotiated authn=SASL
0809 12:10:26.712422 (+ 1160us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0809 12:10:26.712428 (+ 6us) client_negotiation.cc:656] calling sasl_client_start()
0809 12:10:26.712479 (+ 51us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0809 12:10:26.712592 (+ 113us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0809 12:10:26.714489 (+ 1897us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0809 12:10:26.714502 (+ 13us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0809 12:10:26.714510 (+ 8us) client_negotiation.cc:769] Sending connection context
0809 12:10:26.714737 (+ 227us) client_negotiation.cc:240] Negotiation successful
0809 12:10:26.714891 (+ 154us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1409,"thread_start_us":13157,"threads_started":1}
I20260809 12:10:26.716504 6442 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 12:10:26.698618 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.222.2:46145 (local address 127.0.0.1:55470)
0809 12:10:26.712063 (+ 13445us) negotiation.cc:107] Waiting for socket to connect
0809 12:10:26.712093 (+ 30us) client_negotiation.cc:174] Beginning negotiation
0809 12:10:26.712230 (+ 137us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 12:10:26.713522 (+ 1292us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 12:10:26.713527 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 12:10:26.713553 (+ 26us) client_negotiation.cc:189] Negotiated authn=SASL
0809 12:10:26.714234 (+ 681us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0809 12:10:26.714238 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0809 12:10:26.714255 (+ 17us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0809 12:10:26.714305 (+ 50us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0809 12:10:26.714807 (+ 502us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0809 12:10:26.714812 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0809 12:10:26.714817 (+ 5us) client_negotiation.cc:769] Sending connection context
0809 12:10:26.714992 (+ 175us) client_negotiation.cc:240] Negotiation successful
0809 12:10:26.715041 (+ 49us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":13240,"thread_start_us":1137,"threads_started":1}
I20260809 12:10:26.717519 6446 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 12:10:26.700603 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.222.3:34633 (local address 127.0.0.1:36590)
0809 12:10:26.713155 (+ 12552us) negotiation.cc:107] Waiting for socket to connect
0809 12:10:26.713173 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0809 12:10:26.713292 (+ 119us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 12:10:26.714637 (+ 1345us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 12:10:26.714642 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 12:10:26.714670 (+ 28us) client_negotiation.cc:189] Negotiated authn=SASL
0809 12:10:26.716346 (+ 1676us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0809 12:10:26.716349 (+ 3us) client_negotiation.cc:656] calling sasl_client_start()
0809 12:10:26.716367 (+ 18us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0809 12:10:26.716419 (+ 52us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0809 12:10:26.716861 (+ 442us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0809 12:10:26.716866 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0809 12:10:26.716871 (+ 5us) client_negotiation.cc:769] Sending connection context
0809 12:10:26.717017 (+ 146us) client_negotiation.cc:240] Negotiation successful
0809 12:10:26.717073 (+ 56us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":12435,"thread_start_us":12292,"threads_started":1}
I20260809 12:10:26.717512 6448 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 12:10:26.701223 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36590 (local address 127.0.222.3:34633)
0809 12:10:26.709864 (+ 8641us) server_negotiation.cc:206] Beginning negotiation
0809 12:10:26.709874 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0809 12:10:26.713310 (+ 3436us) server_negotiation.cc:407] Connection header received
0809 12:10:26.713415 (+ 105us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 12:10:26.713424 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 12:10:26.713625 (+ 201us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 12:10:26.713794 (+ 169us) server_negotiation.cc:226] Negotiated authn=SASL
0809 12:10:26.716628 (+ 2834us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 12:10:26.716636 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 12:10:26.716648 (+ 12us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0809 12:10:26.716666 (+ 18us) server_negotiation.cc:922] calling sasl_server_start()
0809 12:10:26.716722 (+ 56us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0809 12:10:26.716743 (+ 21us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 12:10:26.716950 (+ 207us) server_negotiation.cc:1035] Waiting for connection context
0809 12:10:26.717030 (+ 80us) server_negotiation.cc:299] Negotiation successful
0809 12:10:26.717124 (+ 94us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":8453,"thread_start_us":8261,"threads_started":1}
I20260809 12:10:26.720585 6080 tablet_service.cc:1467] Processing CreateTablet for tablet 57c28e4572344489bd42bf75cd9633f6 (DEFAULT_TABLE table=test-table [id=0827bcb23dfa4ff78f49bf32670aeeb7]), partition=RANGE (key) PARTITION UNBOUNDED
I20260809 12:10:26.722038 6080 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 57c28e4572344489bd42bf75cd9633f6. 1 dirs total, 0 dirs full, 0 dirs failed
I20260809 12:10:26.721599 6218 tablet_service.cc:1467] Processing CreateTablet for tablet 57c28e4572344489bd42bf75cd9633f6 (DEFAULT_TABLE table=test-table [id=0827bcb23dfa4ff78f49bf32670aeeb7]), partition=RANGE (key) PARTITION UNBOUNDED
I20260809 12:10:26.721599 6361 tablet_service.cc:1467] Processing CreateTablet for tablet 57c28e4572344489bd42bf75cd9633f6 (DEFAULT_TABLE table=test-table [id=0827bcb23dfa4ff78f49bf32670aeeb7]), partition=RANGE (key) PARTITION UNBOUNDED
I20260809 12:10:26.723182 6218 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 57c28e4572344489bd42bf75cd9633f6. 1 dirs total, 0 dirs full, 0 dirs failed
I20260809 12:10:26.723181 6361 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 57c28e4572344489bd42bf75cd9633f6. 1 dirs total, 0 dirs full, 0 dirs failed
I20260809 12:10:26.734359 6449 tablet_bootstrap.cc:492] T 57c28e4572344489bd42bf75cd9633f6 P 6dbe0cd9d5eb454e8ab7c80039960221: Bootstrap starting.
I20260809 12:10:26.737953 6449 tablet_bootstrap.cc:654] T 57c28e4572344489bd42bf75cd9633f6 P 6dbe0cd9d5eb454e8ab7c80039960221: Neither blocks nor log segments found. Creating new log.
I20260809 12:10:26.739796 6450 tablet_bootstrap.cc:492] T 57c28e4572344489bd42bf75cd9633f6 P 8a8a59f6be334a37b5fd635eb046cd88: Bootstrap starting.
I20260809 12:10:26.739856 6449 log.cc:824] T 57c28e4572344489bd42bf75cd9633f6 P 6dbe0cd9d5eb454e8ab7c80039960221: Log is configured to *not* fsync() on all Append() calls
I20260809 12:10:26.743031 6451 tablet_bootstrap.cc:492] T 57c28e4572344489bd42bf75cd9633f6 P 95d66d0c4b2f4416afdf59f353e0538e: Bootstrap starting.
I20260809 12:10:26.743821 6450 tablet_bootstrap.cc:654] T 57c28e4572344489bd42bf75cd9633f6 P 8a8a59f6be334a37b5fd635eb046cd88: Neither blocks nor log segments found. Creating new log.
I20260809 12:10:26.743964 6449 tablet_bootstrap.cc:492] T 57c28e4572344489bd42bf75cd9633f6 P 6dbe0cd9d5eb454e8ab7c80039960221: No bootstrap required, opened a new log
I20260809 12:10:26.744273 6449 ts_tablet_manager.cc:1397] T 57c28e4572344489bd42bf75cd9633f6 P 6dbe0cd9d5eb454e8ab7c80039960221: Time spent bootstrapping tablet: real 0.010s user 0.003s sys 0.005s
I20260809 12:10:26.745225 6450 log.cc:824] T 57c28e4572344489bd42bf75cd9633f6 P 8a8a59f6be334a37b5fd635eb046cd88: Log is configured to *not* fsync() on all Append() calls
I20260809 12:10:26.746227 6451 tablet_bootstrap.cc:654] T 57c28e4572344489bd42bf75cd9633f6 P 95d66d0c4b2f4416afdf59f353e0538e: Neither blocks nor log segments found. Creating new log.
I20260809 12:10:26.747521 6451 log.cc:824] T 57c28e4572344489bd42bf75cd9633f6 P 95d66d0c4b2f4416afdf59f353e0538e: Log is configured to *not* fsync() on all Append() calls
I20260809 12:10:26.749460 6450 tablet_bootstrap.cc:492] T 57c28e4572344489bd42bf75cd9633f6 P 8a8a59f6be334a37b5fd635eb046cd88: No bootstrap required, opened a new log
I20260809 12:10:26.749789 6450 ts_tablet_manager.cc:1397] T 57c28e4572344489bd42bf75cd9633f6 P 8a8a59f6be334a37b5fd635eb046cd88: Time spent bootstrapping tablet: real 0.010s user 0.006s sys 0.002s
I20260809 12:10:26.750618 6451 tablet_bootstrap.cc:492] T 57c28e4572344489bd42bf75cd9633f6 P 95d66d0c4b2f4416afdf59f353e0538e: No bootstrap required, opened a new log
I20260809 12:10:26.750947 6451 ts_tablet_manager.cc:1397] T 57c28e4572344489bd42bf75cd9633f6 P 95d66d0c4b2f4416afdf59f353e0538e: Time spent bootstrapping tablet: real 0.009s user 0.008s sys 0.000s
I20260809 12:10:26.756122 6449 raft_consensus.cc:348] T 57c28e4572344489bd42bf75cd9633f6 P 6dbe0cd9d5eb454e8ab7c80039960221 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6dbe0cd9d5eb454e8ab7c80039960221" member_type: VOTER last_known_addr { host: "127.0.222.1" port: 44611 } } peers { permanent_uuid: "95d66d0c4b2f4416afdf59f353e0538e" member_type: VOTER last_known_addr { host: "127.0.222.2" port: 46145 } } peers { permanent_uuid: "8a8a59f6be334a37b5fd635eb046cd88" member_type: VOTER last_known_addr { host: "127.0.222.3" port: 34633 } }
I20260809 12:10:26.756639 6449 raft_consensus.cc:374] T 57c28e4572344489bd42bf75cd9633f6 P 6dbe0cd9d5eb454e8ab7c80039960221 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260809 12:10:26.756788 6449 raft_consensus.cc:729] T 57c28e4572344489bd42bf75cd9633f6 P 6dbe0cd9d5eb454e8ab7c80039960221 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6dbe0cd9d5eb454e8ab7c80039960221, State: Initialized, Role: FOLLOWER
I20260809 12:10:26.757584 6449 consensus_queue.cc:260] T 57c28e4572344489bd42bf75cd9633f6 P 6dbe0cd9d5eb454e8ab7c80039960221 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6dbe0cd9d5eb454e8ab7c80039960221" member_type: VOTER last_known_addr { host: "127.0.222.1" port: 44611 } } peers { permanent_uuid: "95d66d0c4b2f4416afdf59f353e0538e" member_type: VOTER last_known_addr { host: "127.0.222.2" port: 46145 } } peers { permanent_uuid: "8a8a59f6be334a37b5fd635eb046cd88" member_type: VOTER last_known_addr { host: "127.0.222.3" port: 34633 } }
I20260809 12:10:26.760960 6451 raft_consensus.cc:348] T 57c28e4572344489bd42bf75cd9633f6 P 95d66d0c4b2f4416afdf59f353e0538e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6dbe0cd9d5eb454e8ab7c80039960221" member_type: VOTER last_known_addr { host: "127.0.222.1" port: 44611 } } peers { permanent_uuid: "95d66d0c4b2f4416afdf59f353e0538e" member_type: VOTER last_known_addr { host: "127.0.222.2" port: 46145 } } peers { permanent_uuid: "8a8a59f6be334a37b5fd635eb046cd88" member_type: VOTER last_known_addr { host: "127.0.222.3" port: 34633 } }
I20260809 12:10:26.761447 6451 raft_consensus.cc:374] T 57c28e4572344489bd42bf75cd9633f6 P 95d66d0c4b2f4416afdf59f353e0538e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260809 12:10:26.761210 6450 raft_consensus.cc:348] T 57c28e4572344489bd42bf75cd9633f6 P 8a8a59f6be334a37b5fd635eb046cd88 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6dbe0cd9d5eb454e8ab7c80039960221" member_type: VOTER last_known_addr { host: "127.0.222.1" port: 44611 } } peers { permanent_uuid: "95d66d0c4b2f4416afdf59f353e0538e" member_type: VOTER last_known_addr { host: "127.0.222.2" port: 46145 } } peers { permanent_uuid: "8a8a59f6be334a37b5fd635eb046cd88" member_type: VOTER last_known_addr { host: "127.0.222.3" port: 34633 } }
I20260809 12:10:26.761581 6451 raft_consensus.cc:729] T 57c28e4572344489bd42bf75cd9633f6 P 95d66d0c4b2f4416afdf59f353e0538e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 95d66d0c4b2f4416afdf59f353e0538e, State: Initialized, Role: FOLLOWER
I20260809 12:10:26.761768 6450 raft_consensus.cc:374] T 57c28e4572344489bd42bf75cd9633f6 P 8a8a59f6be334a37b5fd635eb046cd88 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260809 12:10:26.761951 6450 raft_consensus.cc:729] T 57c28e4572344489bd42bf75cd9633f6 P 8a8a59f6be334a37b5fd635eb046cd88 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8a8a59f6be334a37b5fd635eb046cd88, State: Initialized, Role: FOLLOWER
I20260809 12:10:26.762391 6451 consensus_queue.cc:260] T 57c28e4572344489bd42bf75cd9633f6 P 95d66d0c4b2f4416afdf59f353e0538e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6dbe0cd9d5eb454e8ab7c80039960221" member_type: VOTER last_known_addr { host: "127.0.222.1" port: 44611 } } peers { permanent_uuid: "95d66d0c4b2f4416afdf59f353e0538e" member_type: VOTER last_known_addr { host: "127.0.222.2" port: 46145 } } peers { permanent_uuid: "8a8a59f6be334a37b5fd635eb046cd88" member_type: VOTER last_known_addr { host: "127.0.222.3" port: 34633 } }
I20260809 12:10:26.762719 6450 consensus_queue.cc:260] T 57c28e4572344489bd42bf75cd9633f6 P 8a8a59f6be334a37b5fd635eb046cd88 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6dbe0cd9d5eb454e8ab7c80039960221" member_type: VOTER last_known_addr { host: "127.0.222.1" port: 44611 } } peers { permanent_uuid: "95d66d0c4b2f4416afdf59f353e0538e" member_type: VOTER last_known_addr { host: "127.0.222.2" port: 46145 } } peers { permanent_uuid: "8a8a59f6be334a37b5fd635eb046cd88" member_type: VOTER last_known_addr { host: "127.0.222.3" port: 34633 } }
I20260809 12:10:26.769913 6147 heartbeater.cc:503] Master 127.0.222.62:38945 was elected leader, sending a full tablet report...
I20260809 12:10:26.770782 6451 ts_tablet_manager.cc:1428] T 57c28e4572344489bd42bf75cd9633f6 P 95d66d0c4b2f4416afdf59f353e0538e: Time spent starting tablet: real 0.020s user 0.009s sys 0.008s
I20260809 12:10:26.771169 6288 heartbeater.cc:503] Master 127.0.222.62:38945 was elected leader, sending a full tablet report...
I20260809 12:10:26.776268 6430 heartbeater.cc:503] Master 127.0.222.62:38945 was elected leader, sending a full tablet report...
I20260809 12:10:26.776857 6449 ts_tablet_manager.cc:1428] T 57c28e4572344489bd42bf75cd9633f6 P 6dbe0cd9d5eb454e8ab7c80039960221: Time spent starting tablet: real 0.032s user 0.015s sys 0.008s
I20260809 12:10:26.778400 6450 ts_tablet_manager.cc:1428] T 57c28e4572344489bd42bf75cd9633f6 P 8a8a59f6be334a37b5fd635eb046cd88: Time spent starting tablet: real 0.028s user 0.013s sys 0.013s
I20260809 12:10:26.800902 6458 raft_consensus.cc:482] T 57c28e4572344489bd42bf75cd9633f6 P 8a8a59f6be334a37b5fd635eb046cd88 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260809 12:10:26.801249 6458 raft_consensus.cc:504] T 57c28e4572344489bd42bf75cd9633f6 P 8a8a59f6be334a37b5fd635eb046cd88 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6dbe0cd9d5eb454e8ab7c80039960221" member_type: VOTER last_known_addr { host: "127.0.222.1" port: 44611 } } peers { permanent_uuid: "95d66d0c4b2f4416afdf59f353e0538e" member_type: VOTER last_known_addr { host: "127.0.222.2" port: 46145 } } peers { permanent_uuid: "8a8a59f6be334a37b5fd635eb046cd88" member_type: VOTER last_known_addr { host: "127.0.222.3" port: 34633 } }
I20260809 12:10:26.803053 6458 leader_election.cc:288] T 57c28e4572344489bd42bf75cd9633f6 P 8a8a59f6be334a37b5fd635eb046cd88 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 6dbe0cd9d5eb454e8ab7c80039960221 (127.0.222.1:44611), 95d66d0c4b2f4416afdf59f353e0538e (127.0.222.2:46145)
I20260809 12:10:26.805927 6432 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 12:10:26.803333 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.222.2:46145 (local address 127.0.222.3:54871)
0809 12:10:26.803630 (+ 297us) negotiation.cc:107] Waiting for socket to connect
0809 12:10:26.803656 (+ 26us) client_negotiation.cc:174] Beginning negotiation
0809 12:10:26.803762 (+ 106us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 12:10:26.804299 (+ 537us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 12:10:26.804308 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 12:10:26.804333 (+ 25us) client_negotiation.cc:189] Negotiated authn=SASL
0809 12:10:26.805085 (+ 752us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0809 12:10:26.805088 (+ 3us) client_negotiation.cc:656] calling sasl_client_start()
0809 12:10:26.805104 (+ 16us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0809 12:10:26.805146 (+ 42us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0809 12:10:26.805489 (+ 343us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0809 12:10:26.805493 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0809 12:10:26.805498 (+ 5us) client_negotiation.cc:769] Sending connection context
0809 12:10:26.805599 (+ 101us) client_negotiation.cc:240] Negotiation successful
0809 12:10:26.805656 (+ 57us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":162}
I20260809 12:10:26.805927 6445 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 12:10:26.803718 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.222.3:54871 (local address 127.0.222.2:46145)
0809 12:10:26.803922 (+ 204us) server_negotiation.cc:206] Beginning negotiation
0809 12:10:26.803931 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0809 12:10:26.803954 (+ 23us) server_negotiation.cc:407] Connection header received
0809 12:10:26.804043 (+ 89us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 12:10:26.804048 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 12:10:26.804114 (+ 66us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 12:10:26.804228 (+ 114us) server_negotiation.cc:226] Negotiated authn=SASL
0809 12:10:26.805321 (+ 1093us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 12:10:26.805326 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 12:10:26.805329 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0809 12:10:26.805345 (+ 16us) server_negotiation.cc:922] calling sasl_server_start()
0809 12:10:26.805378 (+ 33us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0809 12:10:26.805390 (+ 12us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 12:10:26.805519 (+ 129us) server_negotiation.cc:1035] Waiting for connection context
0809 12:10:26.805598 (+ 79us) server_negotiation.cc:299] Negotiation successful
0809 12:10:26.805659 (+ 61us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":53}
I20260809 12:10:26.806663 6444 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 12:10:26.803512 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.222.3:34215 (local address 127.0.222.1:44611)
0809 12:10:26.803730 (+ 218us) server_negotiation.cc:206] Beginning negotiation
0809 12:10:26.803740 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0809 12:10:26.804358 (+ 618us) server_negotiation.cc:407] Connection header received
0809 12:10:26.804616 (+ 258us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 12:10:26.804623 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 12:10:26.804694 (+ 71us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 12:10:26.804796 (+ 102us) server_negotiation.cc:226] Negotiated authn=SASL
0809 12:10:26.805938 (+ 1142us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 12:10:26.805943 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 12:10:26.805948 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0809 12:10:26.805969 (+ 21us) server_negotiation.cc:922] calling sasl_server_start()
0809 12:10:26.806031 (+ 62us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0809 12:10:26.806051 (+ 20us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 12:10:26.806176 (+ 125us) server_negotiation.cc:1035] Waiting for connection context
0809 12:10:26.806357 (+ 181us) server_negotiation.cc:299] Negotiation successful
0809 12:10:26.806419 (+ 62us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":77}
I20260809 12:10:26.806897 6459 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 12:10:26.803456 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.222.1:44611 (local address 127.0.222.3:34215)
0809 12:10:26.804247 (+ 791us) negotiation.cc:107] Waiting for socket to connect
0809 12:10:26.804265 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0809 12:10:26.804450 (+ 185us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 12:10:26.804829 (+ 379us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 12:10:26.804832 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 12:10:26.804885 (+ 53us) client_negotiation.cc:189] Negotiated authn=SASL
0809 12:10:26.805727 (+ 842us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0809 12:10:26.805729 (+ 2us) client_negotiation.cc:656] calling sasl_client_start()
0809 12:10:26.805742 (+ 13us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0809 12:10:26.805779 (+ 37us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0809 12:10:26.806216 (+ 437us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0809 12:10:26.806224 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0809 12:10:26.806229 (+ 5us) client_negotiation.cc:769] Sending connection context
0809 12:10:26.806359 (+ 130us) client_negotiation.cc:240] Negotiation successful
0809 12:10:26.806414 (+ 55us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":644,"thread_start_us":819,"threads_started":1}
I20260809 12:10:26.807258 6238 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "57c28e4572344489bd42bf75cd9633f6" candidate_uuid: "8a8a59f6be334a37b5fd635eb046cd88" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "95d66d0c4b2f4416afdf59f353e0538e" is_pre_election: true
I20260809 12:10:26.807883 6238 raft_consensus.cc:2454] T 57c28e4572344489bd42bf75cd9633f6 P 95d66d0c4b2f4416afdf59f353e0538e [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 8a8a59f6be334a37b5fd635eb046cd88 in term 0.
I20260809 12:10:26.807971 6100 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "57c28e4572344489bd42bf75cd9633f6" candidate_uuid: "8a8a59f6be334a37b5fd635eb046cd88" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6dbe0cd9d5eb454e8ab7c80039960221" is_pre_election: true
I20260809 12:10:26.808595 6100 raft_consensus.cc:2454] T 57c28e4572344489bd42bf75cd9633f6 P 6dbe0cd9d5eb454e8ab7c80039960221 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 8a8a59f6be334a37b5fd635eb046cd88 in term 0.
I20260809 12:10:26.808657 6311 leader_election.cc:302] T 57c28e4572344489bd42bf75cd9633f6 P 8a8a59f6be334a37b5fd635eb046cd88 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 8a8a59f6be334a37b5fd635eb046cd88, 95d66d0c4b2f4416afdf59f353e0538e; no voters:
I20260809 12:10:26.809170 6458 raft_consensus.cc:2781] T 57c28e4572344489bd42bf75cd9633f6 P 8a8a59f6be334a37b5fd635eb046cd88 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260809 12:10:26.809327 6458 raft_consensus.cc:482] T 57c28e4572344489bd42bf75cd9633f6 P 8a8a59f6be334a37b5fd635eb046cd88 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260809 12:10:26.809432 6458 raft_consensus.cc:3037] T 57c28e4572344489bd42bf75cd9633f6 P 8a8a59f6be334a37b5fd635eb046cd88 [term 0 FOLLOWER]: Advancing to term 1
I20260809 12:10:26.811847 6458 raft_consensus.cc:504] T 57c28e4572344489bd42bf75cd9633f6 P 8a8a59f6be334a37b5fd635eb046cd88 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6dbe0cd9d5eb454e8ab7c80039960221" member_type: VOTER last_known_addr { host: "127.0.222.1" port: 44611 } } peers { permanent_uuid: "95d66d0c4b2f4416afdf59f353e0538e" member_type: VOTER last_known_addr { host: "127.0.222.2" port: 46145 } } peers { permanent_uuid: "8a8a59f6be334a37b5fd635eb046cd88" member_type: VOTER last_known_addr { host: "127.0.222.3" port: 34633 } }
I20260809 12:10:26.812371 6458 leader_election.cc:288] T 57c28e4572344489bd42bf75cd9633f6 P 8a8a59f6be334a37b5fd635eb046cd88 [CANDIDATE]: Term 1 election: Requested vote from peers 6dbe0cd9d5eb454e8ab7c80039960221 (127.0.222.1:44611), 95d66d0c4b2f4416afdf59f353e0538e (127.0.222.2:46145)
I20260809 12:10:26.812836 6238 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "57c28e4572344489bd42bf75cd9633f6" candidate_uuid: "8a8a59f6be334a37b5fd635eb046cd88" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "95d66d0c4b2f4416afdf59f353e0538e"
I20260809 12:10:26.812862 6100 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "57c28e4572344489bd42bf75cd9633f6" candidate_uuid: "8a8a59f6be334a37b5fd635eb046cd88" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6dbe0cd9d5eb454e8ab7c80039960221"
I20260809 12:10:26.813072 6238 raft_consensus.cc:3037] T 57c28e4572344489bd42bf75cd9633f6 P 95d66d0c4b2f4416afdf59f353e0538e [term 0 FOLLOWER]: Advancing to term 1
I20260809 12:10:26.813076 6100 raft_consensus.cc:3037] T 57c28e4572344489bd42bf75cd9633f6 P 6dbe0cd9d5eb454e8ab7c80039960221 [term 0 FOLLOWER]: Advancing to term 1
I20260809 12:10:26.815384 6238 raft_consensus.cc:2454] T 57c28e4572344489bd42bf75cd9633f6 P 95d66d0c4b2f4416afdf59f353e0538e [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 8a8a59f6be334a37b5fd635eb046cd88 in term 1.
I20260809 12:10:26.815403 6100 raft_consensus.cc:2454] T 57c28e4572344489bd42bf75cd9633f6 P 6dbe0cd9d5eb454e8ab7c80039960221 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 8a8a59f6be334a37b5fd635eb046cd88 in term 1.
I20260809 12:10:26.815824 6311 leader_election.cc:302] T 57c28e4572344489bd42bf75cd9633f6 P 8a8a59f6be334a37b5fd635eb046cd88 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 8a8a59f6be334a37b5fd635eb046cd88, 95d66d0c4b2f4416afdf59f353e0538e; no voters:
I20260809 12:10:26.816179 6458 raft_consensus.cc:2781] T 57c28e4572344489bd42bf75cd9633f6 P 8a8a59f6be334a37b5fd635eb046cd88 [term 1 FOLLOWER]: Leader election won for term 1
I20260809 12:10:26.816615 6458 raft_consensus.cc:686] T 57c28e4572344489bd42bf75cd9633f6 P 8a8a59f6be334a37b5fd635eb046cd88 [term 1 LEADER]: Becoming Leader. State: Replica: 8a8a59f6be334a37b5fd635eb046cd88, State: Running, Role: LEADER
I20260809 12:10:26.817091 6458 consensus_queue.cc:237] T 57c28e4572344489bd42bf75cd9633f6 P 8a8a59f6be334a37b5fd635eb046cd88 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6dbe0cd9d5eb454e8ab7c80039960221" member_type: VOTER last_known_addr { host: "127.0.222.1" port: 44611 } } peers { permanent_uuid: "95d66d0c4b2f4416afdf59f353e0538e" member_type: VOTER last_known_addr { host: "127.0.222.2" port: 46145 } } peers { permanent_uuid: "8a8a59f6be334a37b5fd635eb046cd88" member_type: VOTER last_known_addr { host: "127.0.222.3" port: 34633 } }
I20260809 12:10:26.822213 5946 catalog_manager.cc:5495] T 57c28e4572344489bd42bf75cd9633f6 P 8a8a59f6be334a37b5fd635eb046cd88 reported cstate change: term changed from 0 to 1, leader changed from <none> to 8a8a59f6be334a37b5fd635eb046cd88 (127.0.222.3). New cstate: current_term: 1 leader_uuid: "8a8a59f6be334a37b5fd635eb046cd88" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6dbe0cd9d5eb454e8ab7c80039960221" member_type: VOTER last_known_addr { host: "127.0.222.1" port: 44611 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "95d66d0c4b2f4416afdf59f353e0538e" member_type: VOTER last_known_addr { host: "127.0.222.2" port: 46145 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "8a8a59f6be334a37b5fd635eb046cd88" member_type: VOTER last_known_addr { host: "127.0.222.3" port: 34633 } health_report { overall_health: HEALTHY } } }
I20260809 12:10:26.862253 6448 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 12:10:26.860178 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36598 (local address 127.0.222.3:34633)
0809 12:10:26.860358 (+ 180us) server_negotiation.cc:206] Beginning negotiation
0809 12:10:26.860363 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0809 12:10:26.860381 (+ 18us) server_negotiation.cc:407] Connection header received
0809 12:10:26.860561 (+ 180us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 12:10:26.860565 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 12:10:26.860644 (+ 79us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 12:10:26.860730 (+ 86us) server_negotiation.cc:226] Negotiated authn=SASL
0809 12:10:26.861656 (+ 926us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 12:10:26.861660 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 12:10:26.861662 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0809 12:10:26.861675 (+ 13us) server_negotiation.cc:922] calling sasl_server_start()
0809 12:10:26.861702 (+ 27us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0809 12:10:26.861714 (+ 12us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 12:10:26.861811 (+ 97us) server_negotiation.cc:1035] Waiting for connection context
0809 12:10:26.861919 (+ 108us) server_negotiation.cc:299] Negotiation successful
0809 12:10:26.861967 (+ 48us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":68}
I20260809 12:10:26.871196 6100 raft_consensus.cc:1261] T 57c28e4572344489bd42bf75cd9633f6 P 6dbe0cd9d5eb454e8ab7c80039960221 [term 1 FOLLOWER]: Refusing update from remote peer 8a8a59f6be334a37b5fd635eb046cd88: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260809 12:10:26.871196 6238 raft_consensus.cc:1261] T 57c28e4572344489bd42bf75cd9633f6 P 95d66d0c4b2f4416afdf59f353e0538e [term 1 FOLLOWER]: Refusing update from remote peer 8a8a59f6be334a37b5fd635eb046cd88: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260809 12:10:26.872238 6462 consensus_queue.cc:1035] T 57c28e4572344489bd42bf75cd9633f6 P 8a8a59f6be334a37b5fd635eb046cd88 [LEADER]: Connected to new peer: Peer: permanent_uuid: "95d66d0c4b2f4416afdf59f353e0538e" member_type: VOTER last_known_addr { host: "127.0.222.2" port: 46145 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260809 12:10:26.872617 6458 consensus_queue.cc:1035] T 57c28e4572344489bd42bf75cd9633f6 P 8a8a59f6be334a37b5fd635eb046cd88 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6dbe0cd9d5eb454e8ab7c80039960221" member_type: VOTER last_known_addr { host: "127.0.222.1" port: 44611 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260809 12:10:26.884351 6465 mvcc.cc:205] Tried to move back new op lower bound from 7316592340453081088 to 7316592340247023616. Current Snapshot: MvccSnapshot[applied={T|T < 7316592340453081088}]
I20260809 12:10:26.886114 6467 mvcc.cc:205] Tried to move back new op lower bound from 7316592340453081088 to 7316592340247023616. Current Snapshot: MvccSnapshot[applied={T|T < 7316592340453081088}]
I20260809 12:10:26.899286 5948 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:46698:
table { table_name: "test-table" } modify_external_catalogs: true
I20260809 12:10:26.899562 5948 catalog_manager.cc:2645] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:46698:
table { table_name: "test-table" } modify_external_catalogs: true
I20260809 12:10:26.903375 5948 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 39893086d5c946c08450f232d49c51cf: Sending DeleteTablet for 3 replicas of tablet 57c28e4572344489bd42bf75cd9633f6
I20260809 12:10:26.904637 6080 tablet_service.cc:1514] Processing DeleteTablet for tablet 57c28e4572344489bd42bf75cd9633f6 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-09 12:10:26 UTC) from {username='slave'} at 127.0.0.1:46030
I20260809 12:10:26.904827 6218 tablet_service.cc:1514] Processing DeleteTablet for tablet 57c28e4572344489bd42bf75cd9633f6 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-09 12:10:26 UTC) from {username='slave'} at 127.0.0.1:55470
I20260809 12:10:26.905078 6361 tablet_service.cc:1514] Processing DeleteTablet for tablet 57c28e4572344489bd42bf75cd9633f6 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-09 12:10:26 UTC) from {username='slave'} at 127.0.0.1:36590
I20260809 12:10:26.907707 888 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskgH5Pjm/build/asan/bin/kudu pid 6010
I20260809 12:10:26.915408 6478 tablet_replica.cc:330] T 57c28e4572344489bd42bf75cd9633f6 P 8a8a59f6be334a37b5fd635eb046cd88: stopping tablet replica
I20260809 12:10:26.917194 6475 tablet_replica.cc:330] T 57c28e4572344489bd42bf75cd9633f6 P 6dbe0cd9d5eb454e8ab7c80039960221: stopping tablet replica
I20260809 12:10:26.917714 6475 raft_consensus.cc:2229] T 57c28e4572344489bd42bf75cd9633f6 P 6dbe0cd9d5eb454e8ab7c80039960221 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260809 12:10:26.918289 6475 raft_consensus.cc:2258] T 57c28e4572344489bd42bf75cd9633f6 P 6dbe0cd9d5eb454e8ab7c80039960221 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260809 12:10:26.919263 6478 raft_consensus.cc:2229] T 57c28e4572344489bd42bf75cd9633f6 P 8a8a59f6be334a37b5fd635eb046cd88 [term 1 LEADER]: Raft consensus shutting down.
I20260809 12:10:26.920176 6444 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 12:10:26.909254 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46044 (local address 127.0.222.1:44611)
0809 12:10:26.909468 (+ 214us) server_negotiation.cc:206] Beginning negotiation
0809 12:10:26.909478 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0809 12:10:26.915476 (+ 5998us) server_negotiation.cc:407] Connection header received
0809 12:10:26.915882 (+ 406us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 12:10:26.915889 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 12:10:26.916031 (+ 142us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 12:10:26.916116 (+ 85us) server_negotiation.cc:226] Negotiated authn=SASL
0809 12:10:26.919084 (+ 2968us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 12:10:26.919093 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 12:10:26.919102 (+ 9us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0809 12:10:26.919132 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0809 12:10:26.919188 (+ 56us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0809 12:10:26.919210 (+ 22us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 12:10:26.919395 (+ 185us) server_negotiation.cc:1035] Waiting for connection context
0809 12:10:26.919673 (+ 278us) server_negotiation.cc:299] Negotiation successful
0809 12:10:26.919759 (+ 86us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":64}
I20260809 12:10:26.920174 6478 raft_consensus.cc:2258] T 57c28e4572344489bd42bf75cd9633f6 P 8a8a59f6be334a37b5fd635eb046cd88 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260809 12:10:26.921149 6475 ts_tablet_manager.cc:1905] T 57c28e4572344489bd42bf75cd9633f6 P 6dbe0cd9d5eb454e8ab7c80039960221: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260809 12:10:26.921375 6141 generic_service.cc:196] Checking for leaks (request via RPC)
I20260809 12:10:26.923635 6476 tablet_replica.cc:330] T 57c28e4572344489bd42bf75cd9633f6 P 95d66d0c4b2f4416afdf59f353e0538e: stopping tablet replica
I20260809 12:10:26.924094 6476 raft_consensus.cc:2229] T 57c28e4572344489bd42bf75cd9633f6 P 95d66d0c4b2f4416afdf59f353e0538e [term 1 FOLLOWER]: Raft consensus shutting down.
I20260809 12:10:26.924638 6476 raft_consensus.cc:2258] T 57c28e4572344489bd42bf75cd9633f6 P 95d66d0c4b2f4416afdf59f353e0538e [term 1 FOLLOWER]: Raft consensus is shut down!
I20260809 12:10:26.926185 6478 ts_tablet_manager.cc:1905] T 57c28e4572344489bd42bf75cd9633f6 P 8a8a59f6be334a37b5fd635eb046cd88: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260809 12:10:26.930634 6476 ts_tablet_manager.cc:1905] T 57c28e4572344489bd42bf75cd9633f6 P 95d66d0c4b2f4416afdf59f353e0538e: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260809 12:10:26.935526 6478 ts_tablet_manager.cc:1918] T 57c28e4572344489bd42bf75cd9633f6 P 8a8a59f6be334a37b5fd635eb046cd88: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260809 12:10:26.935788 6478 log.cc:1196] T 57c28e4572344489bd42bf75cd9633f6 P 8a8a59f6be334a37b5fd635eb046cd88: Deleting WAL directory at /tmp/dist-test-taskgH5Pjm/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786277382073938-888-0/minicluster-data/ts-2/wal/wals/57c28e4572344489bd42bf75cd9633f6
I20260809 12:10:26.936477 6478 ts_tablet_manager.cc:1939] T 57c28e4572344489bd42bf75cd9633f6 P 8a8a59f6be334a37b5fd635eb046cd88: Deleting consensus metadata
I20260809 12:10:26.938638 5933 catalog_manager.cc:4841] TS 8a8a59f6be334a37b5fd635eb046cd88 (127.0.222.3:34633): tablet 57c28e4572344489bd42bf75cd9633f6 (table test-table [id=0827bcb23dfa4ff78f49bf32670aeeb7]) successfully deleted
I20260809 12:10:26.938762 6476 ts_tablet_manager.cc:1918] T 57c28e4572344489bd42bf75cd9633f6 P 95d66d0c4b2f4416afdf59f353e0538e: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260809 12:10:26.938947 6476 log.cc:1196] T 57c28e4572344489bd42bf75cd9633f6 P 95d66d0c4b2f4416afdf59f353e0538e: Deleting WAL directory at /tmp/dist-test-taskgH5Pjm/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786277382073938-888-0/minicluster-data/ts-1/wal/wals/57c28e4572344489bd42bf75cd9633f6
I20260809 12:10:26.939585 6476 ts_tablet_manager.cc:1939] T 57c28e4572344489bd42bf75cd9633f6 P 95d66d0c4b2f4416afdf59f353e0538e: Deleting consensus metadata
I20260809 12:10:26.941763 5933 catalog_manager.cc:4841] TS 95d66d0c4b2f4416afdf59f353e0538e (127.0.222.2:46145): tablet 57c28e4572344489bd42bf75cd9633f6 (table test-table [id=0827bcb23dfa4ff78f49bf32670aeeb7]) successfully deleted
I20260809 12:10:27.017509 888 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskgH5Pjm/build/asan/bin/kudu with pid 6010
W20260809 12:10:27.043717 5932 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.0.222.1:44611 (error 108)
I20260809 12:10:27.044063 888 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskgH5Pjm/build/asan/bin/kudu pid 6152
I20260809 12:10:27.046455 6446 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 12:10:27.045787 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.222.1:44611 (local address 127.0.0.1:46060)
0809 12:10:27.046048 (+ 261us) negotiation.cc:107] Waiting for socket to connect
0809 12:10:27.046255 (+ 207us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.222.1:44611: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":146}
W20260809 12:10:27.046773 5932 catalog_manager.cc:4584] TS 6dbe0cd9d5eb454e8ab7c80039960221 (127.0.222.1:44611): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 57c28e4572344489bd42bf75cd9633f6: Network error: Client connection negotiation failed: client connection to 127.0.222.1:44611: connect: Connection refused (error 111)
I20260809 12:10:27.054569 6445 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 12:10:27.045840 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55476 (local address 127.0.222.2:46145)
0809 12:10:27.046226 (+ 386us) server_negotiation.cc:206] Beginning negotiation
0809 12:10:27.046232 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0809 12:10:27.046279 (+ 47us) server_negotiation.cc:407] Connection header received
0809 12:10:27.046374 (+ 95us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 12:10:27.046380 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 12:10:27.046564 (+ 184us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 12:10:27.047030 (+ 466us) server_negotiation.cc:226] Negotiated authn=SASL
0809 12:10:27.053852 (+ 6822us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 12:10:27.053865 (+ 13us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 12:10:27.053870 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0809 12:10:27.053895 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0809 12:10:27.053935 (+ 40us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0809 12:10:27.053948 (+ 13us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 12:10:27.054094 (+ 146us) server_negotiation.cc:1035] Waiting for connection context
0809 12:10:27.054181 (+ 87us) server_negotiation.cc:299] Negotiation successful
0809 12:10:27.054252 (+ 71us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":196}
I20260809 12:10:27.055301 6282 generic_service.cc:196] Checking for leaks (request via RPC)
I20260809 12:10:27.100989 6446 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 12:10:27.100363 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.222.1:44611 (local address 127.0.0.1:46064)
0809 12:10:27.100617 (+ 254us) negotiation.cc:107] Waiting for socket to connect
0809 12:10:27.100754 (+ 137us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.222.1:44611: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":90}
I20260809 12:10:27.102202 6446 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 12:10:27.101762 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.222.1:44611 (local address 127.0.0.1:46078)
0809 12:10:27.101935 (+ 173us) negotiation.cc:107] Waiting for socket to connect
0809 12:10:27.102045 (+ 110us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.222.1:44611: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":56}
I20260809 12:10:27.146288 888 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskgH5Pjm/build/asan/bin/kudu with pid 6152
I20260809 12:10:27.163067 6446 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 12:10:27.162374 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.222.1:44611 (local address 127.0.0.1:46080)
0809 12:10:27.162661 (+ 287us) negotiation.cc:107] Waiting for socket to connect
0809 12:10:27.162806 (+ 145us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.222.1:44611: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":100}
I20260809 12:10:27.164390 6446 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 12:10:27.163931 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.222.1:44611 (local address 127.0.0.1:46094)
0809 12:10:27.164123 (+ 192us) negotiation.cc:107] Waiting for socket to connect
0809 12:10:27.164224 (+ 101us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.222.1:44611: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":67}
I20260809 12:10:27.166803 888 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskgH5Pjm/build/asan/bin/kudu pid 6292
I20260809 12:10:27.170080 6448 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 12:10:27.167711 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36606 (local address 127.0.222.3:34633)
0809 12:10:27.167895 (+ 184us) server_negotiation.cc:206] Beginning negotiation
0809 12:10:27.167900 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0809 12:10:27.167919 (+ 19us) server_negotiation.cc:407] Connection header received
0809 12:10:27.168158 (+ 239us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 12:10:27.168162 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 12:10:27.168218 (+ 56us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 12:10:27.168305 (+ 87us) server_negotiation.cc:226] Negotiated authn=SASL
0809 12:10:27.169301 (+ 996us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 12:10:27.169305 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 12:10:27.169308 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0809 12:10:27.169323 (+ 15us) server_negotiation.cc:922] calling sasl_server_start()
0809 12:10:27.169353 (+ 30us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0809 12:10:27.169366 (+ 13us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 12:10:27.169532 (+ 166us) server_negotiation.cc:1035] Waiting for connection context
0809 12:10:27.169664 (+ 132us) server_negotiation.cc:299] Negotiation successful
0809 12:10:27.169722 (+ 58us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":66}
I20260809 12:10:27.170791 6425 generic_service.cc:196] Checking for leaks (request via RPC)
I20260809 12:10:27.245345 6446 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 12:10:27.244821 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.222.1:44611 (local address 127.0.0.1:46104)
0809 12:10:27.245084 (+ 263us) negotiation.cc:107] Waiting for socket to connect
0809 12:10:27.245190 (+ 106us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.222.1:44611: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":103}
I20260809 12:10:27.246625 6446 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 12:10:27.246215 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.222.1:44611 (local address 127.0.0.1:46114)
0809 12:10:27.246422 (+ 207us) negotiation.cc:107] Waiting for socket to connect
0809 12:10:27.246492 (+ 70us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.222.1:44611: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":109}
I20260809 12:10:27.290432 888 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskgH5Pjm/build/asan/bin/kudu with pid 6292
I20260809 12:10:27.312364 888 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskgH5Pjm/build/asan/bin/kudu pid 5916
I20260809 12:10:27.313614 5979 generic_service.cc:196] Checking for leaks (request via RPC)
I20260809 12:10:27.422664 6446 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 12:10:27.422084 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.222.1:44611 (local address 127.0.0.1:46128)
0809 12:10:27.422331 (+ 247us) negotiation.cc:107] Waiting for socket to connect
0809 12:10:27.422467 (+ 136us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.222.1:44611: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":97}
I20260809 12:10:27.422729 888 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskgH5Pjm/build/asan/bin/kudu with pid 5916
2026-08-09T12:10:27Z chronyd exiting
[ OK ] SecurityITest.TestDisableAuthenticationEncryption (2732 ms)
[ RUN ] SecurityITest.TestJwtMiniCluster
I20260809 12:10:27.469614 888 mini_oidc.cc:152] Starting JWKS server
I20260809 12:10:27.472604 888 webserver.cc:460] Webserver started at https://127.0.0.1:42027/ using document root <none> and password file <none>
I20260809 12:10:27.472905 888 mini_oidc.cc:189] Starting OIDC Discovery server
I20260809 12:10:27.473538 888 webserver.cc:460] Webserver started at http://127.0.0.1:40827/ using document root <none> and password file <none>
2026-08-09T12:10:27Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-09T12:10:27Z Disabled control of system clock
I20260809 12:10:27.508158 888 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskgH5Pjm/build/asan/bin/kudu
/tmp/dist-test-taskgH5Pjm/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskgH5Pjm/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1786277382073938-888-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskgH5Pjm/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1786277382073938-888-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskgH5Pjm/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1786277382073938-888-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskgH5Pjm/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1786277382073938-888-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.222.62:38167
--webserver_interface=127.0.222.62
--webserver_port=0
--builtin_ntp_servers=127.0.222.20:42635
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.222.62:38167
--enable_jwt_token_auth=true
--jwks_url=https://localhost:42027/jwks.json
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--trusted_certificate_file=/tmp/dist-test-taskgH5Pjm/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1786277382073938-888-0/testchainca.cert with env {}
W20260809 12:10:27.847559 6490 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260809 12:10:27.847865 6490 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260809 12:10:27.848048 6490 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260809 12:10:27.855509 6490 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260809 12:10:27.855619 6490 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260809 12:10:27.855695 6490 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260809 12:10:27.855808 6490 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260809 12:10:27.855892 6490 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
W20260809 12:10:27.855983 6490 flags.cc:425] Enabled experimental flag: --enable_jwt_token_auth=true
W20260809 12:10:27.856060 6490 flags.cc:425] Enabled experimental flag: --jwks_url=https://localhost:42027/jwks.json
I20260809 12:10:27.866386 6490 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.222.20:42635
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskgH5Pjm/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1786277382073938-888-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskgH5Pjm/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1786277382073938-888-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.222.62:38167
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.222.62:38167
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--enable_jwt_token_auth=true
--jwks_url=https://localhost:42027/jwks.json
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskgH5Pjm/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1786277382073938-888-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.222.62
--webserver_port=0
--trusted_certificate_file=/tmp/dist-test-taskgH5Pjm/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1786277382073938-888-0/testchainca.cert
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskgH5Pjm/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1786277382073938-888-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 09 Aug 2026 11:06:41 UTC on ccycloud.quasar-bwoogy.root.comops.site
build id 4575991
ASAN enabled
I20260809 12:10:27.867889 6490 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260809 12:10:27.869252 6490 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260809 12:10:27.881544 6499 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 12:10:27.881510 6496 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 12:10:27.881752 6495 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260809 12:10:27.882448 6490 server_base.cc:1034] running on GCE node
I20260809 12:10:27.883273 6490 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260809 12:10:27.884627 6490 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260809 12:10:27.885879 6490 hybrid_clock.cc:648] HybridClock initialized: now 1786277427885828 us; error 40 us; skew 500 ppm
I20260809 12:10:27.886209 6490 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260809 12:10:27.888650 6490 webserver.cc:460] Webserver started at http://127.0.222.62:44527/ using document root <none> and password file <none>
I20260809 12:10:27.889305 6490 fs_manager.cc:359] Metadata directory not provided
I20260809 12:10:27.889398 6490 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260809 12:10:27.889658 6490 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260809 12:10:27.892738 6490 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskgH5Pjm/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1786277382073938-888-0/minicluster-data/master-0/data/instance:
uuid: "1a8648675ff044caadbd1277966581a7"
format_stamp: "Formatted at 2026-08-09 12:10:27 on dist-test-slave-2w3w"
server_key: "e521f8e6f4a16f7dc989fcc0315d3582"
server_key_iv: "484006aa0a1749d7d507406e95212289"
server_key_version: "encryptionkey@0"
I20260809 12:10:27.893600 6490 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskgH5Pjm/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1786277382073938-888-0/minicluster-data/master-0/wal/instance:
uuid: "1a8648675ff044caadbd1277966581a7"
format_stamp: "Formatted at 2026-08-09 12:10:27 on dist-test-slave-2w3w"
server_key: "e521f8e6f4a16f7dc989fcc0315d3582"
server_key_iv: "484006aa0a1749d7d507406e95212289"
server_key_version: "encryptionkey@0"
I20260809 12:10:27.899492 6490 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.003s sys 0.004s
I20260809 12:10:27.903100 6507 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260809 12:10:27.904420 6490 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260809 12:10:27.904570 6490 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskgH5Pjm/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1786277382073938-888-0/minicluster-data/master-0/data,/tmp/dist-test-taskgH5Pjm/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1786277382073938-888-0/minicluster-data/master-0/wal
uuid: "1a8648675ff044caadbd1277966581a7"
format_stamp: "Formatted at 2026-08-09 12:10:27 on dist-test-slave-2w3w"
server_key: "e521f8e6f4a16f7dc989fcc0315d3582"
server_key_iv: "484006aa0a1749d7d507406e95212289"
server_key_version: "encryptionkey@0"
I20260809 12:10:27.904749 6490 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskgH5Pjm/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1786277382073938-888-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskgH5Pjm/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1786277382073938-888-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskgH5Pjm/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1786277382073938-888-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260809 12:10:27.919884 6490 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260809 12:10:27.920812 6490 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260809 12:10:27.921113 6490 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260809 12:10:27.939404 6490 rpc_server.cc:307] RPC server started. Bound to: 127.0.222.62:38167
I20260809 12:10:27.939424 6563 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.222.62:38167 every 8 connection(s)
I20260809 12:10:27.940975 6490 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskgH5Pjm/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1786277382073938-888-0/minicluster-data/master-0/data/info.pb
I20260809 12:10:27.944702 6564 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260809 12:10:27.951355 888 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskgH5Pjm/build/asan/bin/kudu as pid 6490
I20260809 12:10:27.951591 888 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskgH5Pjm/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1786277382073938-888-0/minicluster-data/master-0/wal/instance
I20260809 12:10:27.952245 888 external_mini_cluster.cc:1428] Setting key cf0bd2ccde8b4557e3a3d6ea1b771fa8
I20260809 12:10:27.955538 6564 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 1a8648675ff044caadbd1277966581a7: Bootstrap starting.
I20260809 12:10:27.958709 6564 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 1a8648675ff044caadbd1277966581a7: Neither blocks nor log segments found. Creating new log.
I20260809 12:10:27.959821 6564 log.cc:824] T 00000000000000000000000000000000 P 1a8648675ff044caadbd1277966581a7: Log is configured to *not* fsync() on all Append() calls
I20260809 12:10:27.962584 6564 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 1a8648675ff044caadbd1277966581a7: No bootstrap required, opened a new log
I20260809 12:10:27.965726 6567 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 12:10:27.954101 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42748 (local address 127.0.222.62:38167)
0809 12:10:27.955117 (+ 1016us) server_negotiation.cc:206] Beginning negotiation
0809 12:10:27.955128 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0809 12:10:27.955174 (+ 46us) server_negotiation.cc:407] Connection header received
0809 12:10:27.956938 (+ 1764us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 12:10:27.956964 (+ 26us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 12:10:27.957429 (+ 465us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 12:10:27.957827 (+ 398us) server_negotiation.cc:226] Negotiated authn=SASL
0809 12:10:27.958387 (+ 560us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 12:10:27.959853 (+ 1466us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 12:10:27.961233 (+ 1380us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 12:10:27.961875 (+ 642us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 12:10:27.962079 (+ 204us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 12:10:27.963389 (+ 1310us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 12:10:27.963397 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 12:10:27.963410 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0809 12:10:27.963953 (+ 543us) server_negotiation.cc:922] calling sasl_server_start()
0809 12:10:27.964032 (+ 79us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0809 12:10:27.964056 (+ 24us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 12:10:27.964205 (+ 149us) server_negotiation.cc:1035] Waiting for connection context
0809 12:10:27.964484 (+ 279us) server_negotiation.cc:299] Negotiation successful
0809 12:10:27.964780 (+ 296us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":533,"thread_start_us":330,"threads_started":1}
I20260809 12:10:27.969743 888 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260809 12:10:27.971284 6564 raft_consensus.cc:348] T 00000000000000000000000000000000 P 1a8648675ff044caadbd1277966581a7 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1a8648675ff044caadbd1277966581a7" member_type: VOTER last_known_addr { host: "127.0.222.62" port: 38167 } }
I20260809 12:10:27.971741 6564 raft_consensus.cc:374] T 00000000000000000000000000000000 P 1a8648675ff044caadbd1277966581a7 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260809 12:10:27.971851 6564 raft_consensus.cc:729] T 00000000000000000000000000000000 P 1a8648675ff044caadbd1277966581a7 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1a8648675ff044caadbd1277966581a7, State: Initialized, Role: FOLLOWER
I20260809 12:10:27.972582 6564 consensus_queue.cc:260] T 00000000000000000000000000000000 P 1a8648675ff044caadbd1277966581a7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1a8648675ff044caadbd1277966581a7" member_type: VOTER last_known_addr { host: "127.0.222.62" port: 38167 } }
I20260809 12:10:27.972832 6564 raft_consensus.cc:388] T 00000000000000000000000000000000 P 1a8648675ff044caadbd1277966581a7 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260809 12:10:27.973006 6564 raft_consensus.cc:482] T 00000000000000000000000000000000 P 1a8648675ff044caadbd1277966581a7 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260809 12:10:27.973204 6564 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 1a8648675ff044caadbd1277966581a7 [term 0 FOLLOWER]: Advancing to term 1
I20260809 12:10:27.975876 6564 raft_consensus.cc:504] T 00000000000000000000000000000000 P 1a8648675ff044caadbd1277966581a7 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1a8648675ff044caadbd1277966581a7" member_type: VOTER last_known_addr { host: "127.0.222.62" port: 38167 } }
I20260809 12:10:27.976532 6564 leader_election.cc:302] T 00000000000000000000000000000000 P 1a8648675ff044caadbd1277966581a7 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 1a8648675ff044caadbd1277966581a7; no voters:
I20260809 12:10:27.977116 6564 leader_election.cc:288] T 00000000000000000000000000000000 P 1a8648675ff044caadbd1277966581a7 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260809 12:10:27.977246 6569 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 1a8648675ff044caadbd1277966581a7 [term 1 FOLLOWER]: Leader election won for term 1
I20260809 12:10:27.978039 6569 raft_consensus.cc:686] T 00000000000000000000000000000000 P 1a8648675ff044caadbd1277966581a7 [term 1 LEADER]: Becoming Leader. State: Replica: 1a8648675ff044caadbd1277966581a7, State: Running, Role: LEADER
I20260809 12:10:27.978574 6569 consensus_queue.cc:237] T 00000000000000000000000000000000 P 1a8648675ff044caadbd1277966581a7 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1a8648675ff044caadbd1277966581a7" member_type: VOTER last_known_addr { host: "127.0.222.62" port: 38167 } }
I20260809 12:10:27.979426 6564 sys_catalog.cc:564] T 00000000000000000000000000000000 P 1a8648675ff044caadbd1277966581a7 [sys.catalog]: configured and running, proceeding with master startup.
I20260809 12:10:27.981623 6570 sys_catalog.cc:455] T 00000000000000000000000000000000 P 1a8648675ff044caadbd1277966581a7 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "1a8648675ff044caadbd1277966581a7" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1a8648675ff044caadbd1277966581a7" member_type: VOTER last_known_addr { host: "127.0.222.62" port: 38167 } } }
I20260809 12:10:27.982007 6570 sys_catalog.cc:458] T 00000000000000000000000000000000 P 1a8648675ff044caadbd1277966581a7 [sys.catalog]: This master's current role is: LEADER
I20260809 12:10:27.982822 6576 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260809 12:10:27.992623 6576 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260809 12:10:28.000728 6587 catalog_manager.cc:795] Waiting for catalog manager background task thread to start: Service unavailable: Catalog manager is not initialized. State: Starting
I20260809 12:10:28.000521 6571 sys_catalog.cc:455] T 00000000000000000000000000000000 P 1a8648675ff044caadbd1277966581a7 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 1a8648675ff044caadbd1277966581a7. Latest consensus state: current_term: 1 leader_uuid: "1a8648675ff044caadbd1277966581a7" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1a8648675ff044caadbd1277966581a7" member_type: VOTER last_known_addr { host: "127.0.222.62" port: 38167 } } }
I20260809 12:10:28.001547 6571 sys_catalog.cc:458] T 00000000000000000000000000000000 P 1a8648675ff044caadbd1277966581a7 [sys.catalog]: This master's current role is: LEADER
I20260809 12:10:28.005949 6576 catalog_manager.cc:1347] Generated new cluster ID: 8b7ea78996bf4c67aa5e96cab7fb61f3
I20260809 12:10:28.006074 6576 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260809 12:10:28.027029 6576 catalog_manager.cc:1370] Generated new certificate authority record
I20260809 12:10:28.029034 6576 catalog_manager.cc:1504] Loading token signing keys...
I20260809 12:10:28.037708 6576 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 1a8648675ff044caadbd1277966581a7: Generated new TSK 0
I20260809 12:10:28.038730 6576 catalog_manager.cc:1514] Initializing in-progress tserver states...
E20260809 12:10:28.085373 6595 webserver.cc:533] Webserver: sslize failed: error:14094415:SSL routines:ssl3_read_bytes:sslv3 alert certificate expired
E20260809 12:10:28.085597 6595 webserver.cc:533] Webserver: error shutting down SSL connection: error:140E0197:SSL routines:SSL_shutdown:shutdown while in init
I20260809 12:10:28.086814 6567 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 12:10:28.064045 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42754 (local address 127.0.222.62:38167)
0809 12:10:28.064254 (+ 209us) server_negotiation.cc:206] Beginning negotiation
0809 12:10:28.064261 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0809 12:10:28.064774 (+ 513us) server_negotiation.cc:407] Connection header received
0809 12:10:28.065124 (+ 350us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 12:10:28.065130 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 12:10:28.065218 (+ 88us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 12:10:28.065385 (+ 167us) server_negotiation.cc:226] Negotiated authn=JWT
0809 12:10:28.066017 (+ 632us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 12:10:28.067264 (+ 1247us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 12:10:28.068705 (+ 1441us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 12:10:28.069407 (+ 702us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 12:10:28.069537 (+ 130us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 12:10:28.070082 (+ 545us) server_negotiation.cc:365] Received JWT_EXCHANGE NegotiatePB request
0809 12:10:28.086139 (+ 16057us) server_negotiation.cc:394] Sending RPC error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:42027/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: certificate has expired
0809 12:10:28.086505 (+ 366us) negotiation.cc:327] Negotiation complete: Not authorized: Server connection negotiation failed: server connection from 127.0.0.1:42754: Failed to load JWKS: Error downloading JWKS from 'https://localhost:42027/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: certificate has expired
Metrics: {"server-negotiator.queue_time_us":69}
W20260809 12:10:28.086951 6567 negotiation.cc:344] Unauthorized connection attempt: Server connection negotiation failed: server connection from 127.0.0.1:42754: Failed to load JWKS: Error downloading JWKS from 'https://localhost:42027/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: certificate has expired
W20260809 12:10:28.088353 6594 negotiation.cc:337] Failed RPC negotiation. Trace:
0809 12:10:28.063915 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.222.62:38167 (local address 127.0.0.1:42754)
0809 12:10:28.064669 (+ 754us) negotiation.cc:107] Waiting for socket to connect
0809 12:10:28.064686 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0809 12:10:28.064843 (+ 157us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 12:10:28.065467 (+ 624us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 12:10:28.065479 (+ 12us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 12:10:28.065490 (+ 11us) client_negotiation.cc:189] Negotiated authn=JWT
0809 12:10:28.065834 (+ 344us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 12:10:28.065844 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 12:10:28.067417 (+ 1573us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 12:10:28.067421 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 12:10:28.068540 (+ 1119us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 12:10:28.068551 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 12:10:28.069585 (+ 1034us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 12:10:28.069589 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 12:10:28.069714 (+ 125us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 12:10:28.069768 (+ 54us) client_negotiation.cc:252] Sending JWT_EXCHANGE NegotiatePB request
0809 12:10:28.086402 (+ 16634us) client_negotiation.cc:283] Received error response from server: Runtime error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:42027/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: certificate has expired
0809 12:10:28.086663 (+ 261us) negotiation.cc:327] Negotiation complete: Runtime error: Client connection negotiation failed: client connection to 127.0.222.62:38167: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:42027/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: certificate has expired
Metrics: {"client-negotiator.queue_time_us":570,"thread_start_us":405,"threads_started":1}
/data0/somelongdirectorytoavoidrpathissues/src/kudu/src/kudu/integration-tests/security-itest.cc:597: Failure
Failed
Bad status: Runtime error: Could not connect to the cluster: Client connection negotiation failed: client connection to 127.0.222.62:38167: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:42027/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: certificate has expired
Google Test trace:
/data0/somelongdirectorytoavoidrpathissues/src/kudu/src/kudu/integration-tests/security-itest.cc:593: Valid JWT
I20260809 12:10:28.090008 888 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskgH5Pjm/build/asan/bin/kudu pid 6490
I20260809 12:10:28.091485 6559 generic_service.cc:196] Checking for leaks (request via RPC)
I20260809 12:10:28.189405 888 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskgH5Pjm/build/asan/bin/kudu with pid 6490
2026-08-09T12:10:28Z chronyd exiting
I20260809 12:10:28.244385 888 test_util.cc:175] -----------------------------------------------
I20260809 12:10:28.244530 888 test_util.cc:176] Had fatal failures, leaving test files at /tmp/dist-test-taskgH5Pjm/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1786277382073938-888-0
[ FAILED ] SecurityITest.TestJwtMiniCluster (780 ms)
[ RUN ] SecurityITest.TestJwtMiniClusterWithInvalidCert
I20260809 12:10:28.248530 888 mini_oidc.cc:152] Starting JWKS server
I20260809 12:10:28.250420 888 webserver.cc:460] Webserver started at https://127.0.0.1:41869/ using document root <none> and password file <none>
I20260809 12:10:28.250607 888 mini_oidc.cc:189] Starting OIDC Discovery server
I20260809 12:10:28.251274 888 webserver.cc:460] Webserver started at http://127.0.0.1:36491/ using document root <none> and password file <none>
2026-08-09T12:10:28Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-09T12:10:28Z Disabled control of system clock
I20260809 12:10:28.286583 888 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskgH5Pjm/build/asan/bin/kudu
/tmp/dist-test-taskgH5Pjm/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskgH5Pjm/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1786277382073938-888-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskgH5Pjm/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1786277382073938-888-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskgH5Pjm/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1786277382073938-888-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskgH5Pjm/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1786277382073938-888-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.222.62:33233
--webserver_interface=127.0.222.62
--webserver_port=0
--builtin_ntp_servers=127.0.222.20:46015
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.222.62:33233
--enable_jwt_token_auth=true
--jwks_url=https://localhost:41869/jwks.json
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--trusted_certificate_file=/tmp/dist-test-taskgH5Pjm/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1786277382073938-888-0/testchainca.cert with env {}
W20260809 12:10:28.620373 6603 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260809 12:10:28.620673 6603 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260809 12:10:28.620829 6603 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260809 12:10:28.628203 6603 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260809 12:10:28.628319 6603 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260809 12:10:28.628415 6603 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260809 12:10:28.628511 6603 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260809 12:10:28.628592 6603 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
W20260809 12:10:28.628688 6603 flags.cc:425] Enabled experimental flag: --enable_jwt_token_auth=true
W20260809 12:10:28.628746 6603 flags.cc:425] Enabled experimental flag: --jwks_url=https://localhost:41869/jwks.json
I20260809 12:10:28.638942 6603 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.222.20:46015
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskgH5Pjm/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1786277382073938-888-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskgH5Pjm/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1786277382073938-888-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.222.62:33233
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.222.62:33233
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--enable_jwt_token_auth=true
--jwks_url=https://localhost:41869/jwks.json
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskgH5Pjm/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1786277382073938-888-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.222.62
--webserver_port=0
--trusted_certificate_file=/tmp/dist-test-taskgH5Pjm/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1786277382073938-888-0/testchainca.cert
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskgH5Pjm/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1786277382073938-888-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 09 Aug 2026 11:06:41 UTC on ccycloud.quasar-bwoogy.root.comops.site
build id 4575991
ASAN enabled
I20260809 12:10:28.640328 6603 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260809 12:10:28.641640 6603 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260809 12:10:28.653929 6609 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 12:10:28.654047 6613 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 12:10:28.654052 6610 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260809 12:10:28.655393 6603 server_base.cc:1034] running on GCE node
I20260809 12:10:28.656208 6603 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260809 12:10:28.657764 6603 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260809 12:10:28.659063 6603 hybrid_clock.cc:648] HybridClock initialized: now 1786277428659006 us; error 51 us; skew 500 ppm
I20260809 12:10:28.659430 6603 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260809 12:10:28.661774 6603 webserver.cc:460] Webserver started at http://127.0.222.62:42603/ using document root <none> and password file <none>
I20260809 12:10:28.662487 6603 fs_manager.cc:359] Metadata directory not provided
I20260809 12:10:28.662628 6603 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260809 12:10:28.663039 6603 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260809 12:10:28.665977 6603 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskgH5Pjm/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1786277382073938-888-0/minicluster-data/master-0/data/instance:
uuid: "83ec5e90fb55454abd37fabbc04b7dc6"
format_stamp: "Formatted at 2026-08-09 12:10:28 on dist-test-slave-2w3w"
server_key: "62de4e41d7f499dfc67e83265a9645cd"
server_key_iv: "a437d3c20fd9b3dccfe43f561d3d66bb"
server_key_version: "encryptionkey@0"
I20260809 12:10:28.666886 6603 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskgH5Pjm/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1786277382073938-888-0/minicluster-data/master-0/wal/instance:
uuid: "83ec5e90fb55454abd37fabbc04b7dc6"
format_stamp: "Formatted at 2026-08-09 12:10:28 on dist-test-slave-2w3w"
server_key: "62de4e41d7f499dfc67e83265a9645cd"
server_key_iv: "a437d3c20fd9b3dccfe43f561d3d66bb"
server_key_version: "encryptionkey@0"
I20260809 12:10:28.672161 6603 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.004s sys 0.000s
I20260809 12:10:28.675863 6619 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260809 12:10:28.677317 6603 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260809 12:10:28.677520 6603 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskgH5Pjm/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1786277382073938-888-0/minicluster-data/master-0/data,/tmp/dist-test-taskgH5Pjm/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1786277382073938-888-0/minicluster-data/master-0/wal
uuid: "83ec5e90fb55454abd37fabbc04b7dc6"
format_stamp: "Formatted at 2026-08-09 12:10:28 on dist-test-slave-2w3w"
server_key: "62de4e41d7f499dfc67e83265a9645cd"
server_key_iv: "a437d3c20fd9b3dccfe43f561d3d66bb"
server_key_version: "encryptionkey@0"
I20260809 12:10:28.677759 6603 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskgH5Pjm/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1786277382073938-888-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskgH5Pjm/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1786277382073938-888-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskgH5Pjm/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1786277382073938-888-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260809 12:10:28.698210 6603 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260809 12:10:28.699184 6603 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260809 12:10:28.699527 6603 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260809 12:10:28.719061 6603 rpc_server.cc:307] RPC server started. Bound to: 127.0.222.62:33233
I20260809 12:10:28.719110 6672 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.222.62:33233 every 8 connection(s)
I20260809 12:10:28.720501 6603 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskgH5Pjm/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1786277382073938-888-0/minicluster-data/master-0/data/info.pb
I20260809 12:10:28.724426 6674 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260809 12:10:28.728139 888 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskgH5Pjm/build/asan/bin/kudu as pid 6603
I20260809 12:10:28.728336 888 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskgH5Pjm/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1786277382073938-888-0/minicluster-data/master-0/wal/instance
I20260809 12:10:28.729028 888 external_mini_cluster.cc:1428] Setting key 48f4646bfddeb3f5ec54a90c70bc6fe7
I20260809 12:10:28.737120 6674 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 83ec5e90fb55454abd37fabbc04b7dc6: Bootstrap starting.
I20260809 12:10:28.740888 6674 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 83ec5e90fb55454abd37fabbc04b7dc6: Neither blocks nor log segments found. Creating new log.
I20260809 12:10:28.742238 6674 log.cc:824] T 00000000000000000000000000000000 P 83ec5e90fb55454abd37fabbc04b7dc6: Log is configured to *not* fsync() on all Append() calls
I20260809 12:10:28.745437 6678 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 12:10:28.731098 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55488 (local address 127.0.222.62:33233)
0809 12:10:28.732188 (+ 1090us) server_negotiation.cc:206] Beginning negotiation
0809 12:10:28.732202 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0809 12:10:28.732255 (+ 53us) server_negotiation.cc:407] Connection header received
0809 12:10:28.735258 (+ 3003us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 12:10:28.735289 (+ 31us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 12:10:28.735880 (+ 591us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 12:10:28.736300 (+ 420us) server_negotiation.cc:226] Negotiated authn=SASL
0809 12:10:28.736855 (+ 555us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 12:10:28.738861 (+ 2006us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 12:10:28.740189 (+ 1328us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 12:10:28.741328 (+ 1139us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 12:10:28.741527 (+ 199us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 12:10:28.742619 (+ 1092us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 12:10:28.742631 (+ 12us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 12:10:28.742649 (+ 18us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0809 12:10:28.743418 (+ 769us) server_negotiation.cc:922] calling sasl_server_start()
0809 12:10:28.743492 (+ 74us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0809 12:10:28.743550 (+ 58us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 12:10:28.743670 (+ 120us) server_negotiation.cc:1035] Waiting for connection context
0809 12:10:28.743879 (+ 209us) server_negotiation.cc:299] Negotiation successful
0809 12:10:28.744194 (+ 315us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":584,"thread_start_us":344,"threads_started":1}
I20260809 12:10:28.745602 6674 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 83ec5e90fb55454abd37fabbc04b7dc6: No bootstrap required, opened a new log
I20260809 12:10:28.749432 888 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260809 12:10:28.754710 6674 raft_consensus.cc:348] T 00000000000000000000000000000000 P 83ec5e90fb55454abd37fabbc04b7dc6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "83ec5e90fb55454abd37fabbc04b7dc6" member_type: VOTER last_known_addr { host: "127.0.222.62" port: 33233 } }
I20260809 12:10:28.755251 6674 raft_consensus.cc:374] T 00000000000000000000000000000000 P 83ec5e90fb55454abd37fabbc04b7dc6 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260809 12:10:28.755378 6674 raft_consensus.cc:729] T 00000000000000000000000000000000 P 83ec5e90fb55454abd37fabbc04b7dc6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 83ec5e90fb55454abd37fabbc04b7dc6, State: Initialized, Role: FOLLOWER
I20260809 12:10:28.755941 6674 consensus_queue.cc:260] T 00000000000000000000000000000000 P 83ec5e90fb55454abd37fabbc04b7dc6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "83ec5e90fb55454abd37fabbc04b7dc6" member_type: VOTER last_known_addr { host: "127.0.222.62" port: 33233 } }
I20260809 12:10:28.756181 6674 raft_consensus.cc:388] T 00000000000000000000000000000000 P 83ec5e90fb55454abd37fabbc04b7dc6 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260809 12:10:28.756318 6674 raft_consensus.cc:482] T 00000000000000000000000000000000 P 83ec5e90fb55454abd37fabbc04b7dc6 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260809 12:10:28.756510 6674 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 83ec5e90fb55454abd37fabbc04b7dc6 [term 0 FOLLOWER]: Advancing to term 1
I20260809 12:10:28.758900 6674 raft_consensus.cc:504] T 00000000000000000000000000000000 P 83ec5e90fb55454abd37fabbc04b7dc6 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "83ec5e90fb55454abd37fabbc04b7dc6" member_type: VOTER last_known_addr { host: "127.0.222.62" port: 33233 } }
I20260809 12:10:28.759531 6674 leader_election.cc:302] T 00000000000000000000000000000000 P 83ec5e90fb55454abd37fabbc04b7dc6 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 83ec5e90fb55454abd37fabbc04b7dc6; no voters:
I20260809 12:10:28.760120 6674 leader_election.cc:288] T 00000000000000000000000000000000 P 83ec5e90fb55454abd37fabbc04b7dc6 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260809 12:10:28.760282 6681 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 83ec5e90fb55454abd37fabbc04b7dc6 [term 1 FOLLOWER]: Leader election won for term 1
I20260809 12:10:28.761406 6681 raft_consensus.cc:686] T 00000000000000000000000000000000 P 83ec5e90fb55454abd37fabbc04b7dc6 [term 1 LEADER]: Becoming Leader. State: Replica: 83ec5e90fb55454abd37fabbc04b7dc6, State: Running, Role: LEADER
I20260809 12:10:28.761879 6681 consensus_queue.cc:237] T 00000000000000000000000000000000 P 83ec5e90fb55454abd37fabbc04b7dc6 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "83ec5e90fb55454abd37fabbc04b7dc6" member_type: VOTER last_known_addr { host: "127.0.222.62" port: 33233 } }
I20260809 12:10:28.762192 6674 sys_catalog.cc:564] T 00000000000000000000000000000000 P 83ec5e90fb55454abd37fabbc04b7dc6 [sys.catalog]: configured and running, proceeding with master startup.
I20260809 12:10:28.765112 6682 sys_catalog.cc:455] T 00000000000000000000000000000000 P 83ec5e90fb55454abd37fabbc04b7dc6 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "83ec5e90fb55454abd37fabbc04b7dc6" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "83ec5e90fb55454abd37fabbc04b7dc6" member_type: VOTER last_known_addr { host: "127.0.222.62" port: 33233 } } }
I20260809 12:10:28.765594 6682 sys_catalog.cc:458] T 00000000000000000000000000000000 P 83ec5e90fb55454abd37fabbc04b7dc6 [sys.catalog]: This master's current role is: LEADER
I20260809 12:10:28.765136 6683 sys_catalog.cc:455] T 00000000000000000000000000000000 P 83ec5e90fb55454abd37fabbc04b7dc6 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 83ec5e90fb55454abd37fabbc04b7dc6. Latest consensus state: current_term: 1 leader_uuid: "83ec5e90fb55454abd37fabbc04b7dc6" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "83ec5e90fb55454abd37fabbc04b7dc6" member_type: VOTER last_known_addr { host: "127.0.222.62" port: 33233 } } }
I20260809 12:10:28.766820 6683 sys_catalog.cc:458] T 00000000000000000000000000000000 P 83ec5e90fb55454abd37fabbc04b7dc6 [sys.catalog]: This master's current role is: LEADER
I20260809 12:10:28.768816 6691 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260809 12:10:28.773257 6691 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260809 12:10:28.781845 6691 catalog_manager.cc:1347] Generated new cluster ID: 5e2fb97d83a24746bbf9b0612dc8b073
I20260809 12:10:28.781984 6691 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260809 12:10:28.804646 6691 catalog_manager.cc:1370] Generated new certificate authority record
I20260809 12:10:28.805953 6691 catalog_manager.cc:1504] Loading token signing keys...
I20260809 12:10:28.823680 6691 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 83ec5e90fb55454abd37fabbc04b7dc6: Generated new TSK 0
I20260809 12:10:28.824642 6691 catalog_manager.cc:1514] Initializing in-progress tserver states...
E20260809 12:10:28.859153 6708 webserver.cc:533] Webserver: sslize failed: error:14094418:SSL routines:ssl3_read_bytes:tlsv1 alert unknown ca
E20260809 12:10:28.859285 6708 webserver.cc:533] Webserver: error shutting down SSL connection: error:140E0197:SSL routines:SSL_shutdown:shutdown while in init
I20260809 12:10:28.860584 6678 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 12:10:28.840087 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55490 (local address 127.0.222.62:33233)
0809 12:10:28.840288 (+ 201us) server_negotiation.cc:206] Beginning negotiation
0809 12:10:28.840295 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0809 12:10:28.840614 (+ 319us) server_negotiation.cc:407] Connection header received
0809 12:10:28.840830 (+ 216us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 12:10:28.840836 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 12:10:28.840925 (+ 89us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 12:10:28.841093 (+ 168us) server_negotiation.cc:226] Negotiated authn=JWT
0809 12:10:28.841530 (+ 437us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 12:10:28.842682 (+ 1152us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 12:10:28.844121 (+ 1439us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 12:10:28.844826 (+ 705us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 12:10:28.844959 (+ 133us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 12:10:28.845442 (+ 483us) server_negotiation.cc:365] Received JWT_EXCHANGE NegotiatePB request
0809 12:10:28.859872 (+ 14430us) server_negotiation.cc:394] Sending RPC error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:41869/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: unable to get local issuer certificate
0809 12:10:28.860254 (+ 382us) negotiation.cc:327] Negotiation complete: Not authorized: Server connection negotiation failed: server connection from 127.0.0.1:55490: Failed to load JWKS: Error downloading JWKS from 'https://localhost:41869/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: unable to get local issuer certificate
Metrics: {"server-negotiator.queue_time_us":73}
W20260809 12:10:28.860610 6707 negotiation.cc:337] Failed RPC negotiation. Trace:
0809 12:10:28.839861 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.222.62:33233 (local address 127.0.0.1:55490)
0809 12:10:28.840508 (+ 647us) negotiation.cc:107] Waiting for socket to connect
0809 12:10:28.840525 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0809 12:10:28.840646 (+ 121us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 12:10:28.841108 (+ 462us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 12:10:28.841112 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 12:10:28.841121 (+ 9us) client_negotiation.cc:189] Negotiated authn=JWT
0809 12:10:28.841384 (+ 263us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 12:10:28.841391 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 12:10:28.842846 (+ 1455us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 12:10:28.842850 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 12:10:28.843974 (+ 1124us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 12:10:28.843988 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 12:10:28.844987 (+ 999us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 12:10:28.844992 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 12:10:28.845130 (+ 138us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 12:10:28.845163 (+ 33us) client_negotiation.cc:252] Sending JWT_EXCHANGE NegotiatePB request
0809 12:10:28.860106 (+ 14943us) client_negotiation.cc:283] Received error response from server: Runtime error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:41869/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: unable to get local issuer certificate
0809 12:10:28.860310 (+ 204us) negotiation.cc:327] Negotiation complete: Runtime error: Client connection negotiation failed: client connection to 127.0.222.62:33233: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:41869/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: unable to get local issuer certificate
Metrics: {"client-negotiator.queue_time_us":514,"thread_start_us":371,"threads_started":1}
W20260809 12:10:28.860723 6678 negotiation.cc:344] Unauthorized connection attempt: Server connection negotiation failed: server connection from 127.0.0.1:55490: Failed to load JWKS: Error downloading JWKS from 'https://localhost:41869/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: unable to get local issuer certificate
I20260809 12:10:28.862190 888 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskgH5Pjm/build/asan/bin/kudu pid 6603
I20260809 12:10:28.863445 6668 generic_service.cc:196] Checking for leaks (request via RPC)
I20260809 12:10:28.961146 888 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskgH5Pjm/build/asan/bin/kudu with pid 6603
2026-08-09T12:10:29Z chronyd exiting
[ OK ] SecurityITest.TestJwtMiniClusterWithInvalidCert (772 ms)
[ RUN ] SecurityITest.TestJwtMiniClusterWithUntrustedCert
I20260809 12:10:29.021754 888 mini_oidc.cc:152] Starting JWKS server
I20260809 12:10:29.023852 888 webserver.cc:460] Webserver started at https://127.0.0.1:37655/ using document root <none> and password file <none>
I20260809 12:10:29.024132 888 mini_oidc.cc:189] Starting OIDC Discovery server
I20260809 12:10:29.024791 888 webserver.cc:460] Webserver started at http://127.0.0.1:37853/ using document root <none> and password file <none>
2026-08-09T12:10:29Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-09T12:10:29Z Disabled control of system clock
I20260809 12:10:29.060871 888 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskgH5Pjm/build/asan/bin/kudu
/tmp/dist-test-taskgH5Pjm/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskgH5Pjm/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1786277382073938-888-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskgH5Pjm/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1786277382073938-888-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskgH5Pjm/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1786277382073938-888-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskgH5Pjm/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1786277382073938-888-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.222.62:36091
--webserver_interface=127.0.222.62
--webserver_port=0
--builtin_ntp_servers=127.0.222.20:37025
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.222.62:36091
--enable_jwt_token_auth=true
--jwks_url=https://localhost:37655/jwks.json
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {}
W20260809 12:10:29.396543 6715 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260809 12:10:29.396919 6715 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260809 12:10:29.397085 6715 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260809 12:10:29.404574 6715 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260809 12:10:29.404733 6715 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260809 12:10:29.404835 6715 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260809 12:10:29.404956 6715 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260809 12:10:29.405052 6715 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
W20260809 12:10:29.405145 6715 flags.cc:425] Enabled experimental flag: --enable_jwt_token_auth=true
W20260809 12:10:29.405229 6715 flags.cc:425] Enabled experimental flag: --jwks_url=https://localhost:37655/jwks.json
I20260809 12:10:29.414656 6715 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.222.20:37025
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskgH5Pjm/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1786277382073938-888-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskgH5Pjm/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1786277382073938-888-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.222.62:36091
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.222.62:36091
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--enable_jwt_token_auth=true
--jwks_url=https://localhost:37655/jwks.json
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskgH5Pjm/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1786277382073938-888-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.222.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskgH5Pjm/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1786277382073938-888-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 09 Aug 2026 11:06:41 UTC on ccycloud.quasar-bwoogy.root.comops.site
build id 4575991
ASAN enabled
I20260809 12:10:29.416082 6715 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260809 12:10:29.417959 6715 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260809 12:10:29.430049 6720 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 12:10:29.430344 6721 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 12:10:29.430599 6723 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260809 12:10:29.431483 6715 server_base.cc:1034] running on GCE node
I20260809 12:10:29.432394 6715 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260809 12:10:29.433836 6715 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260809 12:10:29.435065 6715 hybrid_clock.cc:648] HybridClock initialized: now 1786277429435005 us; error 42 us; skew 500 ppm
I20260809 12:10:29.435432 6715 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260809 12:10:29.437727 6715 webserver.cc:460] Webserver started at http://127.0.222.62:45313/ using document root <none> and password file <none>
I20260809 12:10:29.438429 6715 fs_manager.cc:359] Metadata directory not provided
I20260809 12:10:29.438563 6715 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260809 12:10:29.438915 6715 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260809 12:10:29.441844 6715 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskgH5Pjm/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1786277382073938-888-0/minicluster-data/master-0/data/instance:
uuid: "0234c07a589f468087145397a9e93f4a"
format_stamp: "Formatted at 2026-08-09 12:10:29 on dist-test-slave-2w3w"
server_key: "dbc7788697e768b0bbd62827d27023ee"
server_key_iv: "e10aeaa6ea52153cbfadb4d24d5e4bfb"
server_key_version: "encryptionkey@0"
I20260809 12:10:29.442713 6715 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskgH5Pjm/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1786277382073938-888-0/minicluster-data/master-0/wal/instance:
uuid: "0234c07a589f468087145397a9e93f4a"
format_stamp: "Formatted at 2026-08-09 12:10:29 on dist-test-slave-2w3w"
server_key: "dbc7788697e768b0bbd62827d27023ee"
server_key_iv: "e10aeaa6ea52153cbfadb4d24d5e4bfb"
server_key_version: "encryptionkey@0"
I20260809 12:10:29.448017 6715 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.002s sys 0.001s
I20260809 12:10:29.451503 6730 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260809 12:10:29.452863 6715 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.001s
I20260809 12:10:29.453047 6715 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskgH5Pjm/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1786277382073938-888-0/minicluster-data/master-0/data,/tmp/dist-test-taskgH5Pjm/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1786277382073938-888-0/minicluster-data/master-0/wal
uuid: "0234c07a589f468087145397a9e93f4a"
format_stamp: "Formatted at 2026-08-09 12:10:29 on dist-test-slave-2w3w"
server_key: "dbc7788697e768b0bbd62827d27023ee"
server_key_iv: "e10aeaa6ea52153cbfadb4d24d5e4bfb"
server_key_version: "encryptionkey@0"
I20260809 12:10:29.453284 6715 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskgH5Pjm/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1786277382073938-888-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskgH5Pjm/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1786277382073938-888-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskgH5Pjm/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1786277382073938-888-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260809 12:10:29.472486 6715 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260809 12:10:29.473362 6715 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260809 12:10:29.473692 6715 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260809 12:10:29.492345 6715 rpc_server.cc:307] RPC server started. Bound to: 127.0.222.62:36091
I20260809 12:10:29.492404 6782 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.222.62:36091 every 8 connection(s)
I20260809 12:10:29.493767 6715 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskgH5Pjm/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1786277382073938-888-0/minicluster-data/master-0/data/info.pb
I20260809 12:10:29.497484 6783 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260809 12:10:29.501534 888 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskgH5Pjm/build/asan/bin/kudu as pid 6715
I20260809 12:10:29.501754 888 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskgH5Pjm/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1786277382073938-888-0/minicluster-data/master-0/wal/instance
I20260809 12:10:29.502413 888 external_mini_cluster.cc:1428] Setting key f1ed52acbdcd429a91fc020df85a09c4
I20260809 12:10:29.510165 6783 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 0234c07a589f468087145397a9e93f4a: Bootstrap starting.
I20260809 12:10:29.513846 6783 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 0234c07a589f468087145397a9e93f4a: Neither blocks nor log segments found. Creating new log.
I20260809 12:10:29.515210 6783 log.cc:824] T 00000000000000000000000000000000 P 0234c07a589f468087145397a9e93f4a: Log is configured to *not* fsync() on all Append() calls
I20260809 12:10:29.518620 6783 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 0234c07a589f468087145397a9e93f4a: No bootstrap required, opened a new log
I20260809 12:10:29.518893 6786 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 12:10:29.504381 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46052 (local address 127.0.222.62:36091)
0809 12:10:29.505566 (+ 1185us) server_negotiation.cc:206] Beginning negotiation
0809 12:10:29.505582 (+ 16us) server_negotiation.cc:399] Waiting for connection header
0809 12:10:29.505632 (+ 50us) server_negotiation.cc:407] Connection header received
0809 12:10:29.508326 (+ 2694us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 12:10:29.508354 (+ 28us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 12:10:29.508942 (+ 588us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 12:10:29.509378 (+ 436us) server_negotiation.cc:226] Negotiated authn=SASL
0809 12:10:29.509933 (+ 555us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 12:10:29.511982 (+ 2049us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 12:10:29.513405 (+ 1423us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 12:10:29.514526 (+ 1121us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 12:10:29.514726 (+ 200us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 12:10:29.516066 (+ 1340us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 12:10:29.516074 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 12:10:29.516087 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0809 12:10:29.516733 (+ 646us) server_negotiation.cc:922] calling sasl_server_start()
0809 12:10:29.516837 (+ 104us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0809 12:10:29.516873 (+ 36us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 12:10:29.516986 (+ 113us) server_negotiation.cc:1035] Waiting for connection context
0809 12:10:29.517239 (+ 253us) server_negotiation.cc:299] Negotiation successful
0809 12:10:29.517583 (+ 344us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":638,"thread_start_us":403,"threads_started":1}
I20260809 12:10:29.523072 888 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260809 12:10:29.528064 6783 raft_consensus.cc:348] T 00000000000000000000000000000000 P 0234c07a589f468087145397a9e93f4a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0234c07a589f468087145397a9e93f4a" member_type: VOTER last_known_addr { host: "127.0.222.62" port: 36091 } }
I20260809 12:10:29.528432 6783 raft_consensus.cc:374] T 00000000000000000000000000000000 P 0234c07a589f468087145397a9e93f4a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260809 12:10:29.528515 6783 raft_consensus.cc:729] T 00000000000000000000000000000000 P 0234c07a589f468087145397a9e93f4a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0234c07a589f468087145397a9e93f4a, State: Initialized, Role: FOLLOWER
I20260809 12:10:29.529016 6783 consensus_queue.cc:260] T 00000000000000000000000000000000 P 0234c07a589f468087145397a9e93f4a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0234c07a589f468087145397a9e93f4a" member_type: VOTER last_known_addr { host: "127.0.222.62" port: 36091 } }
I20260809 12:10:29.529220 6783 raft_consensus.cc:388] T 00000000000000000000000000000000 P 0234c07a589f468087145397a9e93f4a [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260809 12:10:29.529320 6783 raft_consensus.cc:482] T 00000000000000000000000000000000 P 0234c07a589f468087145397a9e93f4a [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260809 12:10:29.529446 6783 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 0234c07a589f468087145397a9e93f4a [term 0 FOLLOWER]: Advancing to term 1
I20260809 12:10:29.531720 6783 raft_consensus.cc:504] T 00000000000000000000000000000000 P 0234c07a589f468087145397a9e93f4a [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0234c07a589f468087145397a9e93f4a" member_type: VOTER last_known_addr { host: "127.0.222.62" port: 36091 } }
I20260809 12:10:29.532235 6783 leader_election.cc:302] T 00000000000000000000000000000000 P 0234c07a589f468087145397a9e93f4a [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 0234c07a589f468087145397a9e93f4a; no voters:
I20260809 12:10:29.532835 6783 leader_election.cc:288] T 00000000000000000000000000000000 P 0234c07a589f468087145397a9e93f4a [CANDIDATE]: Term 1 election: Requested vote from peers
I20260809 12:10:29.533017 6789 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 0234c07a589f468087145397a9e93f4a [term 1 FOLLOWER]: Leader election won for term 1
I20260809 12:10:29.533794 6789 raft_consensus.cc:686] T 00000000000000000000000000000000 P 0234c07a589f468087145397a9e93f4a [term 1 LEADER]: Becoming Leader. State: Replica: 0234c07a589f468087145397a9e93f4a, State: Running, Role: LEADER
I20260809 12:10:29.534405 6789 consensus_queue.cc:237] T 00000000000000000000000000000000 P 0234c07a589f468087145397a9e93f4a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0234c07a589f468087145397a9e93f4a" member_type: VOTER last_known_addr { host: "127.0.222.62" port: 36091 } }
I20260809 12:10:29.535187 6783 sys_catalog.cc:564] T 00000000000000000000000000000000 P 0234c07a589f468087145397a9e93f4a [sys.catalog]: configured and running, proceeding with master startup.
I20260809 12:10:29.536988 6790 sys_catalog.cc:455] T 00000000000000000000000000000000 P 0234c07a589f468087145397a9e93f4a [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "0234c07a589f468087145397a9e93f4a" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0234c07a589f468087145397a9e93f4a" member_type: VOTER last_known_addr { host: "127.0.222.62" port: 36091 } } }
I20260809 12:10:29.537326 6790 sys_catalog.cc:458] T 00000000000000000000000000000000 P 0234c07a589f468087145397a9e93f4a [sys.catalog]: This master's current role is: LEADER
I20260809 12:10:29.538204 6794 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260809 12:10:29.538581 6791 sys_catalog.cc:455] T 00000000000000000000000000000000 P 0234c07a589f468087145397a9e93f4a [sys.catalog]: SysCatalogTable state changed. Reason: New leader 0234c07a589f468087145397a9e93f4a. Latest consensus state: current_term: 1 leader_uuid: "0234c07a589f468087145397a9e93f4a" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0234c07a589f468087145397a9e93f4a" member_type: VOTER last_known_addr { host: "127.0.222.62" port: 36091 } } }
I20260809 12:10:29.538906 6791 sys_catalog.cc:458] T 00000000000000000000000000000000 P 0234c07a589f468087145397a9e93f4a [sys.catalog]: This master's current role is: LEADER
I20260809 12:10:29.547914 6794 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260809 12:10:29.559203 6794 catalog_manager.cc:1347] Generated new cluster ID: f65a0f4320e044c88fe8f142320d40ce
I20260809 12:10:29.559340 6794 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260809 12:10:29.590463 6794 catalog_manager.cc:1370] Generated new certificate authority record
I20260809 12:10:29.592406 6794 catalog_manager.cc:1504] Loading token signing keys...
I20260809 12:10:29.611946 6794 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 0234c07a589f468087145397a9e93f4a: Generated new TSK 0
I20260809 12:10:29.613080 6794 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260809 12:10:29.650619 6786 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 12:10:29.641955 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46062 (local address 127.0.222.62:36091)
0809 12:10:29.642175 (+ 220us) server_negotiation.cc:206] Beginning negotiation
0809 12:10:29.642182 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0809 12:10:29.642595 (+ 413us) server_negotiation.cc:407] Connection header received
0809 12:10:29.642809 (+ 214us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 12:10:29.642815 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 12:10:29.642911 (+ 96us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 12:10:29.643136 (+ 225us) server_negotiation.cc:226] Negotiated authn=JWT
0809 12:10:29.643591 (+ 455us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 12:10:29.644763 (+ 1172us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 12:10:29.646118 (+ 1355us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 12:10:29.646850 (+ 732us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 12:10:29.647015 (+ 165us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 12:10:29.647533 (+ 518us) server_negotiation.cc:365] Received JWT_EXCHANGE NegotiatePB request
0809 12:10:29.649939 (+ 2406us) server_negotiation.cc:394] Sending RPC error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:37655/jwks.json': curl error: Problem with the SSL CA cert (path? access rights?): error setting certificate verify locations:
CAfile: /etc/pki/tls/certs/ca-bundle.crt
CApath: none
0809 12:10:29.650283 (+ 344us) negotiation.cc:327] Negotiation complete: Not authorized: Server connection negotiation failed: server connection from 127.0.0.1:46062: Failed to load JWKS: Error downloading JWKS from 'https://localhost:37655/jwks.json': curl error: Problem with the SSL CA cert (path? access rights?): error setting certificate verify locations:
CAfile: /etc/pki/tls/certs/ca-bundle.crt
CApath: none
Metrics: {"server-negotiator.queue_time_us":72}
W20260809 12:10:29.650763 6786 negotiation.cc:344] Unauthorized connection attempt: Server connection negotiation failed: server connection from 127.0.0.1:46062: Failed to load JWKS: Error downloading JWKS from 'https://localhost:37655/jwks.json': curl error: Problem with the SSL CA cert (path? access rights?): error setting certificate verify locations:
CAfile: /etc/pki/tls/certs/ca-bundle.crt
CApath: none
W20260809 12:10:29.652356 6815 negotiation.cc:337] Failed RPC negotiation. Trace:
0809 12:10:29.641768 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.222.62:36091 (local address 127.0.0.1:46062)
0809 12:10:29.642493 (+ 725us) negotiation.cc:107] Waiting for socket to connect
0809 12:10:29.642509 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0809 12:10:29.642631 (+ 122us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 12:10:29.643144 (+ 513us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 12:10:29.643147 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 12:10:29.643158 (+ 11us) client_negotiation.cc:189] Negotiated authn=JWT
0809 12:10:29.643434 (+ 276us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 12:10:29.643442 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 12:10:29.644909 (+ 1467us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 12:10:29.644913 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 12:10:29.645974 (+ 1061us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 12:10:29.645986 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 12:10:29.647092 (+ 1106us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 12:10:29.647097 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 12:10:29.647232 (+ 135us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 12:10:29.647264 (+ 32us) client_negotiation.cc:252] Sending JWT_EXCHANGE NegotiatePB request
0809 12:10:29.650165 (+ 2901us) client_negotiation.cc:283] Received error response from server: Runtime error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:37655/jwks.json': curl error: Problem with the SSL CA cert (path? access rights?): error setting certificate verify locations:
CAfile: /etc/pki/tls/certs/ca-bundle.crt
CApath: none
0809 12:10:29.651903 (+ 1738us) negotiation.cc:327] Negotiation complete: Runtime error: Client connection negotiation failed: client connection to 127.0.222.62:36091: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:37655/jwks.json': curl error: Problem with the SSL CA cert (path? access rights?): error setting certificate verify locations:
CAfile: /etc/pki/tls/certs/ca-bundle.crt
CApath: none
Metrics: {"client-negotiator.queue_time_us":581,"mutex_wait_us":1355,"thread_start_us":432,"threads_started":1}
I20260809 12:10:29.653681 888 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskgH5Pjm/build/asan/bin/kudu pid 6715
I20260809 12:10:29.654906 6778 generic_service.cc:196] Checking for leaks (request via RPC)
E20260809 12:10:29.665512 6817 webserver.cc:533] Webserver: sslize failed:
E20260809 12:10:29.665647 6817 webserver.cc:533] Webserver: error shutting down SSL connection: error:140E0197:SSL routines:SSL_shutdown:shutdown while in init
I20260809 12:10:29.754222 888 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskgH5Pjm/build/asan/bin/kudu with pid 6715
2026-08-09T12:10:29Z chronyd exiting
[ OK ] SecurityITest.TestJwtMiniClusterWithUntrustedCert (793 ms)
[ RUN ] SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS
2026-08-09T12:10:29Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-09T12:10:29Z Disabled control of system clock
I20260809 12:10:29.850739 888 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskgH5Pjm/build/asan/bin/kudu
/tmp/dist-test-taskgH5Pjm/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskgH5Pjm/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1786277382073938-888-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskgH5Pjm/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1786277382073938-888-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskgH5Pjm/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1786277382073938-888-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskgH5Pjm/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1786277382073938-888-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.222.62:41681
--webserver_interface=127.0.222.62
--webserver_port=0
--builtin_ntp_servers=127.0.222.20:32791
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.222.62:41681
--enable_jwt_token_auth=true
--jwks_url=default.url
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--trusted_certificate_file=/tmp/dist-test-taskgH5Pjm/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1786277382073938-888-0/testchainca.cert with env {}
W20260809 12:10:30.196270 6822 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260809 12:10:30.196595 6822 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260809 12:10:30.196767 6822 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260809 12:10:30.204280 6822 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260809 12:10:30.204404 6822 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260809 12:10:30.204473 6822 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260809 12:10:30.204599 6822 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260809 12:10:30.204670 6822 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
W20260809 12:10:30.204758 6822 flags.cc:425] Enabled experimental flag: --enable_jwt_token_auth=true
W20260809 12:10:30.204823 6822 flags.cc:425] Enabled experimental flag: --jwks_url=default.url
I20260809 12:10:30.215322 6822 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.222.20:32791
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskgH5Pjm/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1786277382073938-888-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskgH5Pjm/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1786277382073938-888-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.222.62:41681
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.222.62:41681
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--enable_jwt_token_auth=true
--jwks_url=default.url
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskgH5Pjm/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1786277382073938-888-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.222.62
--webserver_port=0
--trusted_certificate_file=/tmp/dist-test-taskgH5Pjm/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1786277382073938-888-0/testchainca.cert
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskgH5Pjm/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1786277382073938-888-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 09 Aug 2026 11:06:41 UTC on ccycloud.quasar-bwoogy.root.comops.site
build id 4575991
ASAN enabled
I20260809 12:10:30.216763 6822 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260809 12:10:30.218127 6822 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260809 12:10:30.230712 6828 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 12:10:30.230686 6831 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 12:10:30.230988 6829 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260809 12:10:30.232493 6822 server_base.cc:1034] running on GCE node
I20260809 12:10:30.233315 6822 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260809 12:10:30.234732 6822 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260809 12:10:30.235960 6822 hybrid_clock.cc:648] HybridClock initialized: now 1786277430235908 us; error 47 us; skew 500 ppm
I20260809 12:10:30.236286 6822 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260809 12:10:30.238600 6822 webserver.cc:460] Webserver started at http://127.0.222.62:44879/ using document root <none> and password file <none>
I20260809 12:10:30.239351 6822 fs_manager.cc:359] Metadata directory not provided
I20260809 12:10:30.239449 6822 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260809 12:10:30.239732 6822 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260809 12:10:30.242718 6822 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskgH5Pjm/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1786277382073938-888-0/minicluster-data/master-0/data/instance:
uuid: "c7b58562a89948c795c2efbb4afa2de2"
format_stamp: "Formatted at 2026-08-09 12:10:30 on dist-test-slave-2w3w"
server_key: "9df58f949222e0e75813cee8339c3986"
server_key_iv: "42eef31c4637c03926e284b3a57250e5"
server_key_version: "encryptionkey@0"
I20260809 12:10:30.243623 6822 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskgH5Pjm/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1786277382073938-888-0/minicluster-data/master-0/wal/instance:
uuid: "c7b58562a89948c795c2efbb4afa2de2"
format_stamp: "Formatted at 2026-08-09 12:10:30 on dist-test-slave-2w3w"
server_key: "9df58f949222e0e75813cee8339c3986"
server_key_iv: "42eef31c4637c03926e284b3a57250e5"
server_key_version: "encryptionkey@0"
I20260809 12:10:30.249044 6822 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.001s sys 0.003s
I20260809 12:10:30.252511 6837 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260809 12:10:30.253898 6822 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.000s
I20260809 12:10:30.254104 6822 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskgH5Pjm/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1786277382073938-888-0/minicluster-data/master-0/data,/tmp/dist-test-taskgH5Pjm/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1786277382073938-888-0/minicluster-data/master-0/wal
uuid: "c7b58562a89948c795c2efbb4afa2de2"
format_stamp: "Formatted at 2026-08-09 12:10:30 on dist-test-slave-2w3w"
server_key: "9df58f949222e0e75813cee8339c3986"
server_key_iv: "42eef31c4637c03926e284b3a57250e5"
server_key_version: "encryptionkey@0"
I20260809 12:10:30.254350 6822 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskgH5Pjm/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1786277382073938-888-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskgH5Pjm/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1786277382073938-888-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskgH5Pjm/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1786277382073938-888-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260809 12:10:30.274255 6822 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260809 12:10:30.275228 6822 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260809 12:10:30.275588 6822 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260809 12:10:30.294445 6822 rpc_server.cc:307] RPC server started. Bound to: 127.0.222.62:41681
I20260809 12:10:30.294464 6890 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.222.62:41681 every 8 connection(s)
I20260809 12:10:30.295848 6822 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskgH5Pjm/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1786277382073938-888-0/minicluster-data/master-0/data/info.pb
I20260809 12:10:30.299728 6891 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260809 12:10:30.303310 888 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskgH5Pjm/build/asan/bin/kudu as pid 6822
I20260809 12:10:30.303515 888 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskgH5Pjm/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1786277382073938-888-0/minicluster-data/master-0/wal/instance
I20260809 12:10:30.304384 888 external_mini_cluster.cc:1428] Setting key b7dfa5beb808cacd7239e4c219b613ac
I20260809 12:10:30.313024 6891 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P c7b58562a89948c795c2efbb4afa2de2: Bootstrap starting.
I20260809 12:10:30.316804 6891 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P c7b58562a89948c795c2efbb4afa2de2: Neither blocks nor log segments found. Creating new log.
I20260809 12:10:30.317956 6891 log.cc:824] T 00000000000000000000000000000000 P c7b58562a89948c795c2efbb4afa2de2: Log is configured to *not* fsync() on all Append() calls
I20260809 12:10:30.318322 6894 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 12:10:30.306358 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40024 (local address 127.0.222.62:41681)
0809 12:10:30.307390 (+ 1032us) server_negotiation.cc:206] Beginning negotiation
0809 12:10:30.307400 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0809 12:10:30.307443 (+ 43us) server_negotiation.cc:407] Connection header received
0809 12:10:30.309052 (+ 1609us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 12:10:30.309074 (+ 22us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 12:10:30.309548 (+ 474us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 12:10:30.309879 (+ 331us) server_negotiation.cc:226] Negotiated authn=SASL
0809 12:10:30.310499 (+ 620us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 12:10:30.311818 (+ 1319us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 12:10:30.313629 (+ 1811us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 12:10:30.314354 (+ 725us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 12:10:30.314498 (+ 144us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 12:10:30.315863 (+ 1365us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 12:10:30.315869 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 12:10:30.315883 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0809 12:10:30.316438 (+ 555us) server_negotiation.cc:922] calling sasl_server_start()
0809 12:10:30.316532 (+ 94us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0809 12:10:30.316553 (+ 21us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 12:10:30.316640 (+ 87us) server_negotiation.cc:1035] Waiting for connection context
0809 12:10:30.317050 (+ 410us) server_negotiation.cc:299] Negotiation successful
0809 12:10:30.317354 (+ 304us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":632,"thread_start_us":362,"threads_started":1}
I20260809 12:10:30.320909 6891 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P c7b58562a89948c795c2efbb4afa2de2: No bootstrap required, opened a new log
I20260809 12:10:30.322216 888 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260809 12:10:30.329047 6891 raft_consensus.cc:348] T 00000000000000000000000000000000 P c7b58562a89948c795c2efbb4afa2de2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c7b58562a89948c795c2efbb4afa2de2" member_type: VOTER last_known_addr { host: "127.0.222.62" port: 41681 } }
I20260809 12:10:30.329421 6891 raft_consensus.cc:374] T 00000000000000000000000000000000 P c7b58562a89948c795c2efbb4afa2de2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260809 12:10:30.329547 6891 raft_consensus.cc:729] T 00000000000000000000000000000000 P c7b58562a89948c795c2efbb4afa2de2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c7b58562a89948c795c2efbb4afa2de2, State: Initialized, Role: FOLLOWER
I20260809 12:10:30.330101 6891 consensus_queue.cc:260] T 00000000000000000000000000000000 P c7b58562a89948c795c2efbb4afa2de2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c7b58562a89948c795c2efbb4afa2de2" member_type: VOTER last_known_addr { host: "127.0.222.62" port: 41681 } }
I20260809 12:10:30.330348 6891 raft_consensus.cc:388] T 00000000000000000000000000000000 P c7b58562a89948c795c2efbb4afa2de2 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260809 12:10:30.330507 6891 raft_consensus.cc:482] T 00000000000000000000000000000000 P c7b58562a89948c795c2efbb4afa2de2 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260809 12:10:30.330675 6891 raft_consensus.cc:3037] T 00000000000000000000000000000000 P c7b58562a89948c795c2efbb4afa2de2 [term 0 FOLLOWER]: Advancing to term 1
I20260809 12:10:30.332923 6891 raft_consensus.cc:504] T 00000000000000000000000000000000 P c7b58562a89948c795c2efbb4afa2de2 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c7b58562a89948c795c2efbb4afa2de2" member_type: VOTER last_known_addr { host: "127.0.222.62" port: 41681 } }
I20260809 12:10:30.333474 6891 leader_election.cc:302] T 00000000000000000000000000000000 P c7b58562a89948c795c2efbb4afa2de2 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: c7b58562a89948c795c2efbb4afa2de2; no voters:
I20260809 12:10:30.334144 6891 leader_election.cc:288] T 00000000000000000000000000000000 P c7b58562a89948c795c2efbb4afa2de2 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260809 12:10:30.334251 6896 raft_consensus.cc:2781] T 00000000000000000000000000000000 P c7b58562a89948c795c2efbb4afa2de2 [term 1 FOLLOWER]: Leader election won for term 1
I20260809 12:10:30.335753 6896 raft_consensus.cc:686] T 00000000000000000000000000000000 P c7b58562a89948c795c2efbb4afa2de2 [term 1 LEADER]: Becoming Leader. State: Replica: c7b58562a89948c795c2efbb4afa2de2, State: Running, Role: LEADER
I20260809 12:10:30.336265 6896 consensus_queue.cc:237] T 00000000000000000000000000000000 P c7b58562a89948c795c2efbb4afa2de2 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c7b58562a89948c795c2efbb4afa2de2" member_type: VOTER last_known_addr { host: "127.0.222.62" port: 41681 } }
I20260809 12:10:30.336524 6891 sys_catalog.cc:564] T 00000000000000000000000000000000 P c7b58562a89948c795c2efbb4afa2de2 [sys.catalog]: configured and running, proceeding with master startup.
I20260809 12:10:30.339005 6898 sys_catalog.cc:455] T 00000000000000000000000000000000 P c7b58562a89948c795c2efbb4afa2de2 [sys.catalog]: SysCatalogTable state changed. Reason: New leader c7b58562a89948c795c2efbb4afa2de2. Latest consensus state: current_term: 1 leader_uuid: "c7b58562a89948c795c2efbb4afa2de2" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c7b58562a89948c795c2efbb4afa2de2" member_type: VOTER last_known_addr { host: "127.0.222.62" port: 41681 } } }
I20260809 12:10:30.339416 6898 sys_catalog.cc:458] T 00000000000000000000000000000000 P c7b58562a89948c795c2efbb4afa2de2 [sys.catalog]: This master's current role is: LEADER
I20260809 12:10:30.339202 6897 sys_catalog.cc:455] T 00000000000000000000000000000000 P c7b58562a89948c795c2efbb4afa2de2 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "c7b58562a89948c795c2efbb4afa2de2" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c7b58562a89948c795c2efbb4afa2de2" member_type: VOTER last_known_addr { host: "127.0.222.62" port: 41681 } } }
I20260809 12:10:30.339661 6897 sys_catalog.cc:458] T 00000000000000000000000000000000 P c7b58562a89948c795c2efbb4afa2de2 [sys.catalog]: This master's current role is: LEADER
I20260809 12:10:30.341455 6904 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260809 12:10:30.347038 6904 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260809 12:10:30.357613 6904 catalog_manager.cc:1347] Generated new cluster ID: 7bb09fcac5d04b50bbafc7ba3b6c98db
I20260809 12:10:30.357740 6904 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260809 12:10:30.391650 6904 catalog_manager.cc:1370] Generated new certificate authority record
I20260809 12:10:30.392959 6904 catalog_manager.cc:1504] Loading token signing keys...
I20260809 12:10:30.401656 6904 catalog_manager.cc:5868] T 00000000000000000000000000000000 P c7b58562a89948c795c2efbb4afa2de2: Generated new TSK 0
I20260809 12:10:30.402516 6904 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260809 12:10:30.467979 6894 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 12:10:30.422546 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40032 (local address 127.0.222.62:41681)
0809 12:10:30.422714 (+ 168us) server_negotiation.cc:206] Beginning negotiation
0809 12:10:30.422719 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0809 12:10:30.423160 (+ 441us) server_negotiation.cc:407] Connection header received
0809 12:10:30.423396 (+ 236us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 12:10:30.423402 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 12:10:30.423504 (+ 102us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 12:10:30.423679 (+ 175us) server_negotiation.cc:226] Negotiated authn=JWT
0809 12:10:30.424242 (+ 563us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 12:10:30.425072 (+ 830us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 12:10:30.426625 (+ 1553us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 12:10:30.427180 (+ 555us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 12:10:30.427324 (+ 144us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 12:10:30.427869 (+ 545us) server_negotiation.cc:365] Received JWT_EXCHANGE NegotiatePB request
0809 12:10:30.467198 (+ 39329us) server_negotiation.cc:394] Sending RPC error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'default.url': curl error: Couldn't resolve host name: Could not resolve host: default.url
0809 12:10:30.467656 (+ 458us) negotiation.cc:327] Negotiation complete: Not authorized: Server connection negotiation failed: server connection from 127.0.0.1:40032: Failed to load JWKS: Error downloading JWKS from 'default.url': curl error: Couldn't resolve host name: Could not resolve host: default.url
Metrics: {"server-negotiator.queue_time_us":47}
W20260809 12:10:30.468149 6894 negotiation.cc:344] Unauthorized connection attempt: Server connection negotiation failed: server connection from 127.0.0.1:40032: Failed to load JWKS: Error downloading JWKS from 'default.url': curl error: Couldn't resolve host name: Could not resolve host: default.url
W20260809 12:10:30.468297 6924 negotiation.cc:337] Failed RPC negotiation. Trace:
0809 12:10:30.422389 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.222.62:41681 (local address 127.0.0.1:40032)
0809 12:10:30.423055 (+ 666us) negotiation.cc:107] Waiting for socket to connect
0809 12:10:30.423073 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0809 12:10:30.423198 (+ 125us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 12:10:30.423704 (+ 506us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 12:10:30.423707 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 12:10:30.423717 (+ 10us) client_negotiation.cc:189] Negotiated authn=JWT
0809 12:10:30.424069 (+ 352us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 12:10:30.424079 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 12:10:30.425237 (+ 1158us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 12:10:30.425243 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 12:10:30.426453 (+ 1210us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 12:10:30.426466 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 12:10:30.427382 (+ 916us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 12:10:30.427392 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 12:10:30.427559 (+ 167us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 12:10:30.427596 (+ 37us) client_negotiation.cc:252] Sending JWT_EXCHANGE NegotiatePB request
0809 12:10:30.467499 (+ 39903us) client_negotiation.cc:283] Received error response from server: Runtime error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'default.url': curl error: Couldn't resolve host name: Could not resolve host: default.url
0809 12:10:30.467814 (+ 315us) negotiation.cc:327] Negotiation complete: Runtime error: Client connection negotiation failed: client connection to 127.0.222.62:41681: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'default.url': curl error: Couldn't resolve host name: Could not resolve host: default.url
Metrics: {"client-negotiator.queue_time_us":487,"thread_start_us":344,"threads_started":1}
I20260809 12:10:30.469795 888 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskgH5Pjm/build/asan/bin/kudu pid 6822
I20260809 12:10:30.471022 6886 generic_service.cc:196] Checking for leaks (request via RPC)
I20260809 12:10:30.569620 888 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskgH5Pjm/build/asan/bin/kudu with pid 6822
2026-08-09T12:10:30Z chronyd exiting
[ OK ] SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS (793 ms)
[ RUN ] SecurityITest.TestWorldReadableKeytab
[ OK ] SecurityITest.TestWorldReadableKeytab (440 ms)
[ RUN ] SecurityITest.TestWorldReadablePrivateKey
[ OK ] SecurityITest.TestWorldReadablePrivateKey (431 ms)
[ RUN ] SecurityITest.TestCorruptKerberosCC
Loading random data
Initializing database '/tmp/dist-test-taskgH5Pjm/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786277382073938-888-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 09 12:10:31 dist-test-slave-2w3w krb5kdc[6939](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 09 12:10:31 dist-test-slave-2w3w krb5kdc[6939](info): set up 2 sockets
Aug 09 12:10:31 dist-test-slave-2w3w krb5kdc[6939](info): commencing operation
krb5kdc: starting...
W20260809 12:10:32.609536 888 mini_kdc.cc:121] Time spent starting KDC: real 1.127s user 0.000s sys 0.029s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskgH5Pjm/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786277382073938-888-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskgH5Pjm/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786277382073938-888-0/krb5kdc/test-user.keytab.
Aug 09 12:10:32 dist-test-slave-2w3w krb5kdc[6939](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786277432, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-09T12:10:32Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-09T12:10:32Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.222.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.222.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.222.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskgH5Pjm/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786277382073938-888-0/krb5kdc/kudu_127.0.222.62.keytab.
Entry for principal kudu/127.0.222.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskgH5Pjm/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786277382073938-888-0/krb5kdc/kudu_127.0.222.62.keytab.
I20260809 12:10:32.867660 888 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskgH5Pjm/build/asan/bin/kudu
/tmp/dist-test-taskgH5Pjm/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskgH5Pjm/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786277382073938-888-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskgH5Pjm/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786277382073938-888-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskgH5Pjm/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786277382073938-888-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskgH5Pjm/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786277382073938-888-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.222.62:41493
--webserver_interface=127.0.222.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskgH5Pjm/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786277382073938-888-0/krb5kdc/kudu_127.0.222.62.keytab
--principal=kudu/127.0.222.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.222.20:41337
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.222.62:41493
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskgH5Pjm/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786277382073938-888-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskgH5Pjm/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786277382073938-888-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskgH5Pjm/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786277382073938-888-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260809 12:10:33.205715 6954 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260809 12:10:33.206030 6954 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260809 12:10:33.206199 6954 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260809 12:10:33.213598 6954 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260809 12:10:33.213757 6954 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260809 12:10:33.213822 6954 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260809 12:10:33.213907 6954 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260809 12:10:33.213986 6954 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260809 12:10:33.224015 6954 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.222.20:41337
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskgH5Pjm/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786277382073938-888-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskgH5Pjm/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786277382073938-888-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.222.62:41493
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskgH5Pjm/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786277382073938-888-0/krb5kdc/kudu_127.0.222.62.keytab
--principal=kudu/127.0.222.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.222.62:41493
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskgH5Pjm/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786277382073938-888-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.222.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskgH5Pjm/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786277382073938-888-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 09 Aug 2026 11:06:41 UTC on ccycloud.quasar-bwoogy.root.comops.site
build id 4575991
ASAN enabled
I20260809 12:10:33.225402 6954 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260809 12:10:33.226801 6954 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260809 12:10:33.239012 6959 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 12:10:33.239069 6960 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 12:10:33.239071 6962 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260809 12:10:33.240156 6954 server_base.cc:1034] running on GCE node
I20260809 12:10:33.240917 6954 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260809 12:10:33.242390 6954 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260809 12:10:33.243620 6954 hybrid_clock.cc:648] HybridClock initialized: now 1786277433243586 us; error 46 us; skew 500 ppm
I20260809 12:10:33.243993 6954 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 09 12:10:33 dist-test-slave-2w3w krb5kdc[6939](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786277433, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.222.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260809 12:10:33.250207 6954 init.cc:377] Logged in from keytab as kudu/127.0.222.62@KRBTEST.COM (short username kudu)
I20260809 12:10:33.251703 6954 webserver.cc:460] Webserver started at http://127.0.222.62:44559/ using document root <none> and password file <none>
I20260809 12:10:33.252363 6954 fs_manager.cc:359] Metadata directory not provided
I20260809 12:10:33.252470 6954 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260809 12:10:33.252734 6954 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260809 12:10:33.255959 6954 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskgH5Pjm/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786277382073938-888-0/minicluster-data/master-0/data/instance:
uuid: "69e8333fc1194475a902f1ea5f1e347b"
format_stamp: "Formatted at 2026-08-09 12:10:33 on dist-test-slave-2w3w"
server_key: "8fcd59e03cb06195526334282a43d062"
server_key_iv: "258975201c588e1e2a6531195b154ba7"
server_key_version: "encryptionkey@0"
I20260809 12:10:33.256718 6954 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskgH5Pjm/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786277382073938-888-0/minicluster-data/master-0/wal/instance:
uuid: "69e8333fc1194475a902f1ea5f1e347b"
format_stamp: "Formatted at 2026-08-09 12:10:33 on dist-test-slave-2w3w"
server_key: "8fcd59e03cb06195526334282a43d062"
server_key_iv: "258975201c588e1e2a6531195b154ba7"
server_key_version: "encryptionkey@0"
I20260809 12:10:33.261773 6954 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.000s
I20260809 12:10:33.265120 6969 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260809 12:10:33.266394 6954 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260809 12:10:33.266533 6954 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskgH5Pjm/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786277382073938-888-0/minicluster-data/master-0/data,/tmp/dist-test-taskgH5Pjm/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786277382073938-888-0/minicluster-data/master-0/wal
uuid: "69e8333fc1194475a902f1ea5f1e347b"
format_stamp: "Formatted at 2026-08-09 12:10:33 on dist-test-slave-2w3w"
server_key: "8fcd59e03cb06195526334282a43d062"
server_key_iv: "258975201c588e1e2a6531195b154ba7"
server_key_version: "encryptionkey@0"
I20260809 12:10:33.266695 6954 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskgH5Pjm/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786277382073938-888-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskgH5Pjm/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786277382073938-888-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskgH5Pjm/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786277382073938-888-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260809 12:10:33.284721 6954 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260809 12:10:33.288676 6954 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260809 12:10:33.288920 6954 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260809 12:10:33.307806 6954 rpc_server.cc:307] RPC server started. Bound to: 127.0.222.62:41493
I20260809 12:10:33.307884 7023 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.222.62:41493 every 8 connection(s)
I20260809 12:10:33.309175 6954 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskgH5Pjm/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786277382073938-888-0/minicluster-data/master-0/data/info.pb
I20260809 12:10:33.312875 7024 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260809 12:10:33.313655 888 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskgH5Pjm/build/asan/bin/kudu as pid 6954
I20260809 12:10:33.313899 888 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskgH5Pjm/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786277382073938-888-0/minicluster-data/master-0/wal/instance
I20260809 12:10:33.314796 888 external_mini_cluster.cc:1428] Setting key a5e773ca169a4bbf78491e020069fa48
I20260809 12:10:33.327610 7024 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 69e8333fc1194475a902f1ea5f1e347b: Bootstrap starting.
Aug 09 12:10:33 dist-test-slave-2w3w krb5kdc[6939](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786277432, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.222.62@KRBTEST.COM
I20260809 12:10:33.331240 7024 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 69e8333fc1194475a902f1ea5f1e347b: Neither blocks nor log segments found. Creating new log.
I20260809 12:10:33.332389 7024 log.cc:824] T 00000000000000000000000000000000 P 69e8333fc1194475a902f1ea5f1e347b: Log is configured to *not* fsync() on all Append() calls
I20260809 12:10:33.335310 7024 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 69e8333fc1194475a902f1ea5f1e347b: No bootstrap required, opened a new log
I20260809 12:10:33.338376 7027 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 12:10:33.316918 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59160 (local address 127.0.222.62:41493)
0809 12:10:33.317738 (+ 820us) server_negotiation.cc:206] Beginning negotiation
0809 12:10:33.317752 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0809 12:10:33.317807 (+ 55us) server_negotiation.cc:407] Connection header received
0809 12:10:33.319617 (+ 1810us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 12:10:33.319642 (+ 25us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 12:10:33.320030 (+ 388us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 12:10:33.320405 (+ 375us) server_negotiation.cc:226] Negotiated authn=SASL
0809 12:10:33.322133 (+ 1728us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 12:10:33.323485 (+ 1352us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 12:10:33.325264 (+ 1779us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 12:10:33.325927 (+ 663us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 12:10:33.326050 (+ 123us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 12:10:33.330798 (+ 4748us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 12:10:33.330806 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 12:10:33.330820 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0809 12:10:33.330855 (+ 35us) server_negotiation.cc:922] calling sasl_server_start()
0809 12:10:33.334198 (+ 3343us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 12:10:33.334880 (+ 682us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 12:10:33.334893 (+ 13us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 12:10:33.334899 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0809 12:10:33.335151 (+ 252us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 12:10:33.335581 (+ 430us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 12:10:33.335585 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 12:10:33.335587 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0809 12:10:33.336110 (+ 523us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 12:10:33.336451 (+ 341us) server_negotiation.cc:1035] Waiting for connection context
0809 12:10:33.336935 (+ 484us) server_negotiation.cc:299] Negotiation successful
0809 12:10:33.337388 (+ 453us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":613,"thread_start_us":405,"threads_started":1}
I20260809 12:10:33.343405 7024 raft_consensus.cc:348] T 00000000000000000000000000000000 P 69e8333fc1194475a902f1ea5f1e347b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "69e8333fc1194475a902f1ea5f1e347b" member_type: VOTER last_known_addr { host: "127.0.222.62" port: 41493 } }
I20260809 12:10:33.343823 7024 raft_consensus.cc:374] T 00000000000000000000000000000000 P 69e8333fc1194475a902f1ea5f1e347b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260809 12:10:33.343931 7024 raft_consensus.cc:729] T 00000000000000000000000000000000 P 69e8333fc1194475a902f1ea5f1e347b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 69e8333fc1194475a902f1ea5f1e347b, State: Initialized, Role: FOLLOWER
I20260809 12:10:33.344487 7024 consensus_queue.cc:260] T 00000000000000000000000000000000 P 69e8333fc1194475a902f1ea5f1e347b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "69e8333fc1194475a902f1ea5f1e347b" member_type: VOTER last_known_addr { host: "127.0.222.62" port: 41493 } }
I20260809 12:10:33.344774 7024 raft_consensus.cc:388] T 00000000000000000000000000000000 P 69e8333fc1194475a902f1ea5f1e347b [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260809 12:10:33.344923 7024 raft_consensus.cc:482] T 00000000000000000000000000000000 P 69e8333fc1194475a902f1ea5f1e347b [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260809 12:10:33.345147 7024 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 69e8333fc1194475a902f1ea5f1e347b [term 0 FOLLOWER]: Advancing to term 1
I20260809 12:10:33.347600 7024 raft_consensus.cc:504] T 00000000000000000000000000000000 P 69e8333fc1194475a902f1ea5f1e347b [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "69e8333fc1194475a902f1ea5f1e347b" member_type: VOTER last_known_addr { host: "127.0.222.62" port: 41493 } }
I20260809 12:10:33.348083 7024 leader_election.cc:302] T 00000000000000000000000000000000 P 69e8333fc1194475a902f1ea5f1e347b [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 69e8333fc1194475a902f1ea5f1e347b; no voters:
I20260809 12:10:33.348659 7024 leader_election.cc:288] T 00000000000000000000000000000000 P 69e8333fc1194475a902f1ea5f1e347b [CANDIDATE]: Term 1 election: Requested vote from peers
I20260809 12:10:33.348793 7029 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 69e8333fc1194475a902f1ea5f1e347b [term 1 FOLLOWER]: Leader election won for term 1
I20260809 12:10:33.350395 7029 raft_consensus.cc:686] T 00000000000000000000000000000000 P 69e8333fc1194475a902f1ea5f1e347b [term 1 LEADER]: Becoming Leader. State: Replica: 69e8333fc1194475a902f1ea5f1e347b, State: Running, Role: LEADER
I20260809 12:10:33.350807 7024 sys_catalog.cc:564] T 00000000000000000000000000000000 P 69e8333fc1194475a902f1ea5f1e347b [sys.catalog]: configured and running, proceeding with master startup.
I20260809 12:10:33.350890 7029 consensus_queue.cc:237] T 00000000000000000000000000000000 P 69e8333fc1194475a902f1ea5f1e347b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "69e8333fc1194475a902f1ea5f1e347b" member_type: VOTER last_known_addr { host: "127.0.222.62" port: 41493 } }
I20260809 12:10:33.353749 7030 sys_catalog.cc:455] T 00000000000000000000000000000000 P 69e8333fc1194475a902f1ea5f1e347b [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "69e8333fc1194475a902f1ea5f1e347b" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "69e8333fc1194475a902f1ea5f1e347b" member_type: VOTER last_known_addr { host: "127.0.222.62" port: 41493 } } }
I20260809 12:10:33.354202 7030 sys_catalog.cc:458] T 00000000000000000000000000000000 P 69e8333fc1194475a902f1ea5f1e347b [sys.catalog]: This master's current role is: LEADER
I20260809 12:10:33.353734 7031 sys_catalog.cc:455] T 00000000000000000000000000000000 P 69e8333fc1194475a902f1ea5f1e347b [sys.catalog]: SysCatalogTable state changed. Reason: New leader 69e8333fc1194475a902f1ea5f1e347b. Latest consensus state: current_term: 1 leader_uuid: "69e8333fc1194475a902f1ea5f1e347b" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "69e8333fc1194475a902f1ea5f1e347b" member_type: VOTER last_known_addr { host: "127.0.222.62" port: 41493 } } }
I20260809 12:10:33.354559 7031 sys_catalog.cc:458] T 00000000000000000000000000000000 P 69e8333fc1194475a902f1ea5f1e347b [sys.catalog]: This master's current role is: LEADER
I20260809 12:10:33.355613 7038 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260809 12:10:33.361707 7038 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260809 12:10:33.371053 7038 catalog_manager.cc:1347] Generated new cluster ID: 15bd334d7d1a45e6bad4fbc740deefb2
I20260809 12:10:33.371178 7038 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260809 12:10:33.387583 7038 catalog_manager.cc:1370] Generated new certificate authority record
I20260809 12:10:33.389302 7038 catalog_manager.cc:1504] Loading token signing keys...
I20260809 12:10:33.403045 7038 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 69e8333fc1194475a902f1ea5f1e347b: Generated new TSK 0
I20260809 12:10:33.403913 7038 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.0.222.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.222.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.222.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskgH5Pjm/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786277382073938-888-0/krb5kdc/kudu_127.0.222.1.keytab.
Entry for principal kudu/127.0.222.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskgH5Pjm/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786277382073938-888-0/krb5kdc/kudu_127.0.222.1.keytab.
I20260809 12:10:33.482847 888 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskgH5Pjm/build/asan/bin/kudu
/tmp/dist-test-taskgH5Pjm/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskgH5Pjm/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786277382073938-888-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskgH5Pjm/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786277382073938-888-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskgH5Pjm/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786277382073938-888-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskgH5Pjm/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786277382073938-888-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.222.1:0
--local_ip_for_outbound_sockets=127.0.222.1
--webserver_interface=127.0.222.1
--webserver_port=0
--tserver_master_addrs=127.0.222.62:41493
--keytab_file=/tmp/dist-test-taskgH5Pjm/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786277382073938-888-0/krb5kdc/kudu_127.0.222.1.keytab
--principal=kudu/127.0.222.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.222.20:41337
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskgH5Pjm/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786277382073938-888-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskgH5Pjm/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786277382073938-888-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskgH5Pjm/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786277382073938-888-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260809 12:10:33.834164 7050 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260809 12:10:33.834462 7050 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260809 12:10:33.834560 7050 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260809 12:10:33.842015 7050 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260809 12:10:33.842137 7050 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260809 12:10:33.842270 7050 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.222.1
I20260809 12:10:33.852442 7050 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.222.20:41337
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskgH5Pjm/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786277382073938-888-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskgH5Pjm/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786277382073938-888-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskgH5Pjm/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786277382073938-888-0/krb5kdc/kudu_127.0.222.1.keytab
--principal=kudu/127.0.222.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.222.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskgH5Pjm/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786277382073938-888-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.222.1
--webserver_port=0
--tserver_master_addrs=127.0.222.62:41493
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.222.1
--log_dir=/tmp/dist-test-taskgH5Pjm/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786277382073938-888-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 09 Aug 2026 11:06:41 UTC on ccycloud.quasar-bwoogy.root.comops.site
build id 4575991
ASAN enabled
I20260809 12:10:33.853879 7050 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260809 12:10:33.855221 7050 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260809 12:10:33.868636 7057 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 12:10:33.868660 7059 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 12:10:33.869386 7056 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260809 12:10:33.869506 7050 server_base.cc:1034] running on GCE node
I20260809 12:10:33.870214 7050 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260809 12:10:33.871289 7050 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260809 12:10:33.872503 7050 hybrid_clock.cc:648] HybridClock initialized: now 1786277433872459 us; error 38 us; skew 500 ppm
I20260809 12:10:33.872784 7050 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 09 12:10:33 dist-test-slave-2w3w krb5kdc[6939](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786277433, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.222.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260809 12:10:33.876013 7050 init.cc:377] Logged in from keytab as kudu/127.0.222.1@KRBTEST.COM (short username kudu)
I20260809 12:10:33.877579 7050 webserver.cc:460] Webserver started at http://127.0.222.1:37339/ using document root <none> and password file <none>
I20260809 12:10:33.878279 7050 fs_manager.cc:359] Metadata directory not provided
I20260809 12:10:33.878434 7050 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260809 12:10:33.878739 7050 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260809 12:10:33.881924 7050 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskgH5Pjm/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786277382073938-888-0/minicluster-data/ts-0/data/instance:
uuid: "a9389f46f2444792b26e324dbfed3984"
format_stamp: "Formatted at 2026-08-09 12:10:33 on dist-test-slave-2w3w"
server_key: "179fc012edbb5779580acf62ab1b12de"
server_key_iv: "0f07cfbeff9c46406578839d77170c5c"
server_key_version: "encryptionkey@0"
I20260809 12:10:33.882789 7050 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskgH5Pjm/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786277382073938-888-0/minicluster-data/ts-0/wal/instance:
uuid: "a9389f46f2444792b26e324dbfed3984"
format_stamp: "Formatted at 2026-08-09 12:10:33 on dist-test-slave-2w3w"
server_key: "179fc012edbb5779580acf62ab1b12de"
server_key_iv: "0f07cfbeff9c46406578839d77170c5c"
server_key_version: "encryptionkey@0"
I20260809 12:10:33.888284 7050 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.003s sys 0.001s
I20260809 12:10:33.891795 7066 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260809 12:10:33.893075 7050 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.002s
I20260809 12:10:33.893260 7050 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskgH5Pjm/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786277382073938-888-0/minicluster-data/ts-0/data,/tmp/dist-test-taskgH5Pjm/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786277382073938-888-0/minicluster-data/ts-0/wal
uuid: "a9389f46f2444792b26e324dbfed3984"
format_stamp: "Formatted at 2026-08-09 12:10:33 on dist-test-slave-2w3w"
server_key: "179fc012edbb5779580acf62ab1b12de"
server_key_iv: "0f07cfbeff9c46406578839d77170c5c"
server_key_version: "encryptionkey@0"
I20260809 12:10:33.893486 7050 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskgH5Pjm/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786277382073938-888-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskgH5Pjm/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786277382073938-888-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskgH5Pjm/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786277382073938-888-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260809 12:10:33.916160 7050 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260809 12:10:33.920569 7050 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260809 12:10:33.920908 7050 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260809 12:10:33.921952 7050 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260809 12:10:33.923429 7050 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260809 12:10:33.923549 7050 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260809 12:10:33.923698 7050 ts_tablet_manager.cc:610] Registered 0 tablets
I20260809 12:10:33.923771 7050 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260809 12:10:33.958460 7050 rpc_server.cc:307] RPC server started. Bound to: 127.0.222.1:34243
I20260809 12:10:33.958549 7184 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.222.1:34243 every 8 connection(s)
I20260809 12:10:33.959831 7050 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskgH5Pjm/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786277382073938-888-0/minicluster-data/ts-0/data/info.pb
I20260809 12:10:33.962082 888 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskgH5Pjm/build/asan/bin/kudu as pid 7050
I20260809 12:10:33.962291 888 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskgH5Pjm/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786277382073938-888-0/minicluster-data/ts-0/wal/instance
I20260809 12:10:33.963214 888 external_mini_cluster.cc:1428] Setting key 3db5ea38c7917d537220e548813138f4
Aug 09 12:10:33 dist-test-slave-2w3w krb5kdc[6939](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786277433, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.222.1@KRBTEST.COM for kudu/127.0.222.62@KRBTEST.COM
I20260809 12:10:33.985014 7187 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 12:10:33.962603 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.222.1:54797 (local address 127.0.222.62:41493)
0809 12:10:33.963167 (+ 564us) server_negotiation.cc:206] Beginning negotiation
0809 12:10:33.963174 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0809 12:10:33.964309 (+ 1135us) server_negotiation.cc:407] Connection header received
0809 12:10:33.966801 (+ 2492us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 12:10:33.966806 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 12:10:33.966908 (+ 102us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 12:10:33.967096 (+ 188us) server_negotiation.cc:226] Negotiated authn=SASL
0809 12:10:33.969350 (+ 2254us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 12:10:33.970174 (+ 824us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 12:10:33.972797 (+ 2623us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 12:10:33.973243 (+ 446us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 12:10:33.973351 (+ 108us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 12:10:33.977850 (+ 4499us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 12:10:33.977860 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 12:10:33.977868 (+ 8us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0809 12:10:33.977915 (+ 47us) server_negotiation.cc:922] calling sasl_server_start()
0809 12:10:33.981550 (+ 3635us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 12:10:33.982327 (+ 777us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 12:10:33.982336 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 12:10:33.982342 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0809 12:10:33.982507 (+ 165us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 12:10:33.982930 (+ 423us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 12:10:33.982938 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 12:10:33.982943 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0809 12:10:33.983372 (+ 429us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 12:10:33.983551 (+ 179us) server_negotiation.cc:1035] Waiting for connection context
0809 12:10:33.984261 (+ 710us) server_negotiation.cc:299] Negotiation successful
0809 12:10:33.984532 (+ 271us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":423,"thread_start_us":268,"threads_started":1}
I20260809 12:10:33.985652 7188 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 12:10:33.963159 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.222.62:41493 (local address 127.0.222.1:54797)
0809 12:10:33.964094 (+ 935us) negotiation.cc:107] Waiting for socket to connect
0809 12:10:33.964135 (+ 41us) client_negotiation.cc:174] Beginning negotiation
0809 12:10:33.966460 (+ 2325us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 12:10:33.967362 (+ 902us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 12:10:33.967377 (+ 15us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 12:10:33.968238 (+ 861us) client_negotiation.cc:189] Negotiated authn=SASL
0809 12:10:33.969109 (+ 871us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 12:10:33.969145 (+ 36us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 12:10:33.970345 (+ 1200us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 12:10:33.970353 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 12:10:33.972627 (+ 2274us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 12:10:33.972647 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 12:10:33.973490 (+ 843us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 12:10:33.973500 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 12:10:33.973840 (+ 340us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 12:10:33.974628 (+ 788us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0809 12:10:33.974663 (+ 35us) client_negotiation.cc:656] calling sasl_client_start()
0809 12:10:33.977587 (+ 2924us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0809 12:10:33.981707 (+ 4120us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0809 12:10:33.981715 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0809 12:10:33.981728 (+ 13us) client_negotiation.cc:760] calling sasl_client_step()
0809 12:10:33.982172 (+ 444us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0809 12:10:33.982618 (+ 446us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0809 12:10:33.982621 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0809 12:10:33.982624 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0809 12:10:33.982808 (+ 184us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0809 12:10:33.983502 (+ 694us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0809 12:10:33.983507 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0809 12:10:33.983834 (+ 327us) client_negotiation.cc:769] Sending connection context
0809 12:10:33.984079 (+ 245us) client_negotiation.cc:240] Negotiation successful
0809 12:10:33.984480 (+ 401us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":653,"thread_start_us":385,"threads_started":1}
I20260809 12:10:33.987661 7185 heartbeater.cc:347] Connected to a master server at 127.0.222.62:41493
I20260809 12:10:33.988008 7185 heartbeater.cc:464] Registering TS with master...
I20260809 12:10:33.988776 7185 heartbeater.cc:511] Master 127.0.222.62:41493 requested a full tablet report, sending...
I20260809 12:10:33.991056 6988 ts_manager.cc:194] Registered new tserver with Master: a9389f46f2444792b26e324dbfed3984 (127.0.222.1:34243)
WARNING: no policy specified for kudu/127.0.222.2@KRBTEST.COM; defaulting to no policy
I20260809 12:10:33.992664 6988 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.222.1@KRBTEST.COM'} at 127.0.222.1:54797
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.222.2@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.222.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskgH5Pjm/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786277382073938-888-0/krb5kdc/kudu_127.0.222.2.keytab.
Entry for principal kudu/127.0.222.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskgH5Pjm/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786277382073938-888-0/krb5kdc/kudu_127.0.222.2.keytab.
I20260809 12:10:34.023723 888 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskgH5Pjm/build/asan/bin/kudu
/tmp/dist-test-taskgH5Pjm/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskgH5Pjm/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786277382073938-888-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskgH5Pjm/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786277382073938-888-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskgH5Pjm/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786277382073938-888-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskgH5Pjm/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786277382073938-888-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.222.2:0
--local_ip_for_outbound_sockets=127.0.222.2
--webserver_interface=127.0.222.2
--webserver_port=0
--tserver_master_addrs=127.0.222.62:41493
--keytab_file=/tmp/dist-test-taskgH5Pjm/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786277382073938-888-0/krb5kdc/kudu_127.0.222.2.keytab
--principal=kudu/127.0.222.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.222.20:41337
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskgH5Pjm/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786277382073938-888-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskgH5Pjm/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786277382073938-888-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskgH5Pjm/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786277382073938-888-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260809 12:10:34.362553 7191 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260809 12:10:34.362843 7191 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260809 12:10:34.363026 7191 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260809 12:10:34.370391 7191 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260809 12:10:34.370515 7191 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260809 12:10:34.370728 7191 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.222.2
I20260809 12:10:34.380636 7191 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.222.20:41337
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskgH5Pjm/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786277382073938-888-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskgH5Pjm/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786277382073938-888-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskgH5Pjm/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786277382073938-888-0/krb5kdc/kudu_127.0.222.2.keytab
--principal=kudu/127.0.222.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.222.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskgH5Pjm/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786277382073938-888-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.222.2
--webserver_port=0
--tserver_master_addrs=127.0.222.62:41493
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.222.2
--log_dir=/tmp/dist-test-taskgH5Pjm/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786277382073938-888-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 09 Aug 2026 11:06:41 UTC on ccycloud.quasar-bwoogy.root.comops.site
build id 4575991
ASAN enabled
I20260809 12:10:34.381953 7191 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260809 12:10:34.383312 7191 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260809 12:10:34.396667 7197 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 12:10:34.396773 7200 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 12:10:34.396756 7198 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260809 12:10:34.397778 7191 server_base.cc:1034] running on GCE node
I20260809 12:10:34.398506 7191 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260809 12:10:34.399605 7191 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260809 12:10:34.400876 7191 hybrid_clock.cc:648] HybridClock initialized: now 1786277434400813 us; error 59 us; skew 500 ppm
I20260809 12:10:34.401199 7191 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 09 12:10:34 dist-test-slave-2w3w krb5kdc[6939](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786277434, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.222.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260809 12:10:34.404913 7191 init.cc:377] Logged in from keytab as kudu/127.0.222.2@KRBTEST.COM (short username kudu)
I20260809 12:10:34.406410 7191 webserver.cc:460] Webserver started at http://127.0.222.2:39927/ using document root <none> and password file <none>
I20260809 12:10:34.407446 7191 fs_manager.cc:359] Metadata directory not provided
I20260809 12:10:34.407609 7191 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260809 12:10:34.407974 7191 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260809 12:10:34.411309 7191 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskgH5Pjm/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786277382073938-888-0/minicluster-data/ts-1/data/instance:
uuid: "da10e63966aa486daa135d4e60dff1cd"
format_stamp: "Formatted at 2026-08-09 12:10:34 on dist-test-slave-2w3w"
server_key: "f2d0b896c533136335f4e328721b8ecf"
server_key_iv: "b4f63db539b4729865d1f8469902a73d"
server_key_version: "encryptionkey@0"
I20260809 12:10:34.412187 7191 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskgH5Pjm/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786277382073938-888-0/minicluster-data/ts-1/wal/instance:
uuid: "da10e63966aa486daa135d4e60dff1cd"
format_stamp: "Formatted at 2026-08-09 12:10:34 on dist-test-slave-2w3w"
server_key: "f2d0b896c533136335f4e328721b8ecf"
server_key_iv: "b4f63db539b4729865d1f8469902a73d"
server_key_version: "encryptionkey@0"
I20260809 12:10:34.417732 7191 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.000s
I20260809 12:10:34.421746 7207 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260809 12:10:34.423640 7191 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.001s sys 0.001s
I20260809 12:10:34.423784 7191 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskgH5Pjm/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786277382073938-888-0/minicluster-data/ts-1/data,/tmp/dist-test-taskgH5Pjm/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786277382073938-888-0/minicluster-data/ts-1/wal
uuid: "da10e63966aa486daa135d4e60dff1cd"
format_stamp: "Formatted at 2026-08-09 12:10:34 on dist-test-slave-2w3w"
server_key: "f2d0b896c533136335f4e328721b8ecf"
server_key_iv: "b4f63db539b4729865d1f8469902a73d"
server_key_version: "encryptionkey@0"
I20260809 12:10:34.423981 7191 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskgH5Pjm/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786277382073938-888-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskgH5Pjm/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786277382073938-888-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskgH5Pjm/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786277382073938-888-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260809 12:10:34.475884 7191 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260809 12:10:34.480032 7191 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260809 12:10:34.480280 7191 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260809 12:10:34.481276 7191 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260809 12:10:34.482647 7191 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260809 12:10:34.482724 7191 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260809 12:10:34.482822 7191 ts_tablet_manager.cc:610] Registered 0 tablets
I20260809 12:10:34.482868 7191 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260809 12:10:34.516613 7191 rpc_server.cc:307] RPC server started. Bound to: 127.0.222.2:40161
I20260809 12:10:34.516664 7328 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.222.2:40161 every 8 connection(s)
I20260809 12:10:34.517941 7191 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskgH5Pjm/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786277382073938-888-0/minicluster-data/ts-1/data/info.pb
I20260809 12:10:34.519428 888 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskgH5Pjm/build/asan/bin/kudu as pid 7191
I20260809 12:10:34.519615 888 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskgH5Pjm/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786277382073938-888-0/minicluster-data/ts-1/wal/instance
I20260809 12:10:34.520136 888 external_mini_cluster.cc:1428] Setting key d8fa92bcef1939491fdec9025831a4e5
Aug 09 12:10:34 dist-test-slave-2w3w krb5kdc[6939](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786277434, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.222.2@KRBTEST.COM for kudu/127.0.222.62@KRBTEST.COM
I20260809 12:10:34.540519 7331 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 12:10:34.520654 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.222.2:55587 (local address 127.0.222.62:41493)
0809 12:10:34.521147 (+ 493us) server_negotiation.cc:206] Beginning negotiation
0809 12:10:34.521153 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0809 12:10:34.522490 (+ 1337us) server_negotiation.cc:407] Connection header received
0809 12:10:34.525461 (+ 2971us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 12:10:34.525465 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 12:10:34.525540 (+ 75us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 12:10:34.525637 (+ 97us) server_negotiation.cc:226] Negotiated authn=SASL
0809 12:10:34.527845 (+ 2208us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 12:10:34.528588 (+ 743us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 12:10:34.530232 (+ 1644us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 12:10:34.530642 (+ 410us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 12:10:34.530790 (+ 148us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 12:10:34.534583 (+ 3793us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 12:10:34.534588 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 12:10:34.534591 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0809 12:10:34.534623 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0809 12:10:34.537324 (+ 2701us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 12:10:34.538077 (+ 753us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 12:10:34.538081 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 12:10:34.538084 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0809 12:10:34.538211 (+ 127us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 12:10:34.538677 (+ 466us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 12:10:34.538681 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 12:10:34.538683 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0809 12:10:34.538949 (+ 266us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 12:10:34.539168 (+ 219us) server_negotiation.cc:1035] Waiting for connection context
0809 12:10:34.539872 (+ 704us) server_negotiation.cc:299] Negotiation successful
0809 12:10:34.540126 (+ 254us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":352,"thread_start_us":225,"threads_started":1}
I20260809 12:10:34.541507 7333 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 12:10:34.521197 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.222.62:41493 (local address 127.0.222.2:55587)
0809 12:10:34.522076 (+ 879us) negotiation.cc:107] Waiting for socket to connect
0809 12:10:34.522125 (+ 49us) client_negotiation.cc:174] Beginning negotiation
0809 12:10:34.525142 (+ 3017us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 12:10:34.525933 (+ 791us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 12:10:34.525947 (+ 14us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 12:10:34.526784 (+ 837us) client_negotiation.cc:189] Negotiated authn=SASL
0809 12:10:34.527665 (+ 881us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 12:10:34.527693 (+ 28us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 12:10:34.528772 (+ 1079us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 12:10:34.528779 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 12:10:34.530046 (+ 1267us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 12:10:34.530069 (+ 23us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 12:10:34.530885 (+ 816us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 12:10:34.530893 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 12:10:34.531177 (+ 284us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 12:10:34.531871 (+ 694us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0809 12:10:34.531894 (+ 23us) client_negotiation.cc:656] calling sasl_client_start()
0809 12:10:34.534337 (+ 2443us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0809 12:10:34.537476 (+ 3139us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0809 12:10:34.537493 (+ 17us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0809 12:10:34.537510 (+ 17us) client_negotiation.cc:760] calling sasl_client_step()
0809 12:10:34.537931 (+ 421us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0809 12:10:34.538328 (+ 397us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0809 12:10:34.538333 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0809 12:10:34.538336 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0809 12:10:34.538543 (+ 207us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0809 12:10:34.539136 (+ 593us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0809 12:10:34.539144 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0809 12:10:34.539498 (+ 354us) client_negotiation.cc:769] Sending connection context
0809 12:10:34.539768 (+ 270us) client_negotiation.cc:240] Negotiation successful
0809 12:10:34.540300 (+ 532us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":566,"thread_start_us":367,"threads_started":1}
I20260809 12:10:34.543236 7329 heartbeater.cc:347] Connected to a master server at 127.0.222.62:41493
I20260809 12:10:34.543565 7329 heartbeater.cc:464] Registering TS with master...
I20260809 12:10:34.544307 7329 heartbeater.cc:511] Master 127.0.222.62:41493 requested a full tablet report, sending...
I20260809 12:10:34.545881 6988 ts_manager.cc:194] Registered new tserver with Master: da10e63966aa486daa135d4e60dff1cd (127.0.222.2:40161)
WARNING: no policy specified for kudu/127.0.222.3@KRBTEST.COM; defaulting to no policy
I20260809 12:10:34.546909 6988 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.222.2@KRBTEST.COM'} at 127.0.222.2:55587
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.222.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.222.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskgH5Pjm/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786277382073938-888-0/krb5kdc/kudu_127.0.222.3.keytab.
Entry for principal kudu/127.0.222.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskgH5Pjm/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786277382073938-888-0/krb5kdc/kudu_127.0.222.3.keytab.
I20260809 12:10:34.577692 888 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskgH5Pjm/build/asan/bin/kudu
/tmp/dist-test-taskgH5Pjm/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskgH5Pjm/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786277382073938-888-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskgH5Pjm/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786277382073938-888-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskgH5Pjm/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786277382073938-888-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskgH5Pjm/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786277382073938-888-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.222.3:0
--local_ip_for_outbound_sockets=127.0.222.3
--webserver_interface=127.0.222.3
--webserver_port=0
--tserver_master_addrs=127.0.222.62:41493
--keytab_file=/tmp/dist-test-taskgH5Pjm/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786277382073938-888-0/krb5kdc/kudu_127.0.222.3.keytab
--principal=kudu/127.0.222.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.222.20:41337
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskgH5Pjm/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786277382073938-888-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskgH5Pjm/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786277382073938-888-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskgH5Pjm/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786277382073938-888-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260809 12:10:34.908515 7336 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260809 12:10:34.908836 7336 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260809 12:10:34.908979 7336 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260809 12:10:34.916486 7336 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260809 12:10:34.916648 7336 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260809 12:10:34.916828 7336 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.222.3
I20260809 12:10:34.926579 7336 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.222.20:41337
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskgH5Pjm/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786277382073938-888-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskgH5Pjm/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786277382073938-888-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskgH5Pjm/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786277382073938-888-0/krb5kdc/kudu_127.0.222.3.keytab
--principal=kudu/127.0.222.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.222.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskgH5Pjm/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786277382073938-888-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.222.3
--webserver_port=0
--tserver_master_addrs=127.0.222.62:41493
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.222.3
--log_dir=/tmp/dist-test-taskgH5Pjm/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786277382073938-888-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 09 Aug 2026 11:06:41 UTC on ccycloud.quasar-bwoogy.root.comops.site
build id 4575991
ASAN enabled
I20260809 12:10:34.928050 7336 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260809 12:10:34.929466 7336 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260809 12:10:34.943300 7341 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 12:10:34.943255 7344 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 12:10:34.943324 7342 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260809 12:10:34.944113 7336 server_base.cc:1034] running on GCE node
I20260809 12:10:34.944715 7336 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260809 12:10:34.945609 7336 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260809 12:10:34.946810 7336 hybrid_clock.cc:648] HybridClock initialized: now 1786277434946769 us; error 27 us; skew 500 ppm
I20260809 12:10:34.947161 7336 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 09 12:10:34 dist-test-slave-2w3w krb5kdc[6939](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786277434, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.222.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260809 12:10:34.950214 7336 init.cc:377] Logged in from keytab as kudu/127.0.222.3@KRBTEST.COM (short username kudu)
I20260809 12:10:34.951848 7336 webserver.cc:460] Webserver started at http://127.0.222.3:39539/ using document root <none> and password file <none>
I20260809 12:10:34.952593 7336 fs_manager.cc:359] Metadata directory not provided
I20260809 12:10:34.952754 7336 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260809 12:10:34.953069 7336 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260809 12:10:34.956019 7336 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskgH5Pjm/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786277382073938-888-0/minicluster-data/ts-2/data/instance:
uuid: "6e5d8f484ac34dc9ad84fe196ab019cb"
format_stamp: "Formatted at 2026-08-09 12:10:34 on dist-test-slave-2w3w"
server_key: "33f137a8d6ce83aadd2c5f5ba85c0a7e"
server_key_iv: "1afd0f95b537224fe236c06e59d38f96"
server_key_version: "encryptionkey@0"
I20260809 12:10:34.956847 7336 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskgH5Pjm/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786277382073938-888-0/minicluster-data/ts-2/wal/instance:
uuid: "6e5d8f484ac34dc9ad84fe196ab019cb"
format_stamp: "Formatted at 2026-08-09 12:10:34 on dist-test-slave-2w3w"
server_key: "33f137a8d6ce83aadd2c5f5ba85c0a7e"
server_key_iv: "1afd0f95b537224fe236c06e59d38f96"
server_key_version: "encryptionkey@0"
I20260809 12:10:34.962360 7336 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.004s sys 0.000s
I20260809 12:10:34.966020 7352 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260809 12:10:34.967372 7336 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260809 12:10:34.967558 7336 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskgH5Pjm/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786277382073938-888-0/minicluster-data/ts-2/data,/tmp/dist-test-taskgH5Pjm/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786277382073938-888-0/minicluster-data/ts-2/wal
uuid: "6e5d8f484ac34dc9ad84fe196ab019cb"
format_stamp: "Formatted at 2026-08-09 12:10:34 on dist-test-slave-2w3w"
server_key: "33f137a8d6ce83aadd2c5f5ba85c0a7e"
server_key_iv: "1afd0f95b537224fe236c06e59d38f96"
server_key_version: "encryptionkey@0"
I20260809 12:10:34.967767 7336 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskgH5Pjm/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786277382073938-888-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskgH5Pjm/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786277382073938-888-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskgH5Pjm/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786277382073938-888-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260809 12:10:34.996281 7185 heartbeater.cc:503] Master 127.0.222.62:41493 was elected leader, sending a full tablet report...
I20260809 12:10:34.996281 7336 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260809 12:10:35.001217 7336 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260809 12:10:35.001544 7336 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260809 12:10:35.002523 7336 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260809 12:10:35.003948 7336 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260809 12:10:35.004052 7336 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260809 12:10:35.004204 7336 ts_tablet_manager.cc:610] Registered 0 tablets
I20260809 12:10:35.004285 7336 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260809 12:10:35.037560 7336 rpc_server.cc:307] RPC server started. Bound to: 127.0.222.3:33955
I20260809 12:10:35.037685 7473 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.222.3:33955 every 8 connection(s)
I20260809 12:10:35.039009 7336 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskgH5Pjm/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786277382073938-888-0/minicluster-data/ts-2/data/info.pb
I20260809 12:10:35.043430 888 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskgH5Pjm/build/asan/bin/kudu as pid 7336
I20260809 12:10:35.043639 888 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskgH5Pjm/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786277382073938-888-0/minicluster-data/ts-2/wal/instance
I20260809 12:10:35.044210 888 external_mini_cluster.cc:1428] Setting key 19db1d82fce4a980f706757182762054
Aug 09 12:10:35 dist-test-slave-2w3w krb5kdc[6939](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786277434, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.222.3@KRBTEST.COM for kudu/127.0.222.62@KRBTEST.COM
I20260809 12:10:35.064258 7476 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 12:10:35.041586 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.222.3:40521 (local address 127.0.222.62:41493)
0809 12:10:35.042104 (+ 518us) server_negotiation.cc:206] Beginning negotiation
0809 12:10:35.042112 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0809 12:10:35.043233 (+ 1121us) server_negotiation.cc:407] Connection header received
0809 12:10:35.045931 (+ 2698us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 12:10:35.045937 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 12:10:35.046053 (+ 116us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 12:10:35.046288 (+ 235us) server_negotiation.cc:226] Negotiated authn=SASL
0809 12:10:35.048538 (+ 2250us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 12:10:35.049839 (+ 1301us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 12:10:35.052292 (+ 2453us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 12:10:35.052911 (+ 619us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 12:10:35.053010 (+ 99us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 12:10:35.057182 (+ 4172us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 12:10:35.057187 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 12:10:35.057190 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0809 12:10:35.057222 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0809 12:10:35.060944 (+ 3722us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 12:10:35.061686 (+ 742us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 12:10:35.061691 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 12:10:35.061695 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0809 12:10:35.061829 (+ 134us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 12:10:35.062288 (+ 459us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 12:10:35.062293 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 12:10:35.062296 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0809 12:10:35.062575 (+ 279us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 12:10:35.062744 (+ 169us) server_negotiation.cc:1035] Waiting for connection context
0809 12:10:35.063654 (+ 910us) server_negotiation.cc:299] Negotiation successful
0809 12:10:35.063873 (+ 219us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":381,"thread_start_us":227,"threads_started":1}
I20260809 12:10:35.065217 7477 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 12:10:35.042189 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.222.62:41493 (local address 127.0.222.3:40521)
0809 12:10:35.042957 (+ 768us) negotiation.cc:107] Waiting for socket to connect
0809 12:10:35.043027 (+ 70us) client_negotiation.cc:174] Beginning negotiation
0809 12:10:35.045606 (+ 2579us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 12:10:35.046662 (+ 1056us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 12:10:35.046673 (+ 11us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 12:10:35.047573 (+ 900us) client_negotiation.cc:189] Negotiated authn=SASL
0809 12:10:35.048347 (+ 774us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 12:10:35.048367 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 12:10:35.049981 (+ 1614us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 12:10:35.049987 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 12:10:35.052109 (+ 2122us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 12:10:35.052132 (+ 23us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 12:10:35.053385 (+ 1253us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 12:10:35.053395 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 12:10:35.053710 (+ 315us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 12:10:35.054609 (+ 899us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0809 12:10:35.054639 (+ 30us) client_negotiation.cc:656] calling sasl_client_start()
0809 12:10:35.056960 (+ 2321us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0809 12:10:35.061094 (+ 4134us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0809 12:10:35.061103 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0809 12:10:35.061120 (+ 17us) client_negotiation.cc:760] calling sasl_client_step()
0809 12:10:35.061551 (+ 431us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0809 12:10:35.061945 (+ 394us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0809 12:10:35.061950 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0809 12:10:35.061954 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0809 12:10:35.062149 (+ 195us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0809 12:10:35.062711 (+ 562us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0809 12:10:35.062718 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0809 12:10:35.063115 (+ 397us) client_negotiation.cc:769] Sending connection context
0809 12:10:35.063525 (+ 410us) client_negotiation.cc:240] Negotiation successful
0809 12:10:35.064079 (+ 554us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":492,"thread_start_us":293,"threads_started":1}
I20260809 12:10:35.066509 7474 heartbeater.cc:347] Connected to a master server at 127.0.222.62:41493
I20260809 12:10:35.066780 7474 heartbeater.cc:464] Registering TS with master...
I20260809 12:10:35.067530 7474 heartbeater.cc:511] Master 127.0.222.62:41493 requested a full tablet report, sending...
I20260809 12:10:35.068871 6988 ts_manager.cc:194] Registered new tserver with Master: 6e5d8f484ac34dc9ad84fe196ab019cb (127.0.222.3:33955)
I20260809 12:10:35.069743 6988 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.222.3@KRBTEST.COM'} at 127.0.222.3:40521
I20260809 12:10:35.073007 888 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal test-admin with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskgH5Pjm/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786277382073938-888-0/krb5kdc/test-admin.keytab.
Entry for principal test-admin with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskgH5Pjm/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786277382073938-888-0/krb5kdc/test-admin.keytab.
Aug 09 12:10:35 dist-test-slave-2w3w krb5kdc[6939](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786277435, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260809 12:10:35.104282 888 init.cc:377] Logged in from keytab as test-admin@KRBTEST.COM (short username test-admin)
W20260809 12:10:35.110014 7482 negotiation.cc:337] Failed RPC negotiation. Trace:
0809 12:10:35.107273 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42588 (local address 127.0.222.1:34243)
0809 12:10:35.107974 (+ 701us) server_negotiation.cc:206] Beginning negotiation
0809 12:10:35.107981 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0809 12:10:35.108007 (+ 26us) server_negotiation.cc:407] Connection header received
0809 12:10:35.108073 (+ 66us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 12:10:35.108079 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 12:10:35.108279 (+ 200us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 12:10:35.108479 (+ 200us) server_negotiation.cc:226] Negotiated authn=SASL
0809 12:10:35.109538 (+ 1059us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:42588: BlockingRecv error: recv got EOF from 127.0.0.1:42588 (error 108)
Metrics: {"server-negotiator.queue_time_us":565,"thread_start_us":359,"threads_started":1}
W20260809 12:10:35.110114 888 init.cc:300] Could not find kerberos principal in credential cache '/tmp/dist-test-taskgH5Pjm/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786277382073938-888-0/krb5kdc/krb5cc' of type FILE
Aug 09 12:10:35 dist-test-slave-2w3w krb5kdc[6939](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786277435, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260809 12:10:35.111816 888 init.cc:377] Logged in from keytab as test-admin@KRBTEST.COM (short username test-admin)
Aug 09 12:10:35 dist-test-slave-2w3w krb5kdc[6939](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786277435, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260809 12:10:35.113162 888 init.cc:295] Successfully reacquired a new kerberos TGT
Aug 09 12:10:35 dist-test-slave-2w3w krb5kdc[6939](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786277435, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.222.1@KRBTEST.COM
I20260809 12:10:35.128772 7482 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 12:10:35.114765 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42596 (local address 127.0.222.1:34243)
0809 12:10:35.115000 (+ 235us) server_negotiation.cc:206] Beginning negotiation
0809 12:10:35.115008 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0809 12:10:35.115302 (+ 294us) server_negotiation.cc:407] Connection header received
0809 12:10:35.115461 (+ 159us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 12:10:35.115464 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 12:10:35.115530 (+ 66us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 12:10:35.115612 (+ 82us) server_negotiation.cc:226] Negotiated authn=SASL
0809 12:10:35.116614 (+ 1002us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 12:10:35.117565 (+ 951us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 12:10:35.118710 (+ 1145us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 12:10:35.119338 (+ 628us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 12:10:35.119445 (+ 107us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 12:10:35.122813 (+ 3368us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 12:10:35.122824 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 12:10:35.122840 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0809 12:10:35.122879 (+ 39us) server_negotiation.cc:922] calling sasl_server_start()
0809 12:10:35.125866 (+ 2987us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 12:10:35.126585 (+ 719us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 12:10:35.126592 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 12:10:35.126594 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0809 12:10:35.126721 (+ 127us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 12:10:35.127101 (+ 380us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 12:10:35.127104 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 12:10:35.127106 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0809 12:10:35.127599 (+ 493us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 12:10:35.127814 (+ 215us) server_negotiation.cc:1035] Waiting for connection context
0809 12:10:35.128158 (+ 344us) server_negotiation.cc:299] Negotiation successful
0809 12:10:35.128441 (+ 283us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":64}
W20260809 12:10:35.134661 7482 negotiation.cc:337] Failed RPC negotiation. Trace:
0809 12:10:35.132739 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42606 (local address 127.0.222.1:34243)
0809 12:10:35.132937 (+ 198us) server_negotiation.cc:206] Beginning negotiation
0809 12:10:35.132946 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0809 12:10:35.133294 (+ 348us) server_negotiation.cc:407] Connection header received
0809 12:10:35.133481 (+ 187us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 12:10:35.133487 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 12:10:35.133578 (+ 91us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 12:10:35.133684 (+ 106us) server_negotiation.cc:226] Negotiated authn=SASL
0809 12:10:35.134409 (+ 725us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:42606: BlockingRecv error: recv got EOF from 127.0.0.1:42606 (error 108)
Metrics: {"server-negotiator.queue_time_us":57}
W20260809 12:10:35.135067 888 init.cc:300] Could not find kerberos principal in credential cache '/tmp/dist-test-taskgH5Pjm/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786277382073938-888-0/krb5kdc/krb5cc' of type FILE
Aug 09 12:10:35 dist-test-slave-2w3w krb5kdc[6939](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786277435, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260809 12:10:35.136699 888 init.cc:377] Logged in from keytab as test-admin@KRBTEST.COM (short username test-admin)
Aug 09 12:10:35 dist-test-slave-2w3w krb5kdc[6939](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786277435, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260809 12:10:35.138062 888 init.cc:295] Successfully reacquired a new kerberos TGT
Aug 09 12:10:35 dist-test-slave-2w3w krb5kdc[6939](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786277435, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.222.1@KRBTEST.COM
I20260809 12:10:35.151829 7482 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 12:10:35.139529 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42620 (local address 127.0.222.1:34243)
0809 12:10:35.139712 (+ 183us) server_negotiation.cc:206] Beginning negotiation
0809 12:10:35.139718 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0809 12:10:35.139870 (+ 152us) server_negotiation.cc:407] Connection header received
0809 12:10:35.139987 (+ 117us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 12:10:35.139990 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 12:10:35.140043 (+ 53us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 12:10:35.140120 (+ 77us) server_negotiation.cc:226] Negotiated authn=SASL
0809 12:10:35.141022 (+ 902us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 12:10:35.141721 (+ 699us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 12:10:35.142906 (+ 1185us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 12:10:35.143396 (+ 490us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 12:10:35.143499 (+ 103us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 12:10:35.147026 (+ 3527us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 12:10:35.147032 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 12:10:35.147035 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0809 12:10:35.147066 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0809 12:10:35.149479 (+ 2413us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 12:10:35.150091 (+ 612us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 12:10:35.150095 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 12:10:35.150097 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0809 12:10:35.150216 (+ 119us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 12:10:35.150550 (+ 334us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 12:10:35.150554 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 12:10:35.150556 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0809 12:10:35.150858 (+ 302us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 12:10:35.151110 (+ 252us) server_negotiation.cc:1035] Waiting for connection context
0809 12:10:35.151382 (+ 272us) server_negotiation.cc:299] Negotiation successful
0809 12:10:35.151577 (+ 195us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":66}
W20260809 12:10:35.156263 7482 negotiation.cc:337] Failed RPC negotiation. Trace:
0809 12:10:35.154434 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42632 (local address 127.0.222.1:34243)
0809 12:10:35.154630 (+ 196us) server_negotiation.cc:206] Beginning negotiation
0809 12:10:35.154636 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0809 12:10:35.154935 (+ 299us) server_negotiation.cc:407] Connection header received
0809 12:10:35.155148 (+ 213us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 12:10:35.155152 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 12:10:35.155219 (+ 67us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 12:10:35.155323 (+ 104us) server_negotiation.cc:226] Negotiated authn=SASL
0809 12:10:35.156014 (+ 691us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:42632: BlockingRecv error: recv got EOF from 127.0.0.1:42632 (error 108)
Metrics: {"server-negotiator.queue_time_us":71}
W20260809 12:10:35.156894 888 init.cc:244] Error while opening credential cache '/tmp/dist-test-taskgH5Pjm/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786277382073938-888-0/krb5kdc/krb5cc' of type FILE: Runtime error: End of credential cache reached (filename: /tmp/dist-test-taskgH5Pjm/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786277382073938-888-0/krb5kdc/krb5cc)
Aug 09 12:10:35 dist-test-slave-2w3w krb5kdc[6939](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786277435, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260809 12:10:35.158540 888 init.cc:377] Logged in from keytab as test-admin@KRBTEST.COM (short username test-admin)
Aug 09 12:10:35 dist-test-slave-2w3w krb5kdc[6939](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786277435, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260809 12:10:35.159946 888 init.cc:295] Successfully reacquired a new kerberos TGT
Aug 09 12:10:35 dist-test-slave-2w3w krb5kdc[6939](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786277435, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.222.1@KRBTEST.COM
I20260809 12:10:35.174363 7482 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 12:10:35.161203 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42648 (local address 127.0.222.1:34243)
0809 12:10:35.161412 (+ 209us) server_negotiation.cc:206] Beginning negotiation
0809 12:10:35.161418 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0809 12:10:35.161593 (+ 175us) server_negotiation.cc:407] Connection header received
0809 12:10:35.161752 (+ 159us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 12:10:35.161764 (+ 12us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 12:10:35.161820 (+ 56us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 12:10:35.161902 (+ 82us) server_negotiation.cc:226] Negotiated authn=SASL
0809 12:10:35.162783 (+ 881us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 12:10:35.163571 (+ 788us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 12:10:35.164778 (+ 1207us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 12:10:35.165186 (+ 408us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 12:10:35.165288 (+ 102us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 12:10:35.168711 (+ 3423us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 12:10:35.168715 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 12:10:35.168718 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0809 12:10:35.168750 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0809 12:10:35.171510 (+ 2760us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 12:10:35.172245 (+ 735us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 12:10:35.172249 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 12:10:35.172252 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0809 12:10:35.172404 (+ 152us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 12:10:35.172906 (+ 502us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 12:10:35.172913 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 12:10:35.172916 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0809 12:10:35.173295 (+ 379us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 12:10:35.173541 (+ 246us) server_negotiation.cc:1035] Waiting for connection context
0809 12:10:35.173751 (+ 210us) server_negotiation.cc:299] Negotiation successful
0809 12:10:35.173967 (+ 216us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":79}
I20260809 12:10:35.176168 888 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskgH5Pjm/build/asan/bin/kudu pid 7050
I20260809 12:10:35.191193 7482 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 12:10:35.177067 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42656 (local address 127.0.222.1:34243)
0809 12:10:35.177223 (+ 156us) server_negotiation.cc:206] Beginning negotiation
0809 12:10:35.177227 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0809 12:10:35.179454 (+ 2227us) server_negotiation.cc:407] Connection header received
0809 12:10:35.179655 (+ 201us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 12:10:35.179667 (+ 12us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 12:10:35.179729 (+ 62us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 12:10:35.179839 (+ 110us) server_negotiation.cc:226] Negotiated authn=SASL
0809 12:10:35.180886 (+ 1047us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 12:10:35.181582 (+ 696us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 12:10:35.182803 (+ 1221us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 12:10:35.183314 (+ 511us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 12:10:35.183460 (+ 146us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 12:10:35.185647 (+ 2187us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 12:10:35.185653 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 12:10:35.185659 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0809 12:10:35.185699 (+ 40us) server_negotiation.cc:922] calling sasl_server_start()
0809 12:10:35.188388 (+ 2689us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 12:10:35.189041 (+ 653us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 12:10:35.189048 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 12:10:35.189052 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0809 12:10:35.189277 (+ 225us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 12:10:35.189667 (+ 390us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 12:10:35.189672 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 12:10:35.189676 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0809 12:10:35.189963 (+ 287us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 12:10:35.190265 (+ 302us) server_negotiation.cc:1035] Waiting for connection context
0809 12:10:35.190605 (+ 340us) server_negotiation.cc:299] Negotiation successful
0809 12:10:35.190821 (+ 216us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":58}
I20260809 12:10:35.191926 7180 generic_service.cc:196] Checking for leaks (request via RPC)
I20260809 12:10:35.284945 888 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskgH5Pjm/build/asan/bin/kudu with pid 7050
I20260809 12:10:35.305716 888 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskgH5Pjm/build/asan/bin/kudu pid 7191
Aug 09 12:10:35 dist-test-slave-2w3w krb5kdc[6939](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786277435, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.222.2@KRBTEST.COM
I20260809 12:10:35.322216 7495 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 12:10:35.306710 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52048 (local address 127.0.222.2:40161)
0809 12:10:35.307303 (+ 593us) server_negotiation.cc:206] Beginning negotiation
0809 12:10:35.307310 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0809 12:10:35.307336 (+ 26us) server_negotiation.cc:407] Connection header received
0809 12:10:35.307440 (+ 104us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 12:10:35.307449 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 12:10:35.307686 (+ 237us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 12:10:35.307863 (+ 177us) server_negotiation.cc:226] Negotiated authn=SASL
0809 12:10:35.308929 (+ 1066us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 12:10:35.309991 (+ 1062us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 12:10:35.311882 (+ 1891us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 12:10:35.312451 (+ 569us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 12:10:35.312603 (+ 152us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 12:10:35.316555 (+ 3952us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 12:10:35.316566 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 12:10:35.316580 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0809 12:10:35.316614 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0809 12:10:35.319379 (+ 2765us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 12:10:35.320057 (+ 678us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 12:10:35.320064 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 12:10:35.320066 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0809 12:10:35.320190 (+ 124us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 12:10:35.320586 (+ 396us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 12:10:35.320591 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 12:10:35.320594 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0809 12:10:35.320938 (+ 344us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 12:10:35.321266 (+ 328us) server_negotiation.cc:1035] Waiting for connection context
0809 12:10:35.321620 (+ 354us) server_negotiation.cc:299] Negotiation successful
0809 12:10:35.321886 (+ 266us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":461,"thread_start_us":341,"threads_started":1}
I20260809 12:10:35.323568 7323 generic_service.cc:196] Checking for leaks (request via RPC)
I20260809 12:10:35.406628 888 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskgH5Pjm/build/asan/bin/kudu with pid 7191
I20260809 12:10:35.425760 888 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskgH5Pjm/build/asan/bin/kudu pid 7336
Aug 09 12:10:35 dist-test-slave-2w3w krb5kdc[6939](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786277435, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.222.3@KRBTEST.COM
I20260809 12:10:35.449376 7497 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 12:10:35.427040 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40500 (local address 127.0.222.3:33955)
0809 12:10:35.427754 (+ 714us) server_negotiation.cc:206] Beginning negotiation
0809 12:10:35.427761 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0809 12:10:35.427791 (+ 30us) server_negotiation.cc:407] Connection header received
0809 12:10:35.427851 (+ 60us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 12:10:35.427857 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 12:10:35.428079 (+ 222us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 12:10:35.428203 (+ 124us) server_negotiation.cc:226] Negotiated authn=SASL
0809 12:10:35.429211 (+ 1008us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 12:10:35.430218 (+ 1007us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 12:10:35.431556 (+ 1338us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 12:10:35.432115 (+ 559us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 12:10:35.432220 (+ 105us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 12:10:35.443934 (+ 11714us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 12:10:35.443948 (+ 14us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 12:10:35.443964 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0809 12:10:35.444011 (+ 47us) server_negotiation.cc:922] calling sasl_server_start()
0809 12:10:35.446912 (+ 2901us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 12:10:35.447641 (+ 729us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 12:10:35.447648 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 12:10:35.447651 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0809 12:10:35.447773 (+ 122us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 12:10:35.448096 (+ 323us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 12:10:35.448099 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 12:10:35.448101 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0809 12:10:35.448443 (+ 342us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 12:10:35.448697 (+ 254us) server_negotiation.cc:1035] Waiting for connection context
0809 12:10:35.448883 (+ 186us) server_negotiation.cc:299] Negotiation successful
0809 12:10:35.449095 (+ 212us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":563,"thread_start_us":392,"threads_started":1}
I20260809 12:10:35.450773 7468 generic_service.cc:196] Checking for leaks (request via RPC)
I20260809 12:10:35.542434 888 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskgH5Pjm/build/asan/bin/kudu with pid 7336
I20260809 12:10:35.565653 888 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskgH5Pjm/build/asan/bin/kudu pid 6954
I20260809 12:10:35.566892 7019 generic_service.cc:196] Checking for leaks (request via RPC)
I20260809 12:10:35.673393 888 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskgH5Pjm/build/asan/bin/kudu with pid 6954
2026-08-09T12:10:35Z chronyd exiting
[ OK ] SecurityITest.TestCorruptKerberosCC (4239 ms)
[ RUN ] SecurityITest.TestNonDefaultPrincipalMultipleMaster
Loading random data
Initializing database '/tmp/dist-test-taskgH5Pjm/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786277382073938-888-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 09 12:10:35 dist-test-slave-2w3w krb5kdc[7502](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 09 12:10:35 dist-test-slave-2w3w krb5kdc[7502](info): set up 2 sockets
Aug 09 12:10:35 dist-test-slave-2w3w krb5kdc[7502](info): commencing operation
krb5kdc: starting...
W20260809 12:10:36.833542 888 mini_kdc.cc:121] Time spent starting KDC: real 1.112s user 0.003s sys 0.030s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskgH5Pjm/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786277382073938-888-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskgH5Pjm/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786277382073938-888-0/krb5kdc/test-user.keytab.
Aug 09 12:10:37 dist-test-slave-2w3w krb5kdc[7502](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786277437, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-09T12:10:37Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-09T12:10:37Z Disabled control of system clock
WARNING: no policy specified for oryx/127.0.222.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.0.222.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.0.222.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskgH5Pjm/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786277382073938-888-0/krb5kdc/oryx_127.0.222.62.keytab.
Entry for principal oryx/127.0.222.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskgH5Pjm/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786277382073938-888-0/krb5kdc/oryx_127.0.222.62.keytab.
I20260809 12:10:37.094309 888 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskgH5Pjm/build/asan/bin/kudu
/tmp/dist-test-taskgH5Pjm/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskgH5Pjm/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786277382073938-888-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskgH5Pjm/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786277382073938-888-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskgH5Pjm/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786277382073938-888-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskgH5Pjm/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786277382073938-888-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.222.62:41983
--webserver_interface=127.0.222.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskgH5Pjm/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786277382073938-888-0/krb5kdc/oryx_127.0.222.62.keytab
--principal=oryx/127.0.222.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.222.20:38069
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.222.62:41983
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskgH5Pjm/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786277382073938-888-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskgH5Pjm/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786277382073938-888-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskgH5Pjm/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786277382073938-888-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260809 12:10:37.430359 7516 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260809 12:10:37.430670 7516 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260809 12:10:37.430843 7516 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260809 12:10:37.438272 7516 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260809 12:10:37.438426 7516 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260809 12:10:37.438517 7516 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260809 12:10:37.438622 7516 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260809 12:10:37.438706 7516 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260809 12:10:37.448827 7516 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.222.20:38069
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskgH5Pjm/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786277382073938-888-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskgH5Pjm/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786277382073938-888-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.222.62:41983
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskgH5Pjm/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786277382073938-888-0/krb5kdc/oryx_127.0.222.62.keytab
--principal=oryx/127.0.222.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.222.62:41983
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskgH5Pjm/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786277382073938-888-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.222.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskgH5Pjm/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786277382073938-888-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 09 Aug 2026 11:06:41 UTC on ccycloud.quasar-bwoogy.root.comops.site
build id 4575991
ASAN enabled
I20260809 12:10:37.450213 7516 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260809 12:10:37.451586 7516 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260809 12:10:37.463491 7521 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 12:10:37.463689 7522 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 12:10:37.463970 7524 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260809 12:10:37.464833 7516 server_base.cc:1034] running on GCE node
I20260809 12:10:37.465624 7516 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260809 12:10:37.467259 7516 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260809 12:10:37.468504 7516 hybrid_clock.cc:648] HybridClock initialized: now 1786277437468434 us; error 62 us; skew 500 ppm
I20260809 12:10:37.468870 7516 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 09 12:10:37 dist-test-slave-2w3w krb5kdc[7502](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786277437, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.222.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260809 12:10:37.472208 7516 init.cc:377] Logged in from keytab as oryx/127.0.222.62@KRBTEST.COM (short username oryx)
I20260809 12:10:37.473852 7516 webserver.cc:460] Webserver started at http://127.0.222.62:34431/ using document root <none> and password file <none>
I20260809 12:10:37.474562 7516 fs_manager.cc:359] Metadata directory not provided
I20260809 12:10:37.474715 7516 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260809 12:10:37.475121 7516 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260809 12:10:37.478147 7516 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskgH5Pjm/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786277382073938-888-0/minicluster-data/master-0/data/instance:
uuid: "9cd5d41083d147f788d83a438b6f7fc2"
format_stamp: "Formatted at 2026-08-09 12:10:37 on dist-test-slave-2w3w"
server_key: "29825da6e1637943c65a63c2730868a2"
server_key_iv: "5386331e1f72079a1cad80049bdfbd8c"
server_key_version: "encryptionkey@0"
I20260809 12:10:37.478924 7516 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskgH5Pjm/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786277382073938-888-0/minicluster-data/master-0/wal/instance:
uuid: "9cd5d41083d147f788d83a438b6f7fc2"
format_stamp: "Formatted at 2026-08-09 12:10:37 on dist-test-slave-2w3w"
server_key: "29825da6e1637943c65a63c2730868a2"
server_key_iv: "5386331e1f72079a1cad80049bdfbd8c"
server_key_version: "encryptionkey@0"
I20260809 12:10:37.484194 7516 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.003s sys 0.001s
I20260809 12:10:37.487699 7531 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260809 12:10:37.489177 7516 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260809 12:10:37.489372 7516 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskgH5Pjm/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786277382073938-888-0/minicluster-data/master-0/data,/tmp/dist-test-taskgH5Pjm/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786277382073938-888-0/minicluster-data/master-0/wal
uuid: "9cd5d41083d147f788d83a438b6f7fc2"
format_stamp: "Formatted at 2026-08-09 12:10:37 on dist-test-slave-2w3w"
server_key: "29825da6e1637943c65a63c2730868a2"
server_key_iv: "5386331e1f72079a1cad80049bdfbd8c"
server_key_version: "encryptionkey@0"
I20260809 12:10:37.489578 7516 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskgH5Pjm/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786277382073938-888-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskgH5Pjm/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786277382073938-888-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskgH5Pjm/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786277382073938-888-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260809 12:10:37.510700 7516 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260809 12:10:37.514997 7516 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260809 12:10:37.515303 7516 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260809 12:10:37.535032 7516 rpc_server.cc:307] RPC server started. Bound to: 127.0.222.62:41983
I20260809 12:10:37.535038 7583 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.222.62:41983 every 8 connection(s)
I20260809 12:10:37.536450 7516 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskgH5Pjm/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786277382073938-888-0/minicluster-data/master-0/data/info.pb
I20260809 12:10:37.539366 888 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskgH5Pjm/build/asan/bin/kudu as pid 7516
I20260809 12:10:37.539592 888 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskgH5Pjm/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786277382073938-888-0/minicluster-data/master-0/wal/instance
I20260809 12:10:37.540575 888 external_mini_cluster.cc:1428] Setting key 03a8778ccb495369ec7049e859224288
I20260809 12:10:37.541352 7585 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
Aug 09 12:10:37 dist-test-slave-2w3w krb5kdc[7502](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786277437, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.0.222.62@KRBTEST.COM
I20260809 12:10:37.556533 7585 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 9cd5d41083d147f788d83a438b6f7fc2: Bootstrap starting.
I20260809 12:10:37.559944 7585 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 9cd5d41083d147f788d83a438b6f7fc2: Neither blocks nor log segments found. Creating new log.
I20260809 12:10:37.561003 7585 log.cc:824] T 00000000000000000000000000000000 P 9cd5d41083d147f788d83a438b6f7fc2: Log is configured to *not* fsync() on all Append() calls
I20260809 12:10:37.563807 7588 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 12:10:37.542787 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44084 (local address 127.0.222.62:41983)
0809 12:10:37.543642 (+ 855us) server_negotiation.cc:206] Beginning negotiation
0809 12:10:37.543653 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0809 12:10:37.543695 (+ 42us) server_negotiation.cc:407] Connection header received
0809 12:10:37.545222 (+ 1527us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 12:10:37.545242 (+ 20us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 12:10:37.545631 (+ 389us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 12:10:37.545969 (+ 338us) server_negotiation.cc:226] Negotiated authn=SASL
0809 12:10:37.547841 (+ 1872us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 12:10:37.549232 (+ 1391us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 12:10:37.551017 (+ 1785us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 12:10:37.551687 (+ 670us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 12:10:37.551820 (+ 133us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 12:10:37.556522 (+ 4702us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 12:10:37.556530 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 12:10:37.556544 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0809 12:10:37.556588 (+ 44us) server_negotiation.cc:922] calling sasl_server_start()
0809 12:10:37.559682 (+ 3094us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 12:10:37.560439 (+ 757us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 12:10:37.560445 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 12:10:37.560451 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0809 12:10:37.560619 (+ 168us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 12:10:37.561028 (+ 409us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 12:10:37.561040 (+ 12us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 12:10:37.561042 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0809 12:10:37.561621 (+ 579us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 12:10:37.561944 (+ 323us) server_negotiation.cc:1035] Waiting for connection context
0809 12:10:37.562388 (+ 444us) server_negotiation.cc:299] Negotiation successful
0809 12:10:37.562808 (+ 420us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":646,"thread_start_us":434,"threads_started":1}
I20260809 12:10:37.564096 7585 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 9cd5d41083d147f788d83a438b6f7fc2: No bootstrap required, opened a new log
I20260809 12:10:37.571370 7585 raft_consensus.cc:348] T 00000000000000000000000000000000 P 9cd5d41083d147f788d83a438b6f7fc2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9cd5d41083d147f788d83a438b6f7fc2" member_type: VOTER last_known_addr { host: "127.0.222.62" port: 41983 } }
I20260809 12:10:37.571719 7585 raft_consensus.cc:374] T 00000000000000000000000000000000 P 9cd5d41083d147f788d83a438b6f7fc2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260809 12:10:37.571842 7585 raft_consensus.cc:729] T 00000000000000000000000000000000 P 9cd5d41083d147f788d83a438b6f7fc2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9cd5d41083d147f788d83a438b6f7fc2, State: Initialized, Role: FOLLOWER
I20260809 12:10:37.572502 7585 consensus_queue.cc:260] T 00000000000000000000000000000000 P 9cd5d41083d147f788d83a438b6f7fc2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9cd5d41083d147f788d83a438b6f7fc2" member_type: VOTER last_known_addr { host: "127.0.222.62" port: 41983 } }
I20260809 12:10:37.572805 7585 raft_consensus.cc:388] T 00000000000000000000000000000000 P 9cd5d41083d147f788d83a438b6f7fc2 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260809 12:10:37.572971 7585 raft_consensus.cc:482] T 00000000000000000000000000000000 P 9cd5d41083d147f788d83a438b6f7fc2 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260809 12:10:37.573153 7585 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 9cd5d41083d147f788d83a438b6f7fc2 [term 0 FOLLOWER]: Advancing to term 1
I20260809 12:10:37.575385 7585 raft_consensus.cc:504] T 00000000000000000000000000000000 P 9cd5d41083d147f788d83a438b6f7fc2 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9cd5d41083d147f788d83a438b6f7fc2" member_type: VOTER last_known_addr { host: "127.0.222.62" port: 41983 } }
I20260809 12:10:37.575868 7585 leader_election.cc:302] T 00000000000000000000000000000000 P 9cd5d41083d147f788d83a438b6f7fc2 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 9cd5d41083d147f788d83a438b6f7fc2; no voters:
I20260809 12:10:37.576457 7585 leader_election.cc:288] T 00000000000000000000000000000000 P 9cd5d41083d147f788d83a438b6f7fc2 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260809 12:10:37.576612 7590 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 9cd5d41083d147f788d83a438b6f7fc2 [term 1 FOLLOWER]: Leader election won for term 1
I20260809 12:10:37.577432 7590 raft_consensus.cc:686] T 00000000000000000000000000000000 P 9cd5d41083d147f788d83a438b6f7fc2 [term 1 LEADER]: Becoming Leader. State: Replica: 9cd5d41083d147f788d83a438b6f7fc2, State: Running, Role: LEADER
I20260809 12:10:37.577994 7590 consensus_queue.cc:237] T 00000000000000000000000000000000 P 9cd5d41083d147f788d83a438b6f7fc2 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9cd5d41083d147f788d83a438b6f7fc2" member_type: VOTER last_known_addr { host: "127.0.222.62" port: 41983 } }
I20260809 12:10:37.578784 7585 sys_catalog.cc:564] T 00000000000000000000000000000000 P 9cd5d41083d147f788d83a438b6f7fc2 [sys.catalog]: configured and running, proceeding with master startup.
I20260809 12:10:37.580946 7591 sys_catalog.cc:455] T 00000000000000000000000000000000 P 9cd5d41083d147f788d83a438b6f7fc2 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "9cd5d41083d147f788d83a438b6f7fc2" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9cd5d41083d147f788d83a438b6f7fc2" member_type: VOTER last_known_addr { host: "127.0.222.62" port: 41983 } } }
I20260809 12:10:37.580915 7592 sys_catalog.cc:455] T 00000000000000000000000000000000 P 9cd5d41083d147f788d83a438b6f7fc2 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 9cd5d41083d147f788d83a438b6f7fc2. Latest consensus state: current_term: 1 leader_uuid: "9cd5d41083d147f788d83a438b6f7fc2" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9cd5d41083d147f788d83a438b6f7fc2" member_type: VOTER last_known_addr { host: "127.0.222.62" port: 41983 } } }
I20260809 12:10:37.581353 7592 sys_catalog.cc:458] T 00000000000000000000000000000000 P 9cd5d41083d147f788d83a438b6f7fc2 [sys.catalog]: This master's current role is: LEADER
I20260809 12:10:37.581357 7591 sys_catalog.cc:458] T 00000000000000000000000000000000 P 9cd5d41083d147f788d83a438b6f7fc2 [sys.catalog]: This master's current role is: LEADER
I20260809 12:10:37.583014 7595 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260809 12:10:37.588721 7595 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260809 12:10:37.597805 7595 catalog_manager.cc:1347] Generated new cluster ID: 282dd341a1ea4ea895575dd472644e9f
I20260809 12:10:37.597929 7595 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260809 12:10:37.608198 7595 catalog_manager.cc:1370] Generated new certificate authority record
I20260809 12:10:37.609992 7595 catalog_manager.cc:1504] Loading token signing keys...
I20260809 12:10:37.625095 7595 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 9cd5d41083d147f788d83a438b6f7fc2: Generated new TSK 0
I20260809 12:10:37.626148 7595 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for oryx/127.0.222.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.0.222.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.0.222.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskgH5Pjm/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786277382073938-888-0/krb5kdc/oryx_127.0.222.1.keytab.
Entry for principal oryx/127.0.222.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskgH5Pjm/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786277382073938-888-0/krb5kdc/oryx_127.0.222.1.keytab.
I20260809 12:10:37.696342 888 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskgH5Pjm/build/asan/bin/kudu
/tmp/dist-test-taskgH5Pjm/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskgH5Pjm/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786277382073938-888-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskgH5Pjm/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786277382073938-888-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskgH5Pjm/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786277382073938-888-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskgH5Pjm/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786277382073938-888-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.222.1:0
--local_ip_for_outbound_sockets=127.0.222.1
--webserver_interface=127.0.222.1
--webserver_port=0
--tserver_master_addrs=127.0.222.62:41983
--keytab_file=/tmp/dist-test-taskgH5Pjm/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786277382073938-888-0/krb5kdc/oryx_127.0.222.1.keytab
--principal=oryx/127.0.222.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.222.20:38069
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskgH5Pjm/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786277382073938-888-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskgH5Pjm/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786277382073938-888-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskgH5Pjm/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786277382073938-888-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260809 12:10:38.044621 7611 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260809 12:10:38.044953 7611 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260809 12:10:38.045117 7611 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260809 12:10:38.052467 7611 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260809 12:10:38.052609 7611 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260809 12:10:38.052815 7611 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.222.1
I20260809 12:10:38.062572 7611 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.222.20:38069
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskgH5Pjm/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786277382073938-888-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskgH5Pjm/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786277382073938-888-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskgH5Pjm/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786277382073938-888-0/krb5kdc/oryx_127.0.222.1.keytab
--principal=oryx/127.0.222.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.222.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskgH5Pjm/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786277382073938-888-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.222.1
--webserver_port=0
--tserver_master_addrs=127.0.222.62:41983
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.222.1
--log_dir=/tmp/dist-test-taskgH5Pjm/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786277382073938-888-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 09 Aug 2026 11:06:41 UTC on ccycloud.quasar-bwoogy.root.comops.site
build id 4575991
ASAN enabled
I20260809 12:10:38.063923 7611 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260809 12:10:38.065295 7611 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260809 12:10:38.079102 7616 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 12:10:38.079111 7617 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 12:10:38.079061 7619 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260809 12:10:38.080276 7611 server_base.cc:1034] running on GCE node
I20260809 12:10:38.080871 7611 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260809 12:10:38.081888 7611 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260809 12:10:38.083148 7611 hybrid_clock.cc:648] HybridClock initialized: now 1786277438083097 us; error 54 us; skew 500 ppm
I20260809 12:10:38.083583 7611 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 09 12:10:38 dist-test-slave-2w3w krb5kdc[7502](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786277438, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.222.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260809 12:10:38.087062 7611 init.cc:377] Logged in from keytab as oryx/127.0.222.1@KRBTEST.COM (short username oryx)
I20260809 12:10:38.088519 7611 webserver.cc:460] Webserver started at http://127.0.222.1:33359/ using document root <none> and password file <none>
I20260809 12:10:38.089190 7611 fs_manager.cc:359] Metadata directory not provided
I20260809 12:10:38.089296 7611 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260809 12:10:38.089560 7611 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260809 12:10:38.092629 7611 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskgH5Pjm/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786277382073938-888-0/minicluster-data/ts-0/data/instance:
uuid: "df0db1a138554d00956d87f5c172ee10"
format_stamp: "Formatted at 2026-08-09 12:10:38 on dist-test-slave-2w3w"
server_key: "b6a9d17eadb03dcfff0fbce8a306b594"
server_key_iv: "c3eda1fa16fab7b7493b77c4d3a1da80"
server_key_version: "encryptionkey@0"
I20260809 12:10:38.093570 7611 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskgH5Pjm/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786277382073938-888-0/minicluster-data/ts-0/wal/instance:
uuid: "df0db1a138554d00956d87f5c172ee10"
format_stamp: "Formatted at 2026-08-09 12:10:38 on dist-test-slave-2w3w"
server_key: "b6a9d17eadb03dcfff0fbce8a306b594"
server_key_iv: "c3eda1fa16fab7b7493b77c4d3a1da80"
server_key_version: "encryptionkey@0"
I20260809 12:10:38.098860 7611 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.003s sys 0.004s
I20260809 12:10:38.102383 7628 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260809 12:10:38.103782 7611 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260809 12:10:38.103955 7611 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskgH5Pjm/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786277382073938-888-0/minicluster-data/ts-0/data,/tmp/dist-test-taskgH5Pjm/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786277382073938-888-0/minicluster-data/ts-0/wal
uuid: "df0db1a138554d00956d87f5c172ee10"
format_stamp: "Formatted at 2026-08-09 12:10:38 on dist-test-slave-2w3w"
server_key: "b6a9d17eadb03dcfff0fbce8a306b594"
server_key_iv: "c3eda1fa16fab7b7493b77c4d3a1da80"
server_key_version: "encryptionkey@0"
I20260809 12:10:38.104161 7611 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskgH5Pjm/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786277382073938-888-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskgH5Pjm/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786277382073938-888-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskgH5Pjm/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786277382073938-888-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260809 12:10:38.132989 7611 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260809 12:10:38.137253 7611 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260809 12:10:38.137560 7611 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260809 12:10:38.138566 7611 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260809 12:10:38.139971 7611 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260809 12:10:38.140050 7611 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260809 12:10:38.140214 7611 ts_tablet_manager.cc:610] Registered 0 tablets
I20260809 12:10:38.140300 7611 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260809 12:10:38.174886 7611 rpc_server.cc:307] RPC server started. Bound to: 127.0.222.1:43733
I20260809 12:10:38.175019 7745 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.222.1:43733 every 8 connection(s)
I20260809 12:10:38.176201 7611 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskgH5Pjm/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786277382073938-888-0/minicluster-data/ts-0/data/info.pb
I20260809 12:10:38.184882 888 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskgH5Pjm/build/asan/bin/kudu as pid 7611
I20260809 12:10:38.185117 888 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskgH5Pjm/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786277382073938-888-0/minicluster-data/ts-0/wal/instance
I20260809 12:10:38.185873 888 external_mini_cluster.cc:1428] Setting key 9c83fb54879a17e5d52596c2892c9fbe
Aug 09 12:10:38 dist-test-slave-2w3w krb5kdc[7502](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786277438, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.222.1@KRBTEST.COM for oryx/127.0.222.62@KRBTEST.COM
I20260809 12:10:38.199538 7748 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 12:10:38.178744 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.222.1:49233 (local address 127.0.222.62:41983)
0809 12:10:38.179332 (+ 588us) server_negotiation.cc:206] Beginning negotiation
0809 12:10:38.179337 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0809 12:10:38.180338 (+ 1001us) server_negotiation.cc:407] Connection header received
0809 12:10:38.182214 (+ 1876us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 12:10:38.182219 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 12:10:38.182321 (+ 102us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 12:10:38.182445 (+ 124us) server_negotiation.cc:226] Negotiated authn=SASL
0809 12:10:38.184075 (+ 1630us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 12:10:38.185163 (+ 1088us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 12:10:38.187022 (+ 1859us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 12:10:38.187608 (+ 586us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 12:10:38.187776 (+ 168us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 12:10:38.192055 (+ 4279us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 12:10:38.192064 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 12:10:38.192071 (+ 7us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0809 12:10:38.192119 (+ 48us) server_negotiation.cc:922] calling sasl_server_start()
0809 12:10:38.195895 (+ 3776us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 12:10:38.196695 (+ 800us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 12:10:38.196703 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 12:10:38.196709 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0809 12:10:38.196861 (+ 152us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 12:10:38.197275 (+ 414us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 12:10:38.197282 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 12:10:38.197287 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0809 12:10:38.197681 (+ 394us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 12:10:38.197838 (+ 157us) server_negotiation.cc:1035] Waiting for connection context
0809 12:10:38.198788 (+ 950us) server_negotiation.cc:299] Negotiation successful
0809 12:10:38.199092 (+ 304us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":432,"thread_start_us":303,"threads_started":1}
I20260809 12:10:38.200546 7749 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 12:10:38.179426 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.222.62:41983 (local address 127.0.222.1:49233)
0809 12:10:38.180153 (+ 727us) negotiation.cc:107] Waiting for socket to connect
0809 12:10:38.180197 (+ 44us) client_negotiation.cc:174] Beginning negotiation
0809 12:10:38.181947 (+ 1750us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 12:10:38.182692 (+ 745us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 12:10:38.182701 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 12:10:38.183324 (+ 623us) client_negotiation.cc:189] Negotiated authn=SASL
0809 12:10:38.183895 (+ 571us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 12:10:38.183913 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 12:10:38.185349 (+ 1436us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 12:10:38.185354 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 12:10:38.186839 (+ 1485us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 12:10:38.186853 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 12:10:38.187856 (+ 1003us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 12:10:38.187861 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 12:10:38.188184 (+ 323us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 12:10:38.188946 (+ 762us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0809 12:10:38.188980 (+ 34us) client_negotiation.cc:656] calling sasl_client_start()
0809 12:10:38.191816 (+ 2836us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0809 12:10:38.196066 (+ 4250us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0809 12:10:38.196074 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0809 12:10:38.196087 (+ 13us) client_negotiation.cc:760] calling sasl_client_step()
0809 12:10:38.196552 (+ 465us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0809 12:10:38.196972 (+ 420us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0809 12:10:38.196975 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0809 12:10:38.196977 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0809 12:10:38.197156 (+ 179us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0809 12:10:38.197841 (+ 685us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0809 12:10:38.197848 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0809 12:10:38.198286 (+ 438us) client_negotiation.cc:769] Sending connection context
0809 12:10:38.198613 (+ 327us) client_negotiation.cc:240] Negotiation successful
0809 12:10:38.199229 (+ 616us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":501,"thread_start_us":299,"threads_started":1}
I20260809 12:10:38.202283 7746 heartbeater.cc:347] Connected to a master server at 127.0.222.62:41983
I20260809 12:10:38.202592 7746 heartbeater.cc:464] Registering TS with master...
I20260809 12:10:38.203392 7746 heartbeater.cc:511] Master 127.0.222.62:41983 requested a full tablet report, sending...
I20260809 12:10:38.205775 7548 ts_manager.cc:194] Registered new tserver with Master: df0db1a138554d00956d87f5c172ee10 (127.0.222.1:43733)
I20260809 12:10:38.208051 7548 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.0.222.1@KRBTEST.COM'} at 127.0.222.1:49233
WARNING: no policy specified for oryx/127.0.222.2@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.0.222.2@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.0.222.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskgH5Pjm/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786277382073938-888-0/krb5kdc/oryx_127.0.222.2.keytab.
Entry for principal oryx/127.0.222.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskgH5Pjm/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786277382073938-888-0/krb5kdc/oryx_127.0.222.2.keytab.
I20260809 12:10:38.246800 888 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskgH5Pjm/build/asan/bin/kudu
/tmp/dist-test-taskgH5Pjm/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskgH5Pjm/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786277382073938-888-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskgH5Pjm/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786277382073938-888-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskgH5Pjm/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786277382073938-888-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskgH5Pjm/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786277382073938-888-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.222.2:0
--local_ip_for_outbound_sockets=127.0.222.2
--webserver_interface=127.0.222.2
--webserver_port=0
--tserver_master_addrs=127.0.222.62:41983
--keytab_file=/tmp/dist-test-taskgH5Pjm/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786277382073938-888-0/krb5kdc/oryx_127.0.222.2.keytab
--principal=oryx/127.0.222.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.222.20:38069
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskgH5Pjm/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786277382073938-888-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskgH5Pjm/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786277382073938-888-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskgH5Pjm/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786277382073938-888-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260809 12:10:38.583890 7752 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260809 12:10:38.584182 7752 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260809 12:10:38.584367 7752 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260809 12:10:38.591740 7752 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260809 12:10:38.591892 7752 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260809 12:10:38.592088 7752 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.222.2
I20260809 12:10:38.602221 7752 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.222.20:38069
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskgH5Pjm/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786277382073938-888-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskgH5Pjm/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786277382073938-888-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskgH5Pjm/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786277382073938-888-0/krb5kdc/oryx_127.0.222.2.keytab
--principal=oryx/127.0.222.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.222.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskgH5Pjm/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786277382073938-888-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.222.2
--webserver_port=0
--tserver_master_addrs=127.0.222.62:41983
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.222.2
--log_dir=/tmp/dist-test-taskgH5Pjm/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786277382073938-888-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 09 Aug 2026 11:06:41 UTC on ccycloud.quasar-bwoogy.root.comops.site
build id 4575991
ASAN enabled
I20260809 12:10:38.603612 7752 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260809 12:10:38.604992 7752 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260809 12:10:38.618675 7760 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 12:10:38.618731 7758 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 12:10:38.618727 7757 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260809 12:10:38.619269 7752 server_base.cc:1034] running on GCE node
I20260809 12:10:38.619973 7752 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260809 12:10:38.621025 7752 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260809 12:10:38.622259 7752 hybrid_clock.cc:648] HybridClock initialized: now 1786277438622219 us; error 53 us; skew 500 ppm
I20260809 12:10:38.622682 7752 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 09 12:10:38 dist-test-slave-2w3w krb5kdc[7502](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786277438, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.222.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260809 12:10:38.631382 7752 init.cc:377] Logged in from keytab as oryx/127.0.222.2@KRBTEST.COM (short username oryx)
I20260809 12:10:38.632916 7752 webserver.cc:460] Webserver started at http://127.0.222.2:41681/ using document root <none> and password file <none>
I20260809 12:10:38.633672 7752 fs_manager.cc:359] Metadata directory not provided
I20260809 12:10:38.633833 7752 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260809 12:10:38.634164 7752 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260809 12:10:38.637329 7752 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskgH5Pjm/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786277382073938-888-0/minicluster-data/ts-1/data/instance:
uuid: "94413a87e0a14ef195baa2a799a9e0a8"
format_stamp: "Formatted at 2026-08-09 12:10:38 on dist-test-slave-2w3w"
server_key: "fc42e1038ea5fe088e9187d212704211"
server_key_iv: "6a728afac96572a760f842693c2562b5"
server_key_version: "encryptionkey@0"
I20260809 12:10:38.638144 7752 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskgH5Pjm/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786277382073938-888-0/minicluster-data/ts-1/wal/instance:
uuid: "94413a87e0a14ef195baa2a799a9e0a8"
format_stamp: "Formatted at 2026-08-09 12:10:38 on dist-test-slave-2w3w"
server_key: "fc42e1038ea5fe088e9187d212704211"
server_key_iv: "6a728afac96572a760f842693c2562b5"
server_key_version: "encryptionkey@0"
I20260809 12:10:38.643549 7752 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.003s sys 0.004s
I20260809 12:10:38.646911 7767 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260809 12:10:38.648243 7752 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260809 12:10:38.648432 7752 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskgH5Pjm/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786277382073938-888-0/minicluster-data/ts-1/data,/tmp/dist-test-taskgH5Pjm/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786277382073938-888-0/minicluster-data/ts-1/wal
uuid: "94413a87e0a14ef195baa2a799a9e0a8"
format_stamp: "Formatted at 2026-08-09 12:10:38 on dist-test-slave-2w3w"
server_key: "fc42e1038ea5fe088e9187d212704211"
server_key_iv: "6a728afac96572a760f842693c2562b5"
server_key_version: "encryptionkey@0"
I20260809 12:10:38.648635 7752 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskgH5Pjm/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786277382073938-888-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskgH5Pjm/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786277382073938-888-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskgH5Pjm/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786277382073938-888-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260809 12:10:38.666600 7752 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260809 12:10:38.671190 7752 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260809 12:10:38.671495 7752 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260809 12:10:38.672542 7752 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260809 12:10:38.673938 7752 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260809 12:10:38.674053 7752 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260809 12:10:38.674204 7752 ts_tablet_manager.cc:610] Registered 0 tablets
I20260809 12:10:38.674279 7752 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260809 12:10:38.708778 7752 rpc_server.cc:307] RPC server started. Bound to: 127.0.222.2:33587
I20260809 12:10:38.708866 7887 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.222.2:33587 every 8 connection(s)
I20260809 12:10:38.710142 7752 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskgH5Pjm/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786277382073938-888-0/minicluster-data/ts-1/data/info.pb
I20260809 12:10:38.712628 888 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskgH5Pjm/build/asan/bin/kudu as pid 7752
I20260809 12:10:38.712802 888 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskgH5Pjm/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786277382073938-888-0/minicluster-data/ts-1/wal/instance
I20260809 12:10:38.713368 888 external_mini_cluster.cc:1428] Setting key d668cb29a48fd422a4bbadf8385a683b
Aug 09 12:10:38 dist-test-slave-2w3w krb5kdc[7502](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786277438, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.222.2@KRBTEST.COM for oryx/127.0.222.62@KRBTEST.COM
I20260809 12:10:38.732270 7890 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 12:10:38.712808 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.222.2:52271 (local address 127.0.222.62:41983)
0809 12:10:38.713285 (+ 477us) server_negotiation.cc:206] Beginning negotiation
0809 12:10:38.713289 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0809 12:10:38.714498 (+ 1209us) server_negotiation.cc:407] Connection header received
0809 12:10:38.716549 (+ 2051us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 12:10:38.716554 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 12:10:38.716644 (+ 90us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 12:10:38.716773 (+ 129us) server_negotiation.cc:226] Negotiated authn=SASL
0809 12:10:38.718310 (+ 1537us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 12:10:38.719146 (+ 836us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 12:10:38.720731 (+ 1585us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 12:10:38.721168 (+ 437us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 12:10:38.721315 (+ 147us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 12:10:38.725003 (+ 3688us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 12:10:38.725013 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 12:10:38.725020 (+ 7us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0809 12:10:38.725068 (+ 48us) server_negotiation.cc:922] calling sasl_server_start()
0809 12:10:38.728792 (+ 3724us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 12:10:38.729560 (+ 768us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 12:10:38.729569 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 12:10:38.729575 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0809 12:10:38.729742 (+ 167us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 12:10:38.730165 (+ 423us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 12:10:38.730172 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 12:10:38.730178 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0809 12:10:38.730570 (+ 392us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 12:10:38.730743 (+ 173us) server_negotiation.cc:1035] Waiting for connection context
0809 12:10:38.731497 (+ 754us) server_negotiation.cc:299] Negotiation successful
0809 12:10:38.731773 (+ 276us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":363,"thread_start_us":259,"threads_started":1}
I20260809 12:10:38.732877 7891 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 12:10:38.713434 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.222.62:41983 (local address 127.0.222.2:52271)
0809 12:10:38.714332 (+ 898us) negotiation.cc:107] Waiting for socket to connect
0809 12:10:38.714382 (+ 50us) client_negotiation.cc:174] Beginning negotiation
0809 12:10:38.716273 (+ 1891us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 12:10:38.717015 (+ 742us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 12:10:38.717024 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 12:10:38.717533 (+ 509us) client_negotiation.cc:189] Negotiated authn=SASL
0809 12:10:38.718126 (+ 593us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 12:10:38.718143 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 12:10:38.719311 (+ 1168us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 12:10:38.719315 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 12:10:38.720561 (+ 1246us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 12:10:38.720574 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 12:10:38.721411 (+ 837us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 12:10:38.721418 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 12:10:38.721655 (+ 237us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 12:10:38.722367 (+ 712us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0809 12:10:38.722394 (+ 27us) client_negotiation.cc:656] calling sasl_client_start()
0809 12:10:38.724679 (+ 2285us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0809 12:10:38.728947 (+ 4268us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0809 12:10:38.728954 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0809 12:10:38.728967 (+ 13us) client_negotiation.cc:760] calling sasl_client_step()
0809 12:10:38.729419 (+ 452us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0809 12:10:38.729864 (+ 445us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0809 12:10:38.729868 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0809 12:10:38.729870 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0809 12:10:38.730051 (+ 181us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0809 12:10:38.730703 (+ 652us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0809 12:10:38.730708 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0809 12:10:38.731110 (+ 402us) client_negotiation.cc:769] Sending connection context
0809 12:10:38.731356 (+ 246us) client_negotiation.cc:240] Negotiation successful
0809 12:10:38.731739 (+ 383us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":586,"thread_start_us":328,"threads_started":1}
I20260809 12:10:38.734390 7888 heartbeater.cc:347] Connected to a master server at 127.0.222.62:41983
I20260809 12:10:38.734772 7888 heartbeater.cc:464] Registering TS with master...
I20260809 12:10:38.735740 7888 heartbeater.cc:511] Master 127.0.222.62:41983 requested a full tablet report, sending...
I20260809 12:10:38.737502 7548 ts_manager.cc:194] Registered new tserver with Master: 94413a87e0a14ef195baa2a799a9e0a8 (127.0.222.2:33587)
I20260809 12:10:38.738343 7548 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.0.222.2@KRBTEST.COM'} at 127.0.222.2:52271
WARNING: no policy specified for oryx/127.0.222.3@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.0.222.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.0.222.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskgH5Pjm/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786277382073938-888-0/krb5kdc/oryx_127.0.222.3.keytab.
Entry for principal oryx/127.0.222.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskgH5Pjm/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786277382073938-888-0/krb5kdc/oryx_127.0.222.3.keytab.
I20260809 12:10:38.774351 888 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskgH5Pjm/build/asan/bin/kudu
/tmp/dist-test-taskgH5Pjm/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskgH5Pjm/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786277382073938-888-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskgH5Pjm/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786277382073938-888-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskgH5Pjm/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786277382073938-888-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskgH5Pjm/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786277382073938-888-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.222.3:0
--local_ip_for_outbound_sockets=127.0.222.3
--webserver_interface=127.0.222.3
--webserver_port=0
--tserver_master_addrs=127.0.222.62:41983
--keytab_file=/tmp/dist-test-taskgH5Pjm/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786277382073938-888-0/krb5kdc/oryx_127.0.222.3.keytab
--principal=oryx/127.0.222.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.222.20:38069
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskgH5Pjm/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786277382073938-888-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskgH5Pjm/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786277382073938-888-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskgH5Pjm/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786277382073938-888-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260809 12:10:39.112268 7894 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260809 12:10:39.112612 7894 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260809 12:10:39.112790 7894 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260809 12:10:39.120121 7894 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260809 12:10:39.120272 7894 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260809 12:10:39.120448 7894 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.222.3
I20260809 12:10:39.130379 7894 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.222.20:38069
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskgH5Pjm/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786277382073938-888-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskgH5Pjm/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786277382073938-888-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskgH5Pjm/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786277382073938-888-0/krb5kdc/oryx_127.0.222.3.keytab
--principal=oryx/127.0.222.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.222.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskgH5Pjm/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786277382073938-888-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.222.3
--webserver_port=0
--tserver_master_addrs=127.0.222.62:41983
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.222.3
--log_dir=/tmp/dist-test-taskgH5Pjm/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786277382073938-888-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 09 Aug 2026 11:06:41 UTC on ccycloud.quasar-bwoogy.root.comops.site
build id 4575991
ASAN enabled
I20260809 12:10:39.131731 7894 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260809 12:10:39.133074 7894 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260809 12:10:39.146806 7899 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 12:10:39.146837 7900 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 12:10:39.147007 7902 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260809 12:10:39.147979 7894 server_base.cc:1034] running on GCE node
I20260809 12:10:39.148598 7894 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260809 12:10:39.149545 7894 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260809 12:10:39.150774 7894 hybrid_clock.cc:648] HybridClock initialized: now 1786277439150714 us; error 54 us; skew 500 ppm
I20260809 12:10:39.151156 7894 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 09 12:10:39 dist-test-slave-2w3w krb5kdc[7502](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786277439, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.222.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260809 12:10:39.154325 7894 init.cc:377] Logged in from keytab as oryx/127.0.222.3@KRBTEST.COM (short username oryx)
I20260809 12:10:39.155798 7894 webserver.cc:460] Webserver started at http://127.0.222.3:35519/ using document root <none> and password file <none>
I20260809 12:10:39.156509 7894 fs_manager.cc:359] Metadata directory not provided
I20260809 12:10:39.156659 7894 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260809 12:10:39.156975 7894 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260809 12:10:39.159993 7894 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskgH5Pjm/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786277382073938-888-0/minicluster-data/ts-2/data/instance:
uuid: "c40926a9373242b0961349a2719c22ab"
format_stamp: "Formatted at 2026-08-09 12:10:39 on dist-test-slave-2w3w"
server_key: "d550667833dac40fe63877ee3b427f59"
server_key_iv: "c0c7ae44eaf7a0b79e122a514fa08eed"
server_key_version: "encryptionkey@0"
I20260809 12:10:39.160801 7894 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskgH5Pjm/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786277382073938-888-0/minicluster-data/ts-2/wal/instance:
uuid: "c40926a9373242b0961349a2719c22ab"
format_stamp: "Formatted at 2026-08-09 12:10:39 on dist-test-slave-2w3w"
server_key: "d550667833dac40fe63877ee3b427f59"
server_key_iv: "c0c7ae44eaf7a0b79e122a514fa08eed"
server_key_version: "encryptionkey@0"
I20260809 12:10:39.165992 7894 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.000s
I20260809 12:10:39.169572 7909 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260809 12:10:39.170898 7894 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260809 12:10:39.171141 7894 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskgH5Pjm/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786277382073938-888-0/minicluster-data/ts-2/data,/tmp/dist-test-taskgH5Pjm/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786277382073938-888-0/minicluster-data/ts-2/wal
uuid: "c40926a9373242b0961349a2719c22ab"
format_stamp: "Formatted at 2026-08-09 12:10:39 on dist-test-slave-2w3w"
server_key: "d550667833dac40fe63877ee3b427f59"
server_key_iv: "c0c7ae44eaf7a0b79e122a514fa08eed"
server_key_version: "encryptionkey@0"
I20260809 12:10:39.171361 7894 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskgH5Pjm/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786277382073938-888-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskgH5Pjm/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786277382073938-888-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskgH5Pjm/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786277382073938-888-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260809 12:10:39.190227 7894 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260809 12:10:39.194319 7894 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260809 12:10:39.194569 7894 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260809 12:10:39.195591 7894 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260809 12:10:39.196969 7894 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260809 12:10:39.197054 7894 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260809 12:10:39.197141 7894 ts_tablet_manager.cc:610] Registered 0 tablets
I20260809 12:10:39.197187 7894 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260809 12:10:39.211252 7746 heartbeater.cc:503] Master 127.0.222.62:41983 was elected leader, sending a full tablet report...
I20260809 12:10:39.231295 7894 rpc_server.cc:307] RPC server started. Bound to: 127.0.222.3:37357
I20260809 12:10:39.231389 8026 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.222.3:37357 every 8 connection(s)
I20260809 12:10:39.232671 7894 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskgH5Pjm/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786277382073938-888-0/minicluster-data/ts-2/data/info.pb
I20260809 12:10:39.239677 888 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskgH5Pjm/build/asan/bin/kudu as pid 7894
I20260809 12:10:39.239850 888 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskgH5Pjm/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786277382073938-888-0/minicluster-data/ts-2/wal/instance
I20260809 12:10:39.240324 888 external_mini_cluster.cc:1428] Setting key ff7a4c5219f0ee25cc125dc411685573
Aug 09 12:10:39 dist-test-slave-2w3w krb5kdc[7502](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786277439, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.222.3@KRBTEST.COM for oryx/127.0.222.62@KRBTEST.COM
I20260809 12:10:39.253541 8029 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 12:10:39.235557 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.222.3:52931 (local address 127.0.222.62:41983)
0809 12:10:39.236065 (+ 508us) server_negotiation.cc:206] Beginning negotiation
0809 12:10:39.236070 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0809 12:10:39.236862 (+ 792us) server_negotiation.cc:407] Connection header received
0809 12:10:39.238666 (+ 1804us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 12:10:39.238672 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 12:10:39.238763 (+ 91us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 12:10:39.238900 (+ 137us) server_negotiation.cc:226] Negotiated authn=SASL
0809 12:10:39.240581 (+ 1681us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 12:10:39.241547 (+ 966us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 12:10:39.243720 (+ 2173us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 12:10:39.244179 (+ 459us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 12:10:39.244282 (+ 103us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 12:10:39.247662 (+ 3380us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 12:10:39.247667 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 12:10:39.247670 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0809 12:10:39.247704 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0809 12:10:39.250480 (+ 2776us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 12:10:39.251222 (+ 742us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 12:10:39.251226 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 12:10:39.251228 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0809 12:10:39.251369 (+ 141us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 12:10:39.251891 (+ 522us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 12:10:39.251896 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 12:10:39.251899 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0809 12:10:39.252187 (+ 288us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 12:10:39.252328 (+ 141us) server_negotiation.cc:1035] Waiting for connection context
0809 12:10:39.253089 (+ 761us) server_negotiation.cc:299] Negotiation successful
0809 12:10:39.253259 (+ 170us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":399,"thread_start_us":290,"threads_started":1}
I20260809 12:10:39.254714 8031 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 12:10:39.235953 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.222.62:41983 (local address 127.0.222.3:52931)
0809 12:10:39.236656 (+ 703us) negotiation.cc:107] Waiting for socket to connect
0809 12:10:39.236700 (+ 44us) client_negotiation.cc:174] Beginning negotiation
0809 12:10:39.238362 (+ 1662us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 12:10:39.239167 (+ 805us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 12:10:39.239175 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 12:10:39.239709 (+ 534us) client_negotiation.cc:189] Negotiated authn=SASL
0809 12:10:39.240393 (+ 684us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 12:10:39.240420 (+ 27us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 12:10:39.241778 (+ 1358us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 12:10:39.241783 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 12:10:39.243558 (+ 1775us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 12:10:39.243575 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 12:10:39.244366 (+ 791us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 12:10:39.244371 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 12:10:39.244579 (+ 208us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 12:10:39.245192 (+ 613us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0809 12:10:39.245214 (+ 22us) client_negotiation.cc:656] calling sasl_client_start()
0809 12:10:39.247462 (+ 2248us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0809 12:10:39.250595 (+ 3133us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0809 12:10:39.250601 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0809 12:10:39.250614 (+ 13us) client_negotiation.cc:760] calling sasl_client_step()
0809 12:10:39.251110 (+ 496us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0809 12:10:39.251467 (+ 357us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0809 12:10:39.251471 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0809 12:10:39.251473 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0809 12:10:39.251708 (+ 235us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0809 12:10:39.252313 (+ 605us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0809 12:10:39.252318 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0809 12:10:39.252720 (+ 402us) client_negotiation.cc:769] Sending connection context
0809 12:10:39.252962 (+ 242us) client_negotiation.cc:240] Negotiation successful
0809 12:10:39.253424 (+ 462us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":475,"thread_start_us":282,"threads_started":1}
I20260809 12:10:39.256413 8027 heartbeater.cc:347] Connected to a master server at 127.0.222.62:41983
I20260809 12:10:39.256724 8027 heartbeater.cc:464] Registering TS with master...
I20260809 12:10:39.257453 8027 heartbeater.cc:511] Master 127.0.222.62:41983 requested a full tablet report, sending...
I20260809 12:10:39.259037 7547 ts_manager.cc:194] Registered new tserver with Master: c40926a9373242b0961349a2719c22ab (127.0.222.3:37357)
I20260809 12:10:39.259935 7547 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.0.222.3@KRBTEST.COM'} at 127.0.222.3:52931
I20260809 12:10:39.268478 888 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
WARNING: no policy specified for oryx/127.0.222.61@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.0.222.61@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.0.222.61 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskgH5Pjm/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786277382073938-888-0/krb5kdc/oryx_127.0.222.61.keytab.
Entry for principal oryx/127.0.222.61 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskgH5Pjm/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786277382073938-888-0/krb5kdc/oryx_127.0.222.61.keytab.
I20260809 12:10:39.330818 888 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskgH5Pjm/build/asan/bin/kudu
/tmp/dist-test-taskgH5Pjm/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskgH5Pjm/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786277382073938-888-0/minicluster-data/master-1/wal
--fs_data_dirs=/tmp/dist-test-taskgH5Pjm/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786277382073938-888-0/minicluster-data/master-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskgH5Pjm/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786277382073938-888-0/minicluster-data/master-1/logs
--server_dump_info_path=/tmp/dist-test-taskgH5Pjm/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786277382073938-888-0/minicluster-data/master-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.222.61:32831
--webserver_interface=127.0.222.61
--webserver_port=0
--keytab_file=/tmp/dist-test-taskgH5Pjm/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786277382073938-888-0/krb5kdc/oryx_127.0.222.61.keytab
--principal=oryx/127.0.222.61
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.222.20:38069
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.222.62:41983,127.0.222.61:32831
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskgH5Pjm/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786277382073938-888-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskgH5Pjm/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786277382073938-888-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskgH5Pjm/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786277382073938-888-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260809 12:10:39.672099 8034 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260809 12:10:39.672497 8034 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260809 12:10:39.672662 8034 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260809 12:10:39.681347 8034 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260809 12:10:39.681522 8034 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260809 12:10:39.681628 8034 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260809 12:10:39.681730 8034 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260809 12:10:39.681806 8034 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260809 12:10:39.692308 8034 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.222.20:38069
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskgH5Pjm/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786277382073938-888-0/minicluster-data/master-1/data
--fs_wal_dir=/tmp/dist-test-taskgH5Pjm/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786277382073938-888-0/minicluster-data/master-1/wal
--ipki_ca_key_size=768
--master_addresses=127.0.222.62:41983,127.0.222.61:32831
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskgH5Pjm/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786277382073938-888-0/krb5kdc/oryx_127.0.222.61.keytab
--principal=oryx/127.0.222.61
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.222.61:32831
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskgH5Pjm/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786277382073938-888-0/minicluster-data/master-1/data/info.pb
--webserver_interface=127.0.222.61
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskgH5Pjm/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786277382073938-888-0/minicluster-data/master-1/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 09 Aug 2026 11:06:41 UTC on ccycloud.quasar-bwoogy.root.comops.site
build id 4575991
ASAN enabled
W20260809 12:10:39.692745 8034 master_options.cc:55] Only 2 masters are specified by master_addresses_flag ('127.0.222.62:41983,127.0.222.61:32831'), but minimum of 3 are required to tolerate failures of any one master. It is recommended to use at least 3 masters.
I20260809 12:10:39.693974 8034 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260809 12:10:39.695664 8034 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260809 12:10:39.707769 8039 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 12:10:39.707800 8040 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 12:10:39.707875 8042 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260809 12:10:39.708987 8034 server_base.cc:1034] running on GCE node
I20260809 12:10:39.709816 8034 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260809 12:10:39.711208 8034 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260809 12:10:39.712463 8034 hybrid_clock.cc:648] HybridClock initialized: now 1786277439712384 us; error 54 us; skew 500 ppm
I20260809 12:10:39.712849 8034 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 09 12:10:39 dist-test-slave-2w3w krb5kdc[7502](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786277439, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.222.61@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260809 12:10:39.716132 8034 init.cc:377] Logged in from keytab as oryx/127.0.222.61@KRBTEST.COM (short username oryx)
I20260809 12:10:39.717568 8034 webserver.cc:460] Webserver started at http://127.0.222.61:34325/ using document root <none> and password file <none>
I20260809 12:10:39.718261 8034 fs_manager.cc:359] Metadata directory not provided
I20260809 12:10:39.718405 8034 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260809 12:10:39.718724 8034 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260809 12:10:39.721910 8034 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskgH5Pjm/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786277382073938-888-0/minicluster-data/master-1/data/instance:
uuid: "cbbb52854812417da2325b27e509812d"
format_stamp: "Formatted at 2026-08-09 12:10:39 on dist-test-slave-2w3w"
server_key: "0c86a30d73b4f22d89209bc662d6bac9"
server_key_iv: "a23dca56137a4a133a2d91161ad6e5a1"
server_key_version: "encryptionkey@0"
I20260809 12:10:39.722709 8034 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskgH5Pjm/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786277382073938-888-0/minicluster-data/master-1/wal/instance:
uuid: "cbbb52854812417da2325b27e509812d"
format_stamp: "Formatted at 2026-08-09 12:10:39 on dist-test-slave-2w3w"
server_key: "0c86a30d73b4f22d89209bc662d6bac9"
server_key_iv: "a23dca56137a4a133a2d91161ad6e5a1"
server_key_version: "encryptionkey@0"
I20260809 12:10:39.728166 8034 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.000s sys 0.004s
I20260809 12:10:39.731624 8050 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260809 12:10:39.732982 8034 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260809 12:10:39.733186 8034 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskgH5Pjm/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786277382073938-888-0/minicluster-data/master-1/data,/tmp/dist-test-taskgH5Pjm/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786277382073938-888-0/minicluster-data/master-1/wal
uuid: "cbbb52854812417da2325b27e509812d"
format_stamp: "Formatted at 2026-08-09 12:10:39 on dist-test-slave-2w3w"
server_key: "0c86a30d73b4f22d89209bc662d6bac9"
server_key_iv: "a23dca56137a4a133a2d91161ad6e5a1"
server_key_version: "encryptionkey@0"
I20260809 12:10:39.733403 8034 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskgH5Pjm/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786277382073938-888-0/minicluster-data/master-1/wal
metadata directory: /tmp/dist-test-taskgH5Pjm/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786277382073938-888-0/minicluster-data/master-1/wal
1 data directories: /tmp/dist-test-taskgH5Pjm/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786277382073938-888-0/minicluster-data/master-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260809 12:10:39.741676 7888 heartbeater.cc:503] Master 127.0.222.62:41983 was elected leader, sending a full tablet report...
I20260809 12:10:39.745698 8034 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260809 12:10:39.749570 8034 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260809 12:10:39.749840 8034 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260809 12:10:39.769722 8034 rpc_server.cc:307] RPC server started. Bound to: 127.0.222.61:32831
I20260809 12:10:39.769742 8104 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.222.61:32831 every 8 connection(s)
I20260809 12:10:39.771142 8034 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskgH5Pjm/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786277382073938-888-0/minicluster-data/master-1/data/info.pb
I20260809 12:10:39.775040 8105 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260809 12:10:39.776546 888 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskgH5Pjm/build/asan/bin/kudu as pid 8034
I20260809 12:10:39.784561 8105 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.222.62" port: 41983 } has no permanent_uuid. Determining permanent_uuid...
Aug 09 12:10:39 dist-test-slave-2w3w krb5kdc[7502](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786277437, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.0.222.61@KRBTEST.COM
Aug 09 12:10:39 dist-test-slave-2w3w krb5kdc[7502](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786277439, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.222.61@KRBTEST.COM for oryx/127.0.222.62@KRBTEST.COM
I20260809 12:10:39.801259 8108 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 12:10:39.778782 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46484 (local address 127.0.222.61:32831)
0809 12:10:39.779724 (+ 942us) server_negotiation.cc:206] Beginning negotiation
0809 12:10:39.779756 (+ 32us) server_negotiation.cc:399] Waiting for connection header
0809 12:10:39.779802 (+ 46us) server_negotiation.cc:407] Connection header received
0809 12:10:39.781385 (+ 1583us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 12:10:39.781407 (+ 22us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 12:10:39.781786 (+ 379us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 12:10:39.782104 (+ 318us) server_negotiation.cc:226] Negotiated authn=SASL
0809 12:10:39.784055 (+ 1951us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 12:10:39.785471 (+ 1416us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 12:10:39.786894 (+ 1423us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 12:10:39.787860 (+ 966us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 12:10:39.788655 (+ 795us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 12:10:39.793016 (+ 4361us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 12:10:39.793028 (+ 12us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 12:10:39.793043 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0809 12:10:39.793100 (+ 57us) server_negotiation.cc:922] calling sasl_server_start()
0809 12:10:39.796748 (+ 3648us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 12:10:39.797460 (+ 712us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 12:10:39.797467 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 12:10:39.797473 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0809 12:10:39.797617 (+ 144us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 12:10:39.798033 (+ 416us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 12:10:39.798036 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 12:10:39.798038 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0809 12:10:39.798654 (+ 616us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 12:10:39.798864 (+ 210us) server_negotiation.cc:1035] Waiting for connection context
0809 12:10:39.799468 (+ 604us) server_negotiation.cc:299] Negotiation successful
0809 12:10:39.799838 (+ 370us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":653,"thread_start_us":436,"threads_started":1}
I20260809 12:10:39.803862 8109 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 12:10:39.787432 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33030 (local address 127.0.222.62:41983)
0809 12:10:39.787996 (+ 564us) server_negotiation.cc:206] Beginning negotiation
0809 12:10:39.788004 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0809 12:10:39.788775 (+ 771us) server_negotiation.cc:407] Connection header received
0809 12:10:39.788876 (+ 101us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 12:10:39.788883 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 12:10:39.788972 (+ 89us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 12:10:39.789117 (+ 145us) server_negotiation.cc:226] Negotiated authn=SASL
0809 12:10:39.790294 (+ 1177us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 12:10:39.791415 (+ 1121us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 12:10:39.793049 (+ 1634us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 12:10:39.793477 (+ 428us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 12:10:39.793643 (+ 166us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 12:10:39.797462 (+ 3819us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 12:10:39.797469 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 12:10:39.797474 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0809 12:10:39.797522 (+ 48us) server_negotiation.cc:922] calling sasl_server_start()
0809 12:10:39.800661 (+ 3139us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 12:10:39.801674 (+ 1013us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 12:10:39.801680 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 12:10:39.801683 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0809 12:10:39.801819 (+ 136us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 12:10:39.802210 (+ 391us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 12:10:39.802218 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 12:10:39.802221 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0809 12:10:39.802543 (+ 322us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 12:10:39.802704 (+ 161us) server_negotiation.cc:1035] Waiting for connection context
0809 12:10:39.803257 (+ 553us) server_negotiation.cc:299] Negotiation successful
0809 12:10:39.803486 (+ 229us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":418,"thread_start_us":280,"threads_started":1}
I20260809 12:10:39.803917 8110 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 12:10:39.787456 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.222.62:41983 (local address 127.0.0.1:33030)
0809 12:10:39.788430 (+ 974us) negotiation.cc:107] Waiting for socket to connect
0809 12:10:39.788471 (+ 41us) client_negotiation.cc:174] Beginning negotiation
0809 12:10:39.788645 (+ 174us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 12:10:39.789203 (+ 558us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 12:10:39.789208 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 12:10:39.789862 (+ 654us) client_negotiation.cc:189] Negotiated authn=SASL
0809 12:10:39.790120 (+ 258us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 12:10:39.790128 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 12:10:39.791664 (+ 1536us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 12:10:39.791669 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 12:10:39.792911 (+ 1242us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 12:10:39.792923 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 12:10:39.793685 (+ 762us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 12:10:39.793692 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 12:10:39.793968 (+ 276us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 12:10:39.794583 (+ 615us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0809 12:10:39.794614 (+ 31us) client_negotiation.cc:656] calling sasl_client_start()
0809 12:10:39.797261 (+ 2647us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0809 12:10:39.800825 (+ 3564us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0809 12:10:39.800837 (+ 12us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0809 12:10:39.800844 (+ 7us) client_negotiation.cc:760] calling sasl_client_step()
0809 12:10:39.801522 (+ 678us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0809 12:10:39.801935 (+ 413us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0809 12:10:39.801941 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0809 12:10:39.801945 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0809 12:10:39.802094 (+ 149us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0809 12:10:39.802680 (+ 586us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0809 12:10:39.802688 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0809 12:10:39.802874 (+ 186us) client_negotiation.cc:769] Sending connection context
0809 12:10:39.803171 (+ 297us) client_negotiation.cc:240] Negotiation successful
0809 12:10:39.803491 (+ 320us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":720,"thread_start_us":567,"threads_started":1}
I20260809 12:10:39.805812 8105 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.222.61" port: 32831 } has no permanent_uuid. Determining permanent_uuid...
Aug 09 12:10:39 dist-test-slave-2w3w krb5kdc[7502](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786277439, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.222.61@KRBTEST.COM for oryx/127.0.222.61@KRBTEST.COM
I20260809 12:10:39.817830 8110 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 12:10:39.806563 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.222.61:32831 (local address 127.0.0.1:46494)
0809 12:10:39.806754 (+ 191us) negotiation.cc:107] Waiting for socket to connect
0809 12:10:39.806779 (+ 25us) client_negotiation.cc:174] Beginning negotiation
0809 12:10:39.806863 (+ 84us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 12:10:39.807213 (+ 350us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 12:10:39.807216 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 12:10:39.807523 (+ 307us) client_negotiation.cc:189] Negotiated authn=SASL
0809 12:10:39.807720 (+ 197us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 12:10:39.807730 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 12:10:39.808640 (+ 910us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 12:10:39.808644 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 12:10:39.809551 (+ 907us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 12:10:39.809563 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 12:10:39.810234 (+ 671us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 12:10:39.810253 (+ 19us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 12:10:39.810406 (+ 153us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 12:10:39.811045 (+ 639us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0809 12:10:39.811062 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0809 12:10:39.813007 (+ 1945us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0809 12:10:39.815868 (+ 2861us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0809 12:10:39.815872 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0809 12:10:39.815875 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0809 12:10:39.816251 (+ 376us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0809 12:10:39.816574 (+ 323us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0809 12:10:39.816577 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0809 12:10:39.816579 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0809 12:10:39.816701 (+ 122us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0809 12:10:39.817152 (+ 451us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0809 12:10:39.817155 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0809 12:10:39.817267 (+ 112us) client_negotiation.cc:769] Sending connection context
0809 12:10:39.817401 (+ 134us) client_negotiation.cc:240] Negotiation successful
0809 12:10:39.817598 (+ 197us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":56}
I20260809 12:10:39.818048 8108 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 12:10:39.806681 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46494 (local address 127.0.222.61:32831)
0809 12:10:39.806832 (+ 151us) server_negotiation.cc:206] Beginning negotiation
0809 12:10:39.806836 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0809 12:10:39.806852 (+ 16us) server_negotiation.cc:407] Connection header received
0809 12:10:39.806956 (+ 104us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 12:10:39.807002 (+ 46us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 12:10:39.807071 (+ 69us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 12:10:39.807146 (+ 75us) server_negotiation.cc:226] Negotiated authn=SASL
0809 12:10:39.807824 (+ 678us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 12:10:39.808506 (+ 682us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 12:10:39.809676 (+ 1170us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 12:10:39.810067 (+ 391us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 12:10:39.810165 (+ 98us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 12:10:39.813177 (+ 3012us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 12:10:39.813182 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 12:10:39.813185 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0809 12:10:39.813213 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0809 12:10:39.815747 (+ 2534us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 12:10:39.816356 (+ 609us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 12:10:39.816360 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 12:10:39.816362 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0809 12:10:39.816467 (+ 105us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 12:10:39.816791 (+ 324us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 12:10:39.816794 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 12:10:39.816796 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0809 12:10:39.817051 (+ 255us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 12:10:39.817157 (+ 106us) server_negotiation.cc:1035] Waiting for connection context
0809 12:10:39.817478 (+ 321us) server_negotiation.cc:299] Negotiation successful
0809 12:10:39.817728 (+ 250us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":68,"server-negotiator.queue_time_us":58}
I20260809 12:10:39.822620 8105 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P cbbb52854812417da2325b27e509812d: Bootstrap starting.
I20260809 12:10:39.825678 8105 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P cbbb52854812417da2325b27e509812d: Neither blocks nor log segments found. Creating new log.
I20260809 12:10:39.826723 8105 log.cc:824] T 00000000000000000000000000000000 P cbbb52854812417da2325b27e509812d: Log is configured to *not* fsync() on all Append() calls
I20260809 12:10:39.829587 8105 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P cbbb52854812417da2325b27e509812d: No bootstrap required, opened a new log
I20260809 12:10:39.836551 8105 raft_consensus.cc:348] T 00000000000000000000000000000000 P cbbb52854812417da2325b27e509812d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9cd5d41083d147f788d83a438b6f7fc2" member_type: VOTER last_known_addr { host: "127.0.222.62" port: 41983 } } peers { permanent_uuid: "cbbb52854812417da2325b27e509812d" member_type: VOTER last_known_addr { host: "127.0.222.61" port: 32831 } }
I20260809 12:10:39.836889 8105 raft_consensus.cc:374] T 00000000000000000000000000000000 P cbbb52854812417da2325b27e509812d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260809 12:10:39.837006 8105 raft_consensus.cc:729] T 00000000000000000000000000000000 P cbbb52854812417da2325b27e509812d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: cbbb52854812417da2325b27e509812d, State: Initialized, Role: FOLLOWER
I20260809 12:10:39.837652 8105 consensus_queue.cc:260] T 00000000000000000000000000000000 P cbbb52854812417da2325b27e509812d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9cd5d41083d147f788d83a438b6f7fc2" member_type: VOTER last_known_addr { host: "127.0.222.62" port: 41983 } } peers { permanent_uuid: "cbbb52854812417da2325b27e509812d" member_type: VOTER last_known_addr { host: "127.0.222.61" port: 32831 } }
I20260809 12:10:39.838606 8112 sys_catalog.cc:455] T 00000000000000000000000000000000 P cbbb52854812417da2325b27e509812d [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9cd5d41083d147f788d83a438b6f7fc2" member_type: VOTER last_known_addr { host: "127.0.222.62" port: 41983 } } peers { permanent_uuid: "cbbb52854812417da2325b27e509812d" member_type: VOTER last_known_addr { host: "127.0.222.61" port: 32831 } } }
I20260809 12:10:39.838946 8112 sys_catalog.cc:458] T 00000000000000000000000000000000 P cbbb52854812417da2325b27e509812d [sys.catalog]: This master's current role is: FOLLOWER
I20260809 12:10:39.839865 8105 sys_catalog.cc:564] T 00000000000000000000000000000000 P cbbb52854812417da2325b27e509812d [sys.catalog]: configured and running, proceeding with master startup.
W20260809 12:10:39.849433 8125 catalog_manager.cc:1558] T 00000000000000000000000000000000 P cbbb52854812417da2325b27e509812d: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260809 12:10:39.849622 8125 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260809 12:10:39.853266 8034 master_runner.cc:428] Detected that this master cbbb52854812417da2325b27e509812d is joining an existing cluster
I20260809 12:10:39.853358 8034 master_runner.cc:432] Initiating AddMaster RPC to add 127.0.222.61:32831
I20260809 12:10:39.888465 8109 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 12:10:39.861359 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33034 (local address 127.0.222.62:41983)
0809 12:10:39.861572 (+ 213us) server_negotiation.cc:206] Beginning negotiation
0809 12:10:39.861579 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0809 12:10:39.868872 (+ 7293us) server_negotiation.cc:407] Connection header received
0809 12:10:39.869533 (+ 661us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 12:10:39.869539 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 12:10:39.869636 (+ 97us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 12:10:39.869716 (+ 80us) server_negotiation.cc:226] Negotiated authn=SASL
0809 12:10:39.871498 (+ 1782us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 12:10:39.872669 (+ 1171us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 12:10:39.875144 (+ 2475us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 12:10:39.875887 (+ 743us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 12:10:39.876031 (+ 144us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 12:10:39.878946 (+ 2915us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 12:10:39.878953 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 12:10:39.878979 (+ 26us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0809 12:10:39.879023 (+ 44us) server_negotiation.cc:922] calling sasl_server_start()
0809 12:10:39.882657 (+ 3634us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 12:10:39.885534 (+ 2877us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 12:10:39.885542 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 12:10:39.885549 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0809 12:10:39.885813 (+ 264us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 12:10:39.886413 (+ 600us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 12:10:39.886420 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 12:10:39.886426 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0809 12:10:39.886838 (+ 412us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 12:10:39.887016 (+ 178us) server_negotiation.cc:1035] Waiting for connection context
0809 12:10:39.887763 (+ 747us) server_negotiation.cc:299] Negotiation successful
0809 12:10:39.888052 (+ 289us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":82}
I20260809 12:10:39.889055 8132 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 12:10:39.861405 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.222.62:41983 (local address 127.0.0.1:33034)
0809 12:10:39.868699 (+ 7294us) negotiation.cc:107] Waiting for socket to connect
0809 12:10:39.868730 (+ 31us) client_negotiation.cc:174] Beginning negotiation
0809 12:10:39.869342 (+ 612us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 12:10:39.869978 (+ 636us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 12:10:39.869983 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 12:10:39.870730 (+ 747us) client_negotiation.cc:189] Negotiated authn=SASL
0809 12:10:39.871230 (+ 500us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 12:10:39.871245 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 12:10:39.872866 (+ 1621us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 12:10:39.872874 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 12:10:39.874890 (+ 2016us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 12:10:39.874907 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 12:10:39.876392 (+ 1485us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 12:10:39.876401 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 12:10:39.876602 (+ 201us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 12:10:39.877399 (+ 797us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0809 12:10:39.877423 (+ 24us) client_negotiation.cc:656] calling sasl_client_start()
0809 12:10:39.878691 (+ 1268us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0809 12:10:39.882883 (+ 4192us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0809 12:10:39.882888 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0809 12:10:39.882892 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0809 12:10:39.885375 (+ 2483us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0809 12:10:39.885992 (+ 617us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0809 12:10:39.886000 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0809 12:10:39.886007 (+ 7us) client_negotiation.cc:760] calling sasl_client_step()
0809 12:10:39.886258 (+ 251us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0809 12:10:39.887104 (+ 846us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0809 12:10:39.887112 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0809 12:10:39.887387 (+ 275us) client_negotiation.cc:769] Sending connection context
0809 12:10:39.888366 (+ 979us) client_negotiation.cc:240] Negotiation successful
0809 12:10:39.888697 (+ 331us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":7105,"mutex_wait_us":5,"thread_start_us":9743,"threads_started":1}
I20260809 12:10:39.890107 8108 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 12:10:39.861253 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46508 (local address 127.0.222.61:32831)
0809 12:10:39.861489 (+ 236us) server_negotiation.cc:206] Beginning negotiation
0809 12:10:39.861495 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0809 12:10:39.871608 (+ 10113us) server_negotiation.cc:407] Connection header received
0809 12:10:39.871780 (+ 172us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 12:10:39.871785 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 12:10:39.871847 (+ 62us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 12:10:39.871931 (+ 84us) server_negotiation.cc:226] Negotiated authn=SASL
0809 12:10:39.873307 (+ 1376us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 12:10:39.874490 (+ 1183us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 12:10:39.876723 (+ 2233us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 12:10:39.878027 (+ 1304us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 12:10:39.878142 (+ 115us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 12:10:39.881680 (+ 3538us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 12:10:39.881689 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 12:10:39.881696 (+ 7us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0809 12:10:39.881735 (+ 39us) server_negotiation.cc:922] calling sasl_server_start()
0809 12:10:39.885926 (+ 4191us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 12:10:39.886858 (+ 932us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 12:10:39.886864 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 12:10:39.886868 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0809 12:10:39.887033 (+ 165us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 12:10:39.887605 (+ 572us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 12:10:39.887610 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 12:10:39.887613 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0809 12:10:39.887971 (+ 358us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 12:10:39.888122 (+ 151us) server_negotiation.cc:1035] Waiting for connection context
0809 12:10:39.889573 (+ 1451us) server_negotiation.cc:299] Negotiation successful
0809 12:10:39.889793 (+ 220us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":38,"server-negotiator.queue_time_us":78}
I20260809 12:10:39.891108 8131 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 12:10:39.861928 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.222.61:32831 (local address 127.0.0.1:46508)
0809 12:10:39.871503 (+ 9575us) negotiation.cc:107] Waiting for socket to connect
0809 12:10:39.871518 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0809 12:10:39.871658 (+ 140us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 12:10:39.872083 (+ 425us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 12:10:39.872087 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 12:10:39.872755 (+ 668us) client_negotiation.cc:189] Negotiated authn=SASL
0809 12:10:39.873125 (+ 370us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 12:10:39.873151 (+ 26us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 12:10:39.874700 (+ 1549us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 12:10:39.874706 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 12:10:39.876574 (+ 1868us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 12:10:39.876589 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 12:10:39.878574 (+ 1985us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 12:10:39.878584 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 12:10:39.878873 (+ 289us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 12:10:39.879610 (+ 737us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0809 12:10:39.879932 (+ 322us) client_negotiation.cc:656] calling sasl_client_start()
0809 12:10:39.881493 (+ 1561us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0809 12:10:39.886148 (+ 4655us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0809 12:10:39.886152 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0809 12:10:39.886156 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0809 12:10:39.886716 (+ 560us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0809 12:10:39.887164 (+ 448us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0809 12:10:39.887170 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0809 12:10:39.887173 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0809 12:10:39.887475 (+ 302us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0809 12:10:39.888195 (+ 720us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0809 12:10:39.888199 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0809 12:10:39.888352 (+ 153us) client_negotiation.cc:769] Sending connection context
0809 12:10:39.890258 (+ 1906us) client_negotiation.cc:240] Negotiation successful
0809 12:10:39.890530 (+ 272us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":9427,"mutex_wait_us":3,"spinlock_wait_cycles":30592,"thread_start_us":6962,"threads_started":1}
WARNING: no policy specified for oryx/127.0.222.60@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.0.222.60@KRBTEST.COM" created.
Aug 09 12:10:39 dist-test-slave-2w3w krb5kdc[7502](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786277437, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.222.62@KRBTEST.COM for oryx/127.0.222.61@KRBTEST.COM
I20260809 12:10:39.915729 8108 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 12:10:39.897032 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46520 (local address 127.0.222.61:32831)
0809 12:10:39.897220 (+ 188us) server_negotiation.cc:206] Beginning negotiation
0809 12:10:39.897226 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0809 12:10:39.897909 (+ 683us) server_negotiation.cc:407] Connection header received
0809 12:10:39.898296 (+ 387us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 12:10:39.898304 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 12:10:39.898432 (+ 128us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 12:10:39.898616 (+ 184us) server_negotiation.cc:226] Negotiated authn=SASL
0809 12:10:39.900156 (+ 1540us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 12:10:39.901337 (+ 1181us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 12:10:39.903554 (+ 2217us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 12:10:39.904285 (+ 731us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 12:10:39.904421 (+ 136us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 12:10:39.909487 (+ 5066us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 12:10:39.909494 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 12:10:39.909499 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0809 12:10:39.909539 (+ 40us) server_negotiation.cc:922] calling sasl_server_start()
0809 12:10:39.912730 (+ 3191us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 12:10:39.913492 (+ 762us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 12:10:39.913496 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 12:10:39.913499 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0809 12:10:39.913630 (+ 131us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 12:10:39.914038 (+ 408us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 12:10:39.914043 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 12:10:39.914050 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0809 12:10:39.914360 (+ 310us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 12:10:39.914521 (+ 161us) server_negotiation.cc:1035] Waiting for connection context
0809 12:10:39.915080 (+ 559us) server_negotiation.cc:299] Negotiation successful
0809 12:10:39.915334 (+ 254us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":77}
I20260809 12:10:39.915802 8134 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 12:10:39.897038 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.222.61:32831 (local address 127.0.0.1:46520)
0809 12:10:39.897762 (+ 724us) negotiation.cc:107] Waiting for socket to connect
0809 12:10:39.897814 (+ 52us) client_negotiation.cc:174] Beginning negotiation
0809 12:10:39.898051 (+ 237us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 12:10:39.898692 (+ 641us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 12:10:39.898701 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 12:10:39.899647 (+ 946us) client_negotiation.cc:189] Negotiated authn=SASL
0809 12:10:39.899969 (+ 322us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 12:10:39.899983 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 12:10:39.901525 (+ 1542us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 12:10:39.901534 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 12:10:39.903389 (+ 1855us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 12:10:39.903409 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 12:10:39.904484 (+ 1075us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 12:10:39.904493 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 12:10:39.904765 (+ 272us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 12:10:39.905601 (+ 836us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0809 12:10:39.905636 (+ 35us) client_negotiation.cc:656] calling sasl_client_start()
0809 12:10:39.909241 (+ 3605us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0809 12:10:39.912889 (+ 3648us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0809 12:10:39.912904 (+ 15us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0809 12:10:39.912912 (+ 8us) client_negotiation.cc:760] calling sasl_client_step()
0809 12:10:39.913344 (+ 432us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0809 12:10:39.913758 (+ 414us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0809 12:10:39.913764 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0809 12:10:39.913768 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0809 12:10:39.913914 (+ 146us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0809 12:10:39.914502 (+ 588us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0809 12:10:39.914512 (+ 10us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0809 12:10:39.914715 (+ 203us) client_negotiation.cc:769] Sending connection context
0809 12:10:39.914927 (+ 212us) client_negotiation.cc:240] Negotiation successful
0809 12:10:39.915323 (+ 396us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":480,"thread_start_us":318,"threads_started":1}
I20260809 12:10:39.917847 7547 catalog_manager.cc:6979] Initiating ChangeConfig request to add master 127.0.222.61:32831: tablet_id: "00000000000000000000000000000000"
type: ADD_PEER
server {
permanent_uuid: "cbbb52854812417da2325b27e509812d"
member_type: NON_VOTER
last_known_addr {
host: "127.0.222.61"
port: 32831
}
attrs {
promote: true
}
}
dest_uuid: "9cd5d41083d147f788d83a438b6f7fc2"
cas_config_opid_index: -1
I20260809 12:10:39.919166 7547 consensus_queue.cc:237] T 00000000000000000000000000000000 P 9cd5d41083d147f788d83a438b6f7fc2 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "9cd5d41083d147f788d83a438b6f7fc2" member_type: VOTER last_known_addr { host: "127.0.222.62" port: 41983 } } peers { permanent_uuid: "cbbb52854812417da2325b27e509812d" member_type: NON_VOTER last_known_addr { host: "127.0.222.61" port: 32831 } attrs { promote: true } }
I20260809 12:10:39.927934 8079 raft_consensus.cc:3037] T 00000000000000000000000000000000 P cbbb52854812417da2325b27e509812d [term 0 FOLLOWER]: Advancing to term 1
I20260809 12:10:39.928800 8138 raft_consensus.cc:2932] T 00000000000000000000000000000000 P 9cd5d41083d147f788d83a438b6f7fc2 [term 1 LEADER]: Committing config change with OpId 1.5: config changed from index -1 to 5, NON_VOTER cbbb52854812417da2325b27e509812d (127.0.222.61) added. New config: { opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "9cd5d41083d147f788d83a438b6f7fc2" member_type: VOTER last_known_addr { host: "127.0.222.62" port: 41983 } } peers { permanent_uuid: "cbbb52854812417da2325b27e509812d" member_type: NON_VOTER last_known_addr { host: "127.0.222.61" port: 32831 } attrs { promote: true } } }
I20260809 12:10:39.932158 8079 raft_consensus.cc:1261] T 00000000000000000000000000000000 P cbbb52854812417da2325b27e509812d [term 1 FOLLOWER]: Refusing update from remote peer 9cd5d41083d147f788d83a438b6f7fc2: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20260809 12:10:39.933357 8137 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 9cd5d41083d147f788d83a438b6f7fc2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "cbbb52854812417da2325b27e509812d" member_type: NON_VOTER last_known_addr { host: "127.0.222.61" port: 32831 } attrs { promote: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 0, Time since last communication: 0.000s
I20260809 12:10:39.935702 8138 catalog_manager.cc:6983] Successfully completed master ChangeConfig request to add master 127.0.222.61:32831
I20260809 12:10:39.936877 8034 master.cc:537] Master@127.0.222.61:32831 shutting down...
W20260809 12:10:39.938696 7535 proxy.cc:239] Call had error, refreshing address and retrying: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on Master
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.0.222.60 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskgH5Pjm/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786277382073938-888-0/krb5kdc/oryx_127.0.222.60.keytab.
Entry for principal oryx/127.0.222.60 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskgH5Pjm/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786277382073938-888-0/krb5kdc/oryx_127.0.222.60.keytab.
I20260809 12:10:39.941972 888 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskgH5Pjm/build/asan/bin/kudu
/tmp/dist-test-taskgH5Pjm/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskgH5Pjm/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786277382073938-888-0/minicluster-data/master-2/wal
--fs_data_dirs=/tmp/dist-test-taskgH5Pjm/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786277382073938-888-0/minicluster-data/master-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskgH5Pjm/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786277382073938-888-0/minicluster-data/master-2/logs
--server_dump_info_path=/tmp/dist-test-taskgH5Pjm/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786277382073938-888-0/minicluster-data/master-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.222.60:39543
--webserver_interface=127.0.222.60
--webserver_port=0
--keytab_file=/tmp/dist-test-taskgH5Pjm/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786277382073938-888-0/krb5kdc/oryx_127.0.222.60.keytab
--principal=oryx/127.0.222.60
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.222.20:38069
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.222.62:41983,127.0.222.61:32831,127.0.222.60:39543
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskgH5Pjm/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786277382073938-888-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskgH5Pjm/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786277382073938-888-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskgH5Pjm/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786277382073938-888-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260809 12:10:39.942279 8140 sys_catalog.cc:455] T 00000000000000000000000000000000 P 9cd5d41083d147f788d83a438b6f7fc2 [sys.catalog]: SysCatalogTable state changed. Reason: Config change replication complete. Latest consensus state: current_term: 1 leader_uuid: "9cd5d41083d147f788d83a438b6f7fc2" committed_config { opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "9cd5d41083d147f788d83a438b6f7fc2" member_type: VOTER last_known_addr { host: "127.0.222.62" port: 41983 } } peers { permanent_uuid: "cbbb52854812417da2325b27e509812d" member_type: NON_VOTER last_known_addr { host: "127.0.222.61" port: 32831 } attrs { promote: true } } }
I20260809 12:10:39.942633 8140 sys_catalog.cc:458] T 00000000000000000000000000000000 P 9cd5d41083d147f788d83a438b6f7fc2 [sys.catalog]: This master's current role is: LEADER
W20260809 12:10:39.943099 7535 consensus_peers.cc:487] T 00000000000000000000000000000000 P 9cd5d41083d147f788d83a438b6f7fc2 -> Peer cbbb52854812417da2325b27e509812d (127.0.222.61:32831): Couldn't send request to peer cbbb52854812417da2325b27e509812d. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on Master. This is attempt 1: this message will repeat every 5th retry.
I20260809 12:10:39.957248 8034 raft_consensus.cc:2229] T 00000000000000000000000000000000 P cbbb52854812417da2325b27e509812d [term 1 FOLLOWER]: Raft consensus shutting down.
I20260809 12:10:39.957726 8034 raft_consensus.cc:2258] T 00000000000000000000000000000000 P cbbb52854812417da2325b27e509812d [term 1 FOLLOWER]: Raft consensus is shut down!
I20260809 12:10:39.957953 8034 tablet_replica.cc:330] T 00000000000000000000000000000000 P cbbb52854812417da2325b27e509812d: stopping tablet replica
I20260809 12:10:39.975127 8034 master.cc:552] Master@127.0.222.61:32831 shutdown complete.
I20260809 12:10:39.975286 8034 master_runner.cc:305] Clearing existing system tablet
I20260809 12:10:39.975814 8034 fs_manager.cc:359] Metadata directory not provided
I20260809 12:10:39.975944 8034 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260809 12:10:39.978523 8034 fs_manager.cc:709] Time spent opening directory manager: real 0.002s user 0.003s sys 0.000s
I20260809 12:10:39.980360 8147 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260809 12:10:39.980700 8034 fs_report.cc:383] FS layout report
--------------------
wal directory:
metadata directory:
1 data directories: /tmp/dist-test-taskgH5Pjm/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786277382073938-888-0/minicluster-data/master-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260809 12:10:39.981010 8034 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.000s sys 0.001s
I20260809 12:10:39.981112 8034 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskgH5Pjm/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786277382073938-888-0/minicluster-data/master-1/data,/tmp/dist-test-taskgH5Pjm/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786277382073938-888-0/minicluster-data/master-1/wal
uuid: "cbbb52854812417da2325b27e509812d"
format_stamp: "Formatted at 2026-08-09 12:10:39 on dist-test-slave-2w3w"
server_key: "0c86a30d73b4f22d89209bc662d6bac9"
server_key_iv: "a23dca56137a4a133a2d91161ad6e5a1"
server_key_version: "encryptionkey@0"
I20260809 12:10:39.985042 8034 ts_tablet_manager.cc:1905] T 00000000000000000000000000000000 P cbbb52854812417da2325b27e509812d: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260809 12:10:39.990217 8034 ts_tablet_manager.cc:1918] T 00000000000000000000000000000000 P cbbb52854812417da2325b27e509812d: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId unknown
I20260809 12:10:39.990410 8034 log.cc:1196] T 00000000000000000000000000000000 P cbbb52854812417da2325b27e509812d: Deleting WAL directory at /tmp/dist-test-taskgH5Pjm/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786277382073938-888-0/minicluster-data/master-1/wal/wals/00000000000000000000000000000000
I20260809 12:10:39.990845 8034 ts_tablet_manager.cc:1939] T 00000000000000000000000000000000 P cbbb52854812417da2325b27e509812d: Deleting consensus metadata
I20260809 12:10:39.991129 8034 master_runner.cc:315] Copying system tablet from 127.0.222.62:41983
I20260809 12:10:39.994230 8034 tablet_copy_client.cc:316] T 00000000000000000000000000000000 P cbbb52854812417da2325b27e509812d: tablet copy: Beginning tablet copy session from remote peer at address 127.0.222.62:41983
I20260809 12:10:40.007979 8153 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 12:10:39.995223 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.222.62:41983 (local address 127.0.0.1:33046)
0809 12:10:39.995902 (+ 679us) negotiation.cc:107] Waiting for socket to connect
0809 12:10:39.995928 (+ 26us) client_negotiation.cc:174] Beginning negotiation
0809 12:10:39.996104 (+ 176us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 12:10:39.996508 (+ 404us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 12:10:39.996513 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 12:10:39.996941 (+ 428us) client_negotiation.cc:189] Negotiated authn=SASL
0809 12:10:39.997157 (+ 216us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 12:10:39.997167 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 12:10:39.998264 (+ 1097us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 12:10:39.998270 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 12:10:39.999298 (+ 1028us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 12:10:39.999311 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 12:10:40.000146 (+ 835us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 12:10:40.000152 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 12:10:40.000382 (+ 230us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 12:10:40.001051 (+ 669us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0809 12:10:40.001070 (+ 19us) client_negotiation.cc:656] calling sasl_client_start()
0809 12:10:40.001802 (+ 732us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0809 12:10:40.005348 (+ 3546us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0809 12:10:40.005354 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0809 12:10:40.005359 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0809 12:10:40.005730 (+ 371us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0809 12:10:40.006155 (+ 425us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0809 12:10:40.006161 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0809 12:10:40.006165 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0809 12:10:40.006334 (+ 169us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0809 12:10:40.006898 (+ 564us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0809 12:10:40.006903 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0809 12:10:40.007126 (+ 223us) client_negotiation.cc:769] Sending connection context
0809 12:10:40.007306 (+ 180us) client_negotiation.cc:240] Negotiation successful
0809 12:10:40.007551 (+ 245us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":549,"thread_start_us":429,"threads_started":1}
I20260809 12:10:40.008890 8109 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 12:10:39.995263 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33046 (local address 127.0.222.62:41983)
0809 12:10:39.995495 (+ 232us) server_negotiation.cc:206] Beginning negotiation
0809 12:10:39.995505 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0809 12:10:39.996033 (+ 528us) server_negotiation.cc:407] Connection header received
0809 12:10:39.996255 (+ 222us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 12:10:39.996271 (+ 16us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 12:10:39.996355 (+ 84us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 12:10:39.996472 (+ 117us) server_negotiation.cc:226] Negotiated authn=SASL
0809 12:10:39.997335 (+ 863us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 12:10:39.998097 (+ 762us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 12:10:39.999447 (+ 1350us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 12:10:39.999986 (+ 539us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 12:10:40.000131 (+ 145us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 12:10:40.002021 (+ 1890us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 12:10:40.002034 (+ 13us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 12:10:40.002038 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0809 12:10:40.002070 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0809 12:10:40.005197 (+ 3127us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 12:10:40.005878 (+ 681us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 12:10:40.005885 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 12:10:40.005888 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0809 12:10:40.006035 (+ 147us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 12:10:40.006458 (+ 423us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 12:10:40.006465 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 12:10:40.006468 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0809 12:10:40.006763 (+ 295us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 12:10:40.006934 (+ 171us) server_negotiation.cc:1035] Waiting for connection context
0809 12:10:40.008356 (+ 1422us) server_negotiation.cc:299] Negotiation successful
0809 12:10:40.008625 (+ 269us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":68}
I20260809 12:10:40.009711 7569 tablet_copy_service.cc:140] P 9cd5d41083d147f788d83a438b6f7fc2: Received BeginTabletCopySession request for tablet 00000000000000000000000000000000 from peer cbbb52854812417da2325b27e509812d ({username='oryx', principal='oryx/127.0.222.61@KRBTEST.COM'} at 127.0.0.1:33046)
I20260809 12:10:40.010098 7569 tablet_copy_service.cc:161] P 9cd5d41083d147f788d83a438b6f7fc2: Beginning new tablet copy session on tablet 00000000000000000000000000000000 from peer cbbb52854812417da2325b27e509812d at {username='oryx', principal='oryx/127.0.222.61@KRBTEST.COM'} at 127.0.0.1:33046: session id = cbbb52854812417da2325b27e509812d-00000000000000000000000000000000
I20260809 12:10:40.013484 7569 tablet_copy_source_session.cc:205] T 00000000000000000000000000000000 P 9cd5d41083d147f788d83a438b6f7fc2: Tablet Copy: opened 0 blocks and 1 log segments
I20260809 12:10:40.015959 8034 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total
I20260809 12:10:40.020987 8034 tablet_copy_client.cc:791] T 00000000000000000000000000000000 P cbbb52854812417da2325b27e509812d: tablet copy: Starting download of 0 data blocks...
I20260809 12:10:40.021351 8034 tablet_copy_client.cc:655] T 00000000000000000000000000000000 P cbbb52854812417da2325b27e509812d: tablet copy: Starting download of 1 WAL segments...
I20260809 12:10:40.024663 8034 tablet_copy_client.cc:531] T 00000000000000000000000000000000 P cbbb52854812417da2325b27e509812d: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20260809 12:10:40.028057 7569 tablet_copy_service.cc:342] P 9cd5d41083d147f788d83a438b6f7fc2: Request end of tablet copy session cbbb52854812417da2325b27e509812d-00000000000000000000000000000000 received from {username='oryx', principal='oryx/127.0.222.61@KRBTEST.COM'} at 127.0.0.1:33046
I20260809 12:10:40.028316 7569 tablet_copy_service.cc:433] P 9cd5d41083d147f788d83a438b6f7fc2: ending tablet copy session cbbb52854812417da2325b27e509812d-00000000000000000000000000000000 on tablet 00000000000000000000000000000000 with peer cbbb52854812417da2325b27e509812d
I20260809 12:10:40.039630 8034 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260809 12:10:40.042289 8154 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 12:10:40.052783 8157 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 12:10:40.053287 8155 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260809 12:10:40.055872 8034 server_base.cc:1034] running on GCE node
I20260809 12:10:40.056202 8034 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260809 12:10:40.056870 8034 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260809 12:10:40.058102 8034 hybrid_clock.cc:648] HybridClock initialized: now 1786277440058013 us; error 63 us; skew 500 ppm
I20260809 12:10:40.058336 8034 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 09 12:10:40 dist-test-slave-2w3w krb5kdc[7502](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786277440, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.222.61@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260809 12:10:40.060451 8034 init.cc:377] Logged in from keytab as oryx/127.0.222.61@KRBTEST.COM (short username oryx)
I20260809 12:10:40.061925 8034 webserver.cc:460] Webserver started at http://127.0.222.61:38023/ using document root <none> and password file <none>
I20260809 12:10:40.062271 8034 fs_manager.cc:359] Metadata directory not provided
I20260809 12:10:40.062366 8034 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260809 12:10:40.065040 8034 fs_manager.cc:709] Time spent opening directory manager: real 0.002s user 0.002s sys 0.000s
I20260809 12:10:40.066807 8165 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260809 12:10:40.067332 8034 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260809 12:10:40.067492 8034 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskgH5Pjm/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786277382073938-888-0/minicluster-data/master-1/data,/tmp/dist-test-taskgH5Pjm/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786277382073938-888-0/minicluster-data/master-1/wal
uuid: "cbbb52854812417da2325b27e509812d"
format_stamp: "Formatted at 2026-08-09 12:10:39 on dist-test-slave-2w3w"
server_key: "0c86a30d73b4f22d89209bc662d6bac9"
server_key_iv: "a23dca56137a4a133a2d91161ad6e5a1"
server_key_version: "encryptionkey@0"
I20260809 12:10:40.067651 8034 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskgH5Pjm/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786277382073938-888-0/minicluster-data/master-1/wal
metadata directory: /tmp/dist-test-taskgH5Pjm/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786277382073938-888-0/minicluster-data/master-1/wal
1 data directories: /tmp/dist-test-taskgH5Pjm/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786277382073938-888-0/minicluster-data/master-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260809 12:10:40.074884 8034 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260809 12:10:40.078752 8034 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260809 12:10:40.109408 8034 rpc_server.cc:307] RPC server started. Bound to: 127.0.222.61:32831
I20260809 12:10:40.109439 8220 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.222.61:32831 every 8 connection(s)
I20260809 12:10:40.110366 8034 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskgH5Pjm/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786277382073938-888-0/minicluster-data/master-1/data/info.pb
I20260809 12:10:40.117694 8221 sys_catalog.cc:263] Verifying existing consensus state
I20260809 12:10:40.120360 8221 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P cbbb52854812417da2325b27e509812d: Bootstrap starting.
I20260809 12:10:40.134763 8221 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P cbbb52854812417da2325b27e509812d: Bootstrap replayed 1/1 log segments. Stats: ops{read=5 overwritten=0 applied=5 ignored=0} inserts{seen=3 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260809 12:10:40.135360 8221 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P cbbb52854812417da2325b27e509812d: Bootstrap complete.
I20260809 12:10:40.136274 8221 raft_consensus.cc:348] T 00000000000000000000000000000000 P cbbb52854812417da2325b27e509812d [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "9cd5d41083d147f788d83a438b6f7fc2" member_type: VOTER last_known_addr { host: "127.0.222.62" port: 41983 } } peers { permanent_uuid: "cbbb52854812417da2325b27e509812d" member_type: NON_VOTER last_known_addr { host: "127.0.222.61" port: 32831 } attrs { promote: true } }
I20260809 12:10:40.136516 8221 raft_consensus.cc:729] T 00000000000000000000000000000000 P cbbb52854812417da2325b27e509812d [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: cbbb52854812417da2325b27e509812d, State: Initialized, Role: LEARNER
I20260809 12:10:40.136744 8221 consensus_queue.cc:260] T 00000000000000000000000000000000 P cbbb52854812417da2325b27e509812d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 5, Last appended: 1.5, Last appended by leader: 5, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "9cd5d41083d147f788d83a438b6f7fc2" member_type: VOTER last_known_addr { host: "127.0.222.62" port: 41983 } } peers { permanent_uuid: "cbbb52854812417da2325b27e509812d" member_type: NON_VOTER last_known_addr { host: "127.0.222.61" port: 32831 } attrs { promote: true } }
I20260809 12:10:40.137598 8225 sys_catalog.cc:455] T 00000000000000000000000000000000 P cbbb52854812417da2325b27e509812d [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 committed_config { opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "9cd5d41083d147f788d83a438b6f7fc2" member_type: VOTER last_known_addr { host: "127.0.222.62" port: 41983 } } peers { permanent_uuid: "cbbb52854812417da2325b27e509812d" member_type: NON_VOTER last_known_addr { host: "127.0.222.61" port: 32831 } attrs { promote: true } } }
I20260809 12:10:40.137957 8221 sys_catalog.cc:564] T 00000000000000000000000000000000 P cbbb52854812417da2325b27e509812d [sys.catalog]: configured and running, proceeding with master startup.
I20260809 12:10:40.137969 8225 sys_catalog.cc:458] T 00000000000000000000000000000000 P cbbb52854812417da2325b27e509812d [sys.catalog]: This master's current role is: LEARNER
I20260809 12:10:40.147116 8236 catalog_manager.cc:1259] Loaded cluster ID: 282dd341a1ea4ea895575dd472644e9f
I20260809 12:10:40.147234 8236 catalog_manager.cc:1552] T 00000000000000000000000000000000 P cbbb52854812417da2325b27e509812d: loading cluster ID for follower catalog manager: success
I20260809 12:10:40.150096 8236 catalog_manager.cc:1574] T 00000000000000000000000000000000 P cbbb52854812417da2325b27e509812d: acquiring CA information for follower catalog manager: success
I20260809 12:10:40.152190 8236 catalog_manager.cc:1602] T 00000000000000000000000000000000 P cbbb52854812417da2325b27e509812d: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260809 12:10:40.262409 8027 heartbeater.cc:503] Master 127.0.222.62:41983 was elected leader, sending a full tablet report...
W20260809 12:10:40.377444 8145 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260809 12:10:40.377846 8145 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260809 12:10:40.378089 8145 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260809 12:10:40.387984 8145 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260809 12:10:40.388206 8145 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260809 12:10:40.388361 8145 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260809 12:10:40.388583 8145 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260809 12:10:40.388770 8145 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260809 12:10:40.402700 8145 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.222.20:38069
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskgH5Pjm/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786277382073938-888-0/minicluster-data/master-2/data
--fs_wal_dir=/tmp/dist-test-taskgH5Pjm/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786277382073938-888-0/minicluster-data/master-2/wal
--ipki_ca_key_size=768
--master_addresses=127.0.222.62:41983,127.0.222.61:32831,127.0.222.60:39543
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskgH5Pjm/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786277382073938-888-0/krb5kdc/oryx_127.0.222.60.keytab
--principal=oryx/127.0.222.60
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.222.60:39543
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskgH5Pjm/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786277382073938-888-0/minicluster-data/master-2/data/info.pb
--webserver_interface=127.0.222.60
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskgH5Pjm/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786277382073938-888-0/minicluster-data/master-2/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 09 Aug 2026 11:06:41 UTC on ccycloud.quasar-bwoogy.root.comops.site
build id 4575991
ASAN enabled
I20260809 12:10:40.404554 8145 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260809 12:10:40.406031 8145 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260809 12:10:40.422320 8242 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 12:10:40.422585 8243 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 12:10:40.422901 8246 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260809 12:10:40.424401 8145 server_base.cc:1034] running on GCE node
I20260809 12:10:40.425289 8145 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260809 12:10:40.426885 8145 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260809 12:10:40.428175 8145 hybrid_clock.cc:648] HybridClock initialized: now 1786277440428063 us; error 64 us; skew 500 ppm
I20260809 12:10:40.428701 8145 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 09 12:10:40 dist-test-slave-2w3w krb5kdc[7502](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786277440, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.222.60@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260809 12:10:40.433086 8145 init.cc:377] Logged in from keytab as oryx/127.0.222.60@KRBTEST.COM (short username oryx)
I20260809 12:10:40.435082 8145 webserver.cc:460] Webserver started at http://127.0.222.60:46871/ using document root <none> and password file <none>
I20260809 12:10:40.435963 8145 fs_manager.cc:359] Metadata directory not provided
I20260809 12:10:40.436208 8145 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260809 12:10:40.436599 8145 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260809 12:10:40.440776 8145 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskgH5Pjm/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786277382073938-888-0/minicluster-data/master-2/data/instance:
uuid: "e2a7b439c3084cd680be92ac6d1925db"
format_stamp: "Formatted at 2026-08-09 12:10:40 on dist-test-slave-2w3w"
server_key: "67dabfc5aa05d4447bd46bb4b26ce89c"
server_key_iv: "ca9f445d4b4bb45f49d87f7e3864c8d0"
server_key_version: "encryptionkey@0"
I20260809 12:10:40.441757 8145 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskgH5Pjm/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786277382073938-888-0/minicluster-data/master-2/wal/instance:
uuid: "e2a7b439c3084cd680be92ac6d1925db"
format_stamp: "Formatted at 2026-08-09 12:10:40 on dist-test-slave-2w3w"
server_key: "67dabfc5aa05d4447bd46bb4b26ce89c"
server_key_iv: "ca9f445d4b4bb45f49d87f7e3864c8d0"
server_key_version: "encryptionkey@0"
I20260809 12:10:40.447906 8145 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.008s sys 0.000s
I20260809 12:10:40.452158 8253 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260809 12:10:40.453722 8145 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.005s sys 0.000s
I20260809 12:10:40.453866 8145 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskgH5Pjm/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786277382073938-888-0/minicluster-data/master-2/data,/tmp/dist-test-taskgH5Pjm/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786277382073938-888-0/minicluster-data/master-2/wal
uuid: "e2a7b439c3084cd680be92ac6d1925db"
format_stamp: "Formatted at 2026-08-09 12:10:40 on dist-test-slave-2w3w"
server_key: "67dabfc5aa05d4447bd46bb4b26ce89c"
server_key_iv: "ca9f445d4b4bb45f49d87f7e3864c8d0"
server_key_version: "encryptionkey@0"
I20260809 12:10:40.454046 8145 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskgH5Pjm/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786277382073938-888-0/minicluster-data/master-2/wal
metadata directory: /tmp/dist-test-taskgH5Pjm/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786277382073938-888-0/minicluster-data/master-2/wal
1 data directories: /tmp/dist-test-taskgH5Pjm/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786277382073938-888-0/minicluster-data/master-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260809 12:10:40.468392 8145 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260809 12:10:40.473428 8145 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260809 12:10:40.473718 8145 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260809 12:10:40.510654 8145 rpc_server.cc:307] RPC server started. Bound to: 127.0.222.60:39543
I20260809 12:10:40.510653 8305 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.222.60:39543 every 8 connection(s)
I20260809 12:10:40.512588 8145 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskgH5Pjm/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786277382073938-888-0/minicluster-data/master-2/data/info.pb
I20260809 12:10:40.517629 8306 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260809 12:10:40.519688 888 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskgH5Pjm/build/asan/bin/kudu as pid 8145
I20260809 12:10:40.532267 8306 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.222.62" port: 41983 } has no permanent_uuid. Determining permanent_uuid...
I20260809 12:10:40.535526 8312 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 12:10:40.525564 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46528 (local address 127.0.222.61:32831)
0809 12:10:40.526676 (+ 1112us) server_negotiation.cc:206] Beginning negotiation
0809 12:10:40.526682 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0809 12:10:40.527113 (+ 431us) server_negotiation.cc:407] Connection header received
0809 12:10:40.527290 (+ 177us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 12:10:40.527295 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 12:10:40.527404 (+ 109us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 12:10:40.527616 (+ 212us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0809 12:10:40.528322 (+ 706us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 12:10:40.530527 (+ 2205us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 12:10:40.532996 (+ 2469us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 12:10:40.534177 (+ 1181us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 12:10:40.534360 (+ 183us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 12:10:40.534793 (+ 433us) server_negotiation.cc:1035] Waiting for connection context
0809 12:10:40.534936 (+ 143us) server_negotiation.cc:299] Negotiation successful
0809 12:10:40.535112 (+ 176us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":961,"thread_start_us":687,"threads_started":1}
I20260809 12:10:40.535516 8311 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 12:10:40.525397 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.222.61:32831 (local address 127.0.0.1:46528)
0809 12:10:40.526841 (+ 1444us) negotiation.cc:107] Waiting for socket to connect
0809 12:10:40.526865 (+ 24us) client_negotiation.cc:174] Beginning negotiation
0809 12:10:40.527143 (+ 278us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 12:10:40.527709 (+ 566us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 12:10:40.527716 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 12:10:40.527764 (+ 48us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0809 12:10:40.528112 (+ 348us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 12:10:40.528126 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 12:10:40.530744 (+ 2618us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 12:10:40.530754 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 12:10:40.532803 (+ 2049us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 12:10:40.532817 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 12:10:40.534426 (+ 1609us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 12:10:40.534432 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 12:10:40.534680 (+ 248us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 12:10:40.534781 (+ 101us) client_negotiation.cc:769] Sending connection context
0809 12:10:40.534913 (+ 132us) client_negotiation.cc:240] Negotiation successful
0809 12:10:40.535097 (+ 184us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1280,"thread_start_us":1157,"threads_started":1}
I20260809 12:10:40.539506 8193 raft_consensus.cc:1203] T 00000000000000000000000000000000 P cbbb52854812417da2325b27e509812d [term 1 LEARNER]: Deduplicated request from leader. Original: 0.0->[1.1-1.5] Dedup: 1.5->[]
I20260809 12:10:40.540309 8225 sys_catalog.cc:455] T 00000000000000000000000000000000 P cbbb52854812417da2325b27e509812d [sys.catalog]: SysCatalogTable state changed. Reason: New leader 9cd5d41083d147f788d83a438b6f7fc2. Latest consensus state: current_term: 1 leader_uuid: "9cd5d41083d147f788d83a438b6f7fc2" committed_config { opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "9cd5d41083d147f788d83a438b6f7fc2" member_type: VOTER last_known_addr { host: "127.0.222.62" port: 41983 } } peers { permanent_uuid: "cbbb52854812417da2325b27e509812d" member_type: NON_VOTER last_known_addr { host: "127.0.222.61" port: 32831 } attrs { promote: true } } }
I20260809 12:10:40.540681 8225 sys_catalog.cc:458] T 00000000000000000000000000000000 P cbbb52854812417da2325b27e509812d [sys.catalog]: This master's current role is: LEARNER
Aug 09 12:10:40 dist-test-slave-2w3w krb5kdc[7502](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786277437, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.0.222.60@KRBTEST.COM
Aug 09 12:10:40 dist-test-slave-2w3w krb5kdc[7502](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786277440, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.222.60@KRBTEST.COM for oryx/127.0.222.62@KRBTEST.COM
I20260809 12:10:40.559537 8310 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 12:10:40.522623 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42430 (local address 127.0.222.60:39543)
0809 12:10:40.524329 (+ 1706us) server_negotiation.cc:206] Beginning negotiation
0809 12:10:40.524345 (+ 16us) server_negotiation.cc:399] Waiting for connection header
0809 12:10:40.524439 (+ 94us) server_negotiation.cc:407] Connection header received
0809 12:10:40.528072 (+ 3633us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 12:10:40.528102 (+ 30us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 12:10:40.528631 (+ 529us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 12:10:40.529202 (+ 571us) server_negotiation.cc:226] Negotiated authn=SASL
0809 12:10:40.531761 (+ 2559us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 12:10:40.534301 (+ 2540us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 12:10:40.538728 (+ 4427us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 12:10:40.540158 (+ 1430us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 12:10:40.541360 (+ 1202us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 12:10:40.547596 (+ 6236us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 12:10:40.547606 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 12:10:40.547620 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0809 12:10:40.547669 (+ 49us) server_negotiation.cc:922] calling sasl_server_start()
0809 12:10:40.553432 (+ 5763us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 12:10:40.554335 (+ 903us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 12:10:40.554346 (+ 11us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 12:10:40.554359 (+ 13us) server_negotiation.cc:967] calling sasl_server_step()
0809 12:10:40.554627 (+ 268us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 12:10:40.555244 (+ 617us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 12:10:40.555250 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 12:10:40.555253 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0809 12:10:40.556125 (+ 872us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 12:10:40.556563 (+ 438us) server_negotiation.cc:1035] Waiting for connection context
0809 12:10:40.557150 (+ 587us) server_negotiation.cc:299] Negotiation successful
0809 12:10:40.557651 (+ 501us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1321,"thread_start_us":893,"threads_started":1}
I20260809 12:10:40.564708 8315 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 12:10:40.536975 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33056 (local address 127.0.222.62:41983)
0809 12:10:40.543743 (+ 6768us) server_negotiation.cc:206] Beginning negotiation
0809 12:10:40.543750 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0809 12:10:40.543772 (+ 22us) server_negotiation.cc:407] Connection header received
0809 12:10:40.543840 (+ 68us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 12:10:40.543856 (+ 16us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 12:10:40.543963 (+ 107us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 12:10:40.544212 (+ 249us) server_negotiation.cc:226] Negotiated authn=SASL
0809 12:10:40.546164 (+ 1952us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 12:10:40.547783 (+ 1619us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 12:10:40.550114 (+ 2331us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 12:10:40.550697 (+ 583us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 12:10:40.550832 (+ 135us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 12:10:40.555814 (+ 4982us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 12:10:40.555821 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 12:10:40.555825 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0809 12:10:40.555865 (+ 40us) server_negotiation.cc:922] calling sasl_server_start()
0809 12:10:40.561128 (+ 5263us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 12:10:40.562298 (+ 1170us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 12:10:40.562306 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 12:10:40.562309 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0809 12:10:40.562468 (+ 159us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 12:10:40.563080 (+ 612us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 12:10:40.563087 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 12:10:40.563092 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0809 12:10:40.563521 (+ 429us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 12:10:40.563639 (+ 118us) server_negotiation.cc:1035] Waiting for connection context
0809 12:10:40.564135 (+ 496us) server_negotiation.cc:299] Negotiation successful
0809 12:10:40.564299 (+ 164us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":6549,"thread_start_us":6540,"threads_started":1}
I20260809 12:10:40.564720 8314 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 12:10:40.536189 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.222.62:41983 (local address 127.0.0.1:33056)
0809 12:10:40.537893 (+ 1704us) negotiation.cc:107] Waiting for socket to connect
0809 12:10:40.537952 (+ 59us) client_negotiation.cc:174] Beginning negotiation
0809 12:10:40.538263 (+ 311us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 12:10:40.544299 (+ 6036us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 12:10:40.544313 (+ 14us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 12:10:40.545423 (+ 1110us) client_negotiation.cc:189] Negotiated authn=SASL
0809 12:10:40.545959 (+ 536us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 12:10:40.545972 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 12:10:40.547971 (+ 1999us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 12:10:40.547986 (+ 15us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 12:10:40.549892 (+ 1906us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 12:10:40.549905 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 12:10:40.550989 (+ 1084us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 12:10:40.551009 (+ 20us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 12:10:40.551258 (+ 249us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 12:10:40.552030 (+ 772us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0809 12:10:40.552046 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0809 12:10:40.555571 (+ 3525us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0809 12:10:40.561281 (+ 5710us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0809 12:10:40.561290 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0809 12:10:40.561298 (+ 8us) client_negotiation.cc:760] calling sasl_client_step()
0809 12:10:40.562180 (+ 882us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0809 12:10:40.562597 (+ 417us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0809 12:10:40.562602 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0809 12:10:40.562605 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0809 12:10:40.562899 (+ 294us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0809 12:10:40.563664 (+ 765us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0809 12:10:40.563670 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0809 12:10:40.563862 (+ 192us) client_negotiation.cc:769] Sending connection context
0809 12:10:40.564075 (+ 213us) client_negotiation.cc:240] Negotiation successful
0809 12:10:40.564308 (+ 233us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1297,"thread_start_us":986,"threads_started":1}
I20260809 12:10:40.566138 8306 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.222.61" port: 32831 } has no permanent_uuid. Determining permanent_uuid...
Aug 09 12:10:40 dist-test-slave-2w3w krb5kdc[7502](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786277440, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.222.60@KRBTEST.COM for oryx/127.0.222.61@KRBTEST.COM
I20260809 12:10:40.582048 8314 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 12:10:40.567226 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.222.61:32831 (local address 127.0.0.1:46532)
0809 12:10:40.567446 (+ 220us) negotiation.cc:107] Waiting for socket to connect
0809 12:10:40.567466 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0809 12:10:40.567586 (+ 120us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 12:10:40.567844 (+ 258us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 12:10:40.567847 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 12:10:40.568145 (+ 298us) client_negotiation.cc:189] Negotiated authn=SASL
0809 12:10:40.568307 (+ 162us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 12:10:40.568316 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 12:10:40.569756 (+ 1440us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 12:10:40.569760 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 12:10:40.571066 (+ 1306us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 12:10:40.571078 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 12:10:40.571973 (+ 895us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 12:10:40.571977 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 12:10:40.572191 (+ 214us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 12:10:40.572859 (+ 668us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0809 12:10:40.572875 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0809 12:10:40.575440 (+ 2565us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0809 12:10:40.579667 (+ 4227us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0809 12:10:40.579670 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0809 12:10:40.579673 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0809 12:10:40.580170 (+ 497us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0809 12:10:40.580550 (+ 380us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0809 12:10:40.580553 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0809 12:10:40.580555 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0809 12:10:40.580726 (+ 171us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0809 12:10:40.581242 (+ 516us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0809 12:10:40.581245 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0809 12:10:40.581374 (+ 129us) client_negotiation.cc:769] Sending connection context
0809 12:10:40.581489 (+ 115us) client_negotiation.cc:240] Negotiation successful
0809 12:10:40.581772 (+ 283us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":51}
I20260809 12:10:40.582361 8312 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 12:10:40.567076 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46532 (local address 127.0.222.61:32831)
0809 12:10:40.567270 (+ 194us) server_negotiation.cc:206] Beginning negotiation
0809 12:10:40.567275 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0809 12:10:40.567547 (+ 272us) server_negotiation.cc:407] Connection header received
0809 12:10:40.567677 (+ 130us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 12:10:40.567681 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 12:10:40.567741 (+ 60us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 12:10:40.567827 (+ 86us) server_negotiation.cc:226] Negotiated authn=SASL
0809 12:10:40.568406 (+ 579us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 12:10:40.569601 (+ 1195us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 12:10:40.571186 (+ 1585us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 12:10:40.571777 (+ 591us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 12:10:40.571922 (+ 145us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 12:10:40.575705 (+ 3783us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 12:10:40.575710 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 12:10:40.575713 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0809 12:10:40.575797 (+ 84us) server_negotiation.cc:922] calling sasl_server_start()
0809 12:10:40.579523 (+ 3726us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 12:10:40.580279 (+ 756us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 12:10:40.580283 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 12:10:40.580285 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0809 12:10:40.580432 (+ 147us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 12:10:40.580849 (+ 417us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 12:10:40.580852 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 12:10:40.580854 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0809 12:10:40.581143 (+ 289us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 12:10:40.581255 (+ 112us) server_negotiation.cc:1035] Waiting for connection context
0809 12:10:40.581597 (+ 342us) server_negotiation.cc:299] Negotiation successful
0809 12:10:40.581954 (+ 357us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":76}
I20260809 12:10:40.583935 8306 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.222.60" port: 39543 } has no permanent_uuid. Determining permanent_uuid...
Aug 09 12:10:40 dist-test-slave-2w3w krb5kdc[7502](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786277440, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.222.60@KRBTEST.COM for oryx/127.0.222.60@KRBTEST.COM
I20260809 12:10:40.599838 8310 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 12:10:40.584798 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42440 (local address 127.0.222.60:39543)
0809 12:10:40.585009 (+ 211us) server_negotiation.cc:206] Beginning negotiation
0809 12:10:40.585016 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0809 12:10:40.585213 (+ 197us) server_negotiation.cc:407] Connection header received
0809 12:10:40.585386 (+ 173us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 12:10:40.585390 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 12:10:40.585457 (+ 67us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 12:10:40.585552 (+ 95us) server_negotiation.cc:226] Negotiated authn=SASL
0809 12:10:40.586553 (+ 1001us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 12:10:40.587468 (+ 915us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 12:10:40.588754 (+ 1286us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 12:10:40.589338 (+ 584us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 12:10:40.589490 (+ 152us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 12:10:40.593351 (+ 3861us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 12:10:40.593356 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 12:10:40.593359 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0809 12:10:40.593388 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0809 12:10:40.596825 (+ 3437us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 12:10:40.597557 (+ 732us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 12:10:40.597560 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 12:10:40.597563 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0809 12:10:40.597710 (+ 147us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 12:10:40.598220 (+ 510us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 12:10:40.598224 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 12:10:40.598226 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0809 12:10:40.598670 (+ 444us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 12:10:40.598802 (+ 132us) server_negotiation.cc:1035] Waiting for connection context
0809 12:10:40.599301 (+ 499us) server_negotiation.cc:299] Negotiation successful
0809 12:10:40.599526 (+ 225us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":31,"server-negotiator.queue_time_us":79}
I20260809 12:10:40.599872 8314 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 12:10:40.584906 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.222.60:39543 (local address 127.0.0.1:42440)
0809 12:10:40.585129 (+ 223us) negotiation.cc:107] Waiting for socket to connect
0809 12:10:40.585144 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0809 12:10:40.585282 (+ 138us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 12:10:40.585569 (+ 287us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 12:10:40.585573 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 12:10:40.586153 (+ 580us) client_negotiation.cc:189] Negotiated authn=SASL
0809 12:10:40.586417 (+ 264us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 12:10:40.586426 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 12:10:40.587602 (+ 1176us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 12:10:40.587608 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 12:10:40.588616 (+ 1008us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 12:10:40.588627 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 12:10:40.589526 (+ 899us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 12:10:40.589532 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 12:10:40.589851 (+ 319us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 12:10:40.590546 (+ 695us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0809 12:10:40.590563 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0809 12:10:40.593087 (+ 2524us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0809 12:10:40.596951 (+ 3864us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0809 12:10:40.596955 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0809 12:10:40.596957 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0809 12:10:40.597452 (+ 495us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0809 12:10:40.597811 (+ 359us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0809 12:10:40.597816 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0809 12:10:40.597818 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0809 12:10:40.598088 (+ 270us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0809 12:10:40.598781 (+ 693us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0809 12:10:40.598785 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0809 12:10:40.599075 (+ 290us) client_negotiation.cc:769] Sending connection context
0809 12:10:40.599192 (+ 117us) client_negotiation.cc:240] Negotiation successful
0809 12:10:40.599403 (+ 211us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":51}
I20260809 12:10:40.606379 8306 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P e2a7b439c3084cd680be92ac6d1925db: Bootstrap starting.
I20260809 12:10:40.610657 8306 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P e2a7b439c3084cd680be92ac6d1925db: Neither blocks nor log segments found. Creating new log.
I20260809 12:10:40.612324 8306 log.cc:824] T 00000000000000000000000000000000 P e2a7b439c3084cd680be92ac6d1925db: Log is configured to *not* fsync() on all Append() calls
I20260809 12:10:40.616369 8306 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P e2a7b439c3084cd680be92ac6d1925db: No bootstrap required, opened a new log
I20260809 12:10:40.626842 8306 raft_consensus.cc:348] T 00000000000000000000000000000000 P e2a7b439c3084cd680be92ac6d1925db [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9cd5d41083d147f788d83a438b6f7fc2" member_type: VOTER last_known_addr { host: "127.0.222.62" port: 41983 } } peers { permanent_uuid: "cbbb52854812417da2325b27e509812d" member_type: VOTER last_known_addr { host: "127.0.222.61" port: 32831 } } peers { permanent_uuid: "e2a7b439c3084cd680be92ac6d1925db" member_type: VOTER last_known_addr { host: "127.0.222.60" port: 39543 } }
I20260809 12:10:40.627413 8306 raft_consensus.cc:374] T 00000000000000000000000000000000 P e2a7b439c3084cd680be92ac6d1925db [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260809 12:10:40.627626 8306 raft_consensus.cc:729] T 00000000000000000000000000000000 P e2a7b439c3084cd680be92ac6d1925db [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e2a7b439c3084cd680be92ac6d1925db, State: Initialized, Role: FOLLOWER
I20260809 12:10:40.628285 8306 consensus_queue.cc:260] T 00000000000000000000000000000000 P e2a7b439c3084cd680be92ac6d1925db [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9cd5d41083d147f788d83a438b6f7fc2" member_type: VOTER last_known_addr { host: "127.0.222.62" port: 41983 } } peers { permanent_uuid: "cbbb52854812417da2325b27e509812d" member_type: VOTER last_known_addr { host: "127.0.222.61" port: 32831 } } peers { permanent_uuid: "e2a7b439c3084cd680be92ac6d1925db" member_type: VOTER last_known_addr { host: "127.0.222.60" port: 39543 } }
I20260809 12:10:40.629390 8317 sys_catalog.cc:455] T 00000000000000000000000000000000 P e2a7b439c3084cd680be92ac6d1925db [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9cd5d41083d147f788d83a438b6f7fc2" member_type: VOTER last_known_addr { host: "127.0.222.62" port: 41983 } } peers { permanent_uuid: "cbbb52854812417da2325b27e509812d" member_type: VOTER last_known_addr { host: "127.0.222.61" port: 32831 } } peers { permanent_uuid: "e2a7b439c3084cd680be92ac6d1925db" member_type: VOTER last_known_addr { host: "127.0.222.60" port: 39543 } } }
I20260809 12:10:40.629987 8317 sys_catalog.cc:458] T 00000000000000000000000000000000 P e2a7b439c3084cd680be92ac6d1925db [sys.catalog]: This master's current role is: FOLLOWER
I20260809 12:10:40.630493 8306 sys_catalog.cc:564] T 00000000000000000000000000000000 P e2a7b439c3084cd680be92ac6d1925db [sys.catalog]: configured and running, proceeding with master startup.
W20260809 12:10:40.647478 8329 catalog_manager.cc:1558] T 00000000000000000000000000000000 P e2a7b439c3084cd680be92ac6d1925db: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260809 12:10:40.647679 8329 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260809 12:10:40.651099 8145 master_runner.cc:428] Detected that this master e2a7b439c3084cd680be92ac6d1925db is joining an existing cluster
I20260809 12:10:40.651304 8145 master_runner.cc:432] Initiating AddMaster RPC to add 127.0.222.60:39543
I20260809 12:10:40.667650 8317 raft_consensus.cc:482] T 00000000000000000000000000000000 P e2a7b439c3084cd680be92ac6d1925db [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260809 12:10:40.668160 8317 raft_consensus.cc:504] T 00000000000000000000000000000000 P e2a7b439c3084cd680be92ac6d1925db [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9cd5d41083d147f788d83a438b6f7fc2" member_type: VOTER last_known_addr { host: "127.0.222.62" port: 41983 } } peers { permanent_uuid: "cbbb52854812417da2325b27e509812d" member_type: VOTER last_known_addr { host: "127.0.222.61" port: 32831 } } peers { permanent_uuid: "e2a7b439c3084cd680be92ac6d1925db" member_type: VOTER last_known_addr { host: "127.0.222.60" port: 39543 } }
I20260809 12:10:40.672623 888 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskgH5Pjm/build/asan/bin/kudu pid 7611
I20260809 12:10:40.677119 8193 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "e2a7b439c3084cd680be92ac6d1925db" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "cbbb52854812417da2325b27e509812d" is_pre_election: true
I20260809 12:10:40.677356 8317 leader_election.cc:288] T 00000000000000000000000000000000 P e2a7b439c3084cd680be92ac6d1925db [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 9cd5d41083d147f788d83a438b6f7fc2 (127.0.222.62:41983), cbbb52854812417da2325b27e509812d (127.0.222.61:32831)
I20260809 12:10:40.677145 7559 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "e2a7b439c3084cd680be92ac6d1925db" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9cd5d41083d147f788d83a438b6f7fc2" is_pre_election: true
I20260809 12:10:40.679610 8254 leader_election.cc:302] T 00000000000000000000000000000000 P e2a7b439c3084cd680be92ac6d1925db [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: e2a7b439c3084cd680be92ac6d1925db; no voters: 9cd5d41083d147f788d83a438b6f7fc2, cbbb52854812417da2325b27e509812d
I20260809 12:10:40.684335 8317 raft_consensus.cc:2726] T 00000000000000000000000000000000 P e2a7b439c3084cd680be92ac6d1925db [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
Aug 09 12:10:40 dist-test-slave-2w3w krb5kdc[7502](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786277437, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.0.222.1@KRBTEST.COM
I20260809 12:10:40.717283 8338 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 12:10:40.674866 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49252 (local address 127.0.222.1:43733)
0809 12:10:40.684196 (+ 9330us) server_negotiation.cc:206] Beginning negotiation
0809 12:10:40.684216 (+ 20us) server_negotiation.cc:399] Waiting for connection header
0809 12:10:40.684267 (+ 51us) server_negotiation.cc:407] Connection header received
0809 12:10:40.684348 (+ 81us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 12:10:40.684358 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 12:10:40.684724 (+ 366us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 12:10:40.685150 (+ 426us) server_negotiation.cc:226] Negotiated authn=SASL
0809 12:10:40.687724 (+ 2574us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 12:10:40.689634 (+ 1910us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 12:10:40.692399 (+ 2765us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 12:10:40.693930 (+ 1531us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 12:10:40.694999 (+ 1069us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 12:10:40.705261 (+ 10262us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 12:10:40.705290 (+ 29us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 12:10:40.705312 (+ 22us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0809 12:10:40.705392 (+ 80us) server_negotiation.cc:922] calling sasl_server_start()
0809 12:10:40.711726 (+ 6334us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 12:10:40.712792 (+ 1066us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 12:10:40.712803 (+ 11us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 12:10:40.712806 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0809 12:10:40.713079 (+ 273us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 12:10:40.713620 (+ 541us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 12:10:40.713625 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 12:10:40.713629 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0809 12:10:40.714204 (+ 575us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 12:10:40.714727 (+ 523us) server_negotiation.cc:1035] Waiting for connection context
0809 12:10:40.716374 (+ 1647us) server_negotiation.cc:299] Negotiation successful
0809 12:10:40.716690 (+ 316us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":9065,"thread_start_us":8744,"threads_started":1}
I20260809 12:10:40.722405 7741 generic_service.cc:196] Checking for leaks (request via RPC)
I20260809 12:10:40.725307 8310 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 12:10:40.657780 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42454 (local address 127.0.222.60:39543)
0809 12:10:40.657979 (+ 199us) server_negotiation.cc:206] Beginning negotiation
0809 12:10:40.657985 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0809 12:10:40.679104 (+ 21119us) server_negotiation.cc:407] Connection header received
0809 12:10:40.679341 (+ 237us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 12:10:40.679348 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 12:10:40.679438 (+ 90us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 12:10:40.679584 (+ 146us) server_negotiation.cc:226] Negotiated authn=SASL
0809 12:10:40.692647 (+ 13063us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 12:10:40.694954 (+ 2307us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 12:10:40.698135 (+ 3181us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 12:10:40.699204 (+ 1069us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 12:10:40.699347 (+ 143us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 12:10:40.703387 (+ 4040us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 12:10:40.703398 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 12:10:40.703403 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0809 12:10:40.703612 (+ 209us) server_negotiation.cc:922] calling sasl_server_start()
0809 12:10:40.710729 (+ 7117us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 12:10:40.722379 (+ 11650us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 12:10:40.722384 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 12:10:40.722388 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0809 12:10:40.722598 (+ 210us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 12:10:40.723203 (+ 605us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 12:10:40.723208 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 12:10:40.723211 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0809 12:10:40.723690 (+ 479us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 12:10:40.723815 (+ 125us) server_negotiation.cc:1035] Waiting for connection context
0809 12:10:40.724432 (+ 617us) server_negotiation.cc:299] Negotiation successful
0809 12:10:40.724752 (+ 320us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":4,"server-negotiator.queue_time_us":63}
W20260809 12:10:40.726925 8259 tablet.cc:2369] T 00000000000000000000000000000000 P e2a7b439c3084cd680be92ac6d1925db: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260809 12:10:40.728011 8336 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 12:10:40.659268 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.222.60:39543 (local address 127.0.0.1:42454)
0809 12:10:40.679001 (+ 19733us) negotiation.cc:107] Waiting for socket to connect
0809 12:10:40.679022 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0809 12:10:40.679191 (+ 169us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 12:10:40.679852 (+ 661us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 12:10:40.679858 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 12:10:40.686562 (+ 6704us) client_negotiation.cc:189] Negotiated authn=SASL
0809 12:10:40.688124 (+ 1562us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 12:10:40.688141 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 12:10:40.695401 (+ 7260us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 12:10:40.695407 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 12:10:40.697956 (+ 2549us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 12:10:40.697971 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 12:10:40.699806 (+ 1835us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 12:10:40.699814 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 12:10:40.700052 (+ 238us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 12:10:40.700924 (+ 872us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0809 12:10:40.700949 (+ 25us) client_negotiation.cc:656] calling sasl_client_start()
0809 12:10:40.703060 (+ 2111us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0809 12:10:40.721683 (+ 18623us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0809 12:10:40.721689 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0809 12:10:40.721693 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0809 12:10:40.722213 (+ 520us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0809 12:10:40.722751 (+ 538us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0809 12:10:40.722758 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0809 12:10:40.722763 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0809 12:10:40.723087 (+ 324us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0809 12:10:40.723904 (+ 817us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0809 12:10:40.723909 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0809 12:10:40.724135 (+ 226us) client_negotiation.cc:769] Sending connection context
0809 12:10:40.725521 (+ 1386us) client_negotiation.cc:240] Negotiation successful
0809 12:10:40.725894 (+ 373us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":19505,"thread_start_us":21873,"threads_started":1}
I20260809 12:10:40.732460 8312 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 12:10:40.657707 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46542 (local address 127.0.222.61:32831)
0809 12:10:40.657990 (+ 283us) server_negotiation.cc:206] Beginning negotiation
0809 12:10:40.658006 (+ 16us) server_negotiation.cc:399] Waiting for connection header
0809 12:10:40.687789 (+ 29783us) server_negotiation.cc:407] Connection header received
0809 12:10:40.688200 (+ 411us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 12:10:40.688206 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 12:10:40.688391 (+ 185us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 12:10:40.688506 (+ 115us) server_negotiation.cc:226] Negotiated authn=SASL
0809 12:10:40.691369 (+ 2863us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 12:10:40.692886 (+ 1517us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 12:10:40.700564 (+ 7678us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 12:10:40.701469 (+ 905us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 12:10:40.701604 (+ 135us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 12:10:40.707752 (+ 6148us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 12:10:40.707770 (+ 18us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 12:10:40.707775 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0809 12:10:40.707835 (+ 60us) server_negotiation.cc:922] calling sasl_server_start()
0809 12:10:40.714236 (+ 6401us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 12:10:40.719446 (+ 5210us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 12:10:40.719452 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 12:10:40.719456 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0809 12:10:40.719651 (+ 195us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 12:10:40.728871 (+ 9220us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 12:10:40.728878 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 12:10:40.728882 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0809 12:10:40.729353 (+ 471us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 12:10:40.729516 (+ 163us) server_negotiation.cc:1035] Waiting for connection context
0809 12:10:40.731422 (+ 1906us) server_negotiation.cc:299] Negotiation successful
0809 12:10:40.731754 (+ 332us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":60}
I20260809 12:10:40.734493 8337 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 12:10:40.659410 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.222.61:32831 (local address 127.0.0.1:46542)
0809 12:10:40.687527 (+ 28117us) negotiation.cc:107] Waiting for socket to connect
0809 12:10:40.687563 (+ 36us) client_negotiation.cc:174] Beginning negotiation
0809 12:10:40.687992 (+ 429us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 12:10:40.688883 (+ 891us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 12:10:40.688886 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 12:10:40.690144 (+ 1258us) client_negotiation.cc:189] Negotiated authn=SASL
0809 12:10:40.691100 (+ 956us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 12:10:40.691115 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 12:10:40.693100 (+ 1985us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 12:10:40.693105 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 12:10:40.700340 (+ 7235us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 12:10:40.700357 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 12:10:40.702178 (+ 1821us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 12:10:40.702189 (+ 11us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 12:10:40.702511 (+ 322us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 12:10:40.704046 (+ 1535us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0809 12:10:40.704074 (+ 28us) client_negotiation.cc:656] calling sasl_client_start()
0809 12:10:40.707465 (+ 3391us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0809 12:10:40.714694 (+ 7229us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0809 12:10:40.714701 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0809 12:10:40.714707 (+ 6us) client_negotiation.cc:760] calling sasl_client_step()
0809 12:10:40.715421 (+ 714us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0809 12:10:40.720796 (+ 5375us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0809 12:10:40.720804 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0809 12:10:40.720810 (+ 6us) client_negotiation.cc:760] calling sasl_client_step()
0809 12:10:40.728652 (+ 7842us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0809 12:10:40.729612 (+ 960us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0809 12:10:40.729618 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0809 12:10:40.731024 (+ 1406us) client_negotiation.cc:769] Sending connection context
0809 12:10:40.732808 (+ 1784us) client_negotiation.cc:240] Negotiation successful
0809 12:10:40.733205 (+ 397us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":26598,"mutex_wait_us":58,"thread_start_us":19372,"threads_started":1}
I20260809 12:10:40.734580 8315 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 12:10:40.657692 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33066 (local address 127.0.222.62:41983)
0809 12:10:40.657904 (+ 212us) server_negotiation.cc:206] Beginning negotiation
0809 12:10:40.657912 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0809 12:10:40.678418 (+ 20506us) server_negotiation.cc:407] Connection header received
0809 12:10:40.678516 (+ 98us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 12:10:40.678525 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 12:10:40.678645 (+ 120us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 12:10:40.678778 (+ 133us) server_negotiation.cc:226] Negotiated authn=SASL
0809 12:10:40.689013 (+ 10235us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 12:10:40.690851 (+ 1838us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 12:10:40.694367 (+ 3516us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 12:10:40.696034 (+ 1667us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 12:10:40.696200 (+ 166us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 12:10:40.715715 (+ 19515us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 12:10:40.715724 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 12:10:40.715732 (+ 8us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0809 12:10:40.715784 (+ 52us) server_negotiation.cc:922] calling sasl_server_start()
0809 12:10:40.728394 (+ 12610us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 12:10:40.730693 (+ 2299us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 12:10:40.730699 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 12:10:40.730704 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0809 12:10:40.731063 (+ 359us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 12:10:40.731593 (+ 530us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 12:10:40.731598 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 12:10:40.731601 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0809 12:10:40.732229 (+ 628us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 12:10:40.732526 (+ 297us) server_negotiation.cc:1035] Waiting for connection context
0809 12:10:40.733681 (+ 1155us) server_negotiation.cc:299] Negotiation successful
0809 12:10:40.734025 (+ 344us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":71}
I20260809 12:10:40.737469 8335 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 12:10:40.658742 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.222.62:41983 (local address 127.0.0.1:33066)
0809 12:10:40.676851 (+ 18109us) negotiation.cc:107] Waiting for socket to connect
0809 12:10:40.676882 (+ 31us) client_negotiation.cc:174] Beginning negotiation
0809 12:10:40.677115 (+ 233us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 12:10:40.680017 (+ 2902us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 12:10:40.680021 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 12:10:40.687103 (+ 7082us) client_negotiation.cc:189] Negotiated authn=SASL
0809 12:10:40.688768 (+ 1665us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 12:10:40.688786 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 12:10:40.691176 (+ 2390us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 12:10:40.691184 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 12:10:40.694150 (+ 2966us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 12:10:40.694166 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 12:10:40.702700 (+ 8534us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 12:10:40.702708 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 12:10:40.703032 (+ 324us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 12:10:40.707962 (+ 4930us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0809 12:10:40.707988 (+ 26us) client_negotiation.cc:656] calling sasl_client_start()
0809 12:10:40.714516 (+ 6528us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0809 12:10:40.729953 (+ 15437us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0809 12:10:40.729958 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0809 12:10:40.729962 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0809 12:10:40.730477 (+ 515us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0809 12:10:40.731235 (+ 758us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0809 12:10:40.731241 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0809 12:10:40.731248 (+ 7us) client_negotiation.cc:760] calling sasl_client_step()
0809 12:10:40.731453 (+ 205us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0809 12:10:40.732635 (+ 1182us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0809 12:10:40.732640 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0809 12:10:40.733003 (+ 363us) client_negotiation.cc:769] Sending connection context
0809 12:10:40.733146 (+ 143us) client_negotiation.cc:240] Negotiation successful
0809 12:10:40.733713 (+ 567us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":17887,"thread_start_us":24265,"threads_started":1}
Aug 09 12:10:40 dist-test-slave-2w3w krb5kdc[7502](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786277437, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.222.62@KRBTEST.COM for oryx/127.0.222.60@KRBTEST.COM
I20260809 12:10:40.765537 8310 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 12:10:40.745180 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42458 (local address 127.0.222.60:39543)
0809 12:10:40.745348 (+ 168us) server_negotiation.cc:206] Beginning negotiation
0809 12:10:40.745353 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0809 12:10:40.745370 (+ 17us) server_negotiation.cc:407] Connection header received
0809 12:10:40.745454 (+ 84us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 12:10:40.745459 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 12:10:40.745539 (+ 80us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 12:10:40.745639 (+ 100us) server_negotiation.cc:226] Negotiated authn=SASL
0809 12:10:40.746942 (+ 1303us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 12:10:40.748526 (+ 1584us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 12:10:40.750546 (+ 2020us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 12:10:40.751709 (+ 1163us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 12:10:40.751845 (+ 136us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 12:10:40.757721 (+ 5876us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 12:10:40.757730 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 12:10:40.757735 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0809 12:10:40.757795 (+ 60us) server_negotiation.cc:922] calling sasl_server_start()
0809 12:10:40.762254 (+ 4459us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 12:10:40.763153 (+ 899us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 12:10:40.763160 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 12:10:40.763164 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0809 12:10:40.763317 (+ 153us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 12:10:40.763754 (+ 437us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 12:10:40.763759 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 12:10:40.763763 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0809 12:10:40.764090 (+ 327us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 12:10:40.764247 (+ 157us) server_negotiation.cc:1035] Waiting for connection context
0809 12:10:40.764711 (+ 464us) server_negotiation.cc:299] Negotiation successful
0809 12:10:40.764971 (+ 260us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":39}
I20260809 12:10:40.765623 8311 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 12:10:40.744771 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.222.60:39543 (local address 127.0.0.1:42458)
0809 12:10:40.744958 (+ 187us) negotiation.cc:107] Waiting for socket to connect
0809 12:10:40.744979 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0809 12:10:40.745089 (+ 110us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 12:10:40.745708 (+ 619us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 12:10:40.745716 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 12:10:40.746403 (+ 687us) client_negotiation.cc:189] Negotiated authn=SASL
0809 12:10:40.746755 (+ 352us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 12:10:40.746770 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 12:10:40.748695 (+ 1925us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 12:10:40.748702 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 12:10:40.750375 (+ 1673us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 12:10:40.750391 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 12:10:40.751903 (+ 1512us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 12:10:40.751910 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 12:10:40.752199 (+ 289us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 12:10:40.753271 (+ 1072us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0809 12:10:40.753304 (+ 33us) client_negotiation.cc:656] calling sasl_client_start()
0809 12:10:40.757465 (+ 4161us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0809 12:10:40.762418 (+ 4953us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0809 12:10:40.762424 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0809 12:10:40.762428 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0809 12:10:40.763020 (+ 592us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0809 12:10:40.763451 (+ 431us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0809 12:10:40.763457 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0809 12:10:40.763460 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0809 12:10:40.763638 (+ 178us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0809 12:10:40.764229 (+ 591us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0809 12:10:40.764234 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0809 12:10:40.764455 (+ 221us) client_negotiation.cc:769] Sending connection context
0809 12:10:40.764619 (+ 164us) client_negotiation.cc:240] Negotiation successful
0809 12:10:40.764960 (+ 341us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":62}
I20260809 12:10:40.767853 7547 catalog_manager.cc:6979] Initiating ChangeConfig request to add master 127.0.222.60:39543: tablet_id: "00000000000000000000000000000000"
type: ADD_PEER
server {
permanent_uuid: "e2a7b439c3084cd680be92ac6d1925db"
member_type: NON_VOTER
last_known_addr {
host: "127.0.222.60"
port: 39543
}
attrs {
promote: true
}
}
dest_uuid: "9cd5d41083d147f788d83a438b6f7fc2"
cas_config_opid_index: 5
I20260809 12:10:40.769238 7547 consensus_queue.cc:237] T 00000000000000000000000000000000 P 9cd5d41083d147f788d83a438b6f7fc2 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 1.5, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "9cd5d41083d147f788d83a438b6f7fc2" member_type: VOTER last_known_addr { host: "127.0.222.62" port: 41983 } } peers { permanent_uuid: "cbbb52854812417da2325b27e509812d" member_type: NON_VOTER last_known_addr { host: "127.0.222.61" port: 32831 } attrs { promote: true } } peers { permanent_uuid: "e2a7b439c3084cd680be92ac6d1925db" member_type: NON_VOTER last_known_addr { host: "127.0.222.60" port: 39543 } attrs { promote: true } }
I20260809 12:10:40.771576 8309 raft_consensus.cc:2932] T 00000000000000000000000000000000 P 9cd5d41083d147f788d83a438b6f7fc2 [term 1 LEADER]: Committing config change with OpId 1.6: config changed from index 5 to 6, NON_VOTER e2a7b439c3084cd680be92ac6d1925db (127.0.222.60) added. New config: { opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "9cd5d41083d147f788d83a438b6f7fc2" member_type: VOTER last_known_addr { host: "127.0.222.62" port: 41983 } } peers { permanent_uuid: "cbbb52854812417da2325b27e509812d" member_type: NON_VOTER last_known_addr { host: "127.0.222.61" port: 32831 } attrs { promote: true } } peers { permanent_uuid: "e2a7b439c3084cd680be92ac6d1925db" member_type: NON_VOTER last_known_addr { host: "127.0.222.60" port: 39543 } attrs { promote: true } } }
I20260809 12:10:40.776463 8193 raft_consensus.cc:1261] T 00000000000000000000000000000000 P cbbb52854812417da2325b27e509812d [term 1 LEARNER]: Refusing update from remote peer 9cd5d41083d147f788d83a438b6f7fc2: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20260809 12:10:40.779206 8280 raft_consensus.cc:3037] T 00000000000000000000000000000000 P e2a7b439c3084cd680be92ac6d1925db [term 0 FOLLOWER]: Advancing to term 1
I20260809 12:10:40.779693 8341 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 9cd5d41083d147f788d83a438b6f7fc2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "cbbb52854812417da2325b27e509812d" member_type: NON_VOTER last_known_addr { host: "127.0.222.61" port: 32831 } attrs { promote: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20260809 12:10:40.786908 8280 raft_consensus.cc:1261] T 00000000000000000000000000000000 P e2a7b439c3084cd680be92ac6d1925db [term 1 FOLLOWER]: Refusing update from remote peer 9cd5d41083d147f788d83a438b6f7fc2: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20260809 12:10:40.790081 8309 catalog_manager.cc:6983] Successfully completed master ChangeConfig request to add master 127.0.222.60:39543
I20260809 12:10:40.791939 8193 raft_consensus.cc:2932] T 00000000000000000000000000000000 P cbbb52854812417da2325b27e509812d [term 1 LEARNER]: Committing config change with OpId 1.6: config changed from index 5 to 6, NON_VOTER e2a7b439c3084cd680be92ac6d1925db (127.0.222.60) added. New config: { opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "9cd5d41083d147f788d83a438b6f7fc2" member_type: VOTER last_known_addr { host: "127.0.222.62" port: 41983 } } peers { permanent_uuid: "cbbb52854812417da2325b27e509812d" member_type: NON_VOTER last_known_addr { host: "127.0.222.61" port: 32831 } attrs { promote: true } } peers { permanent_uuid: "e2a7b439c3084cd680be92ac6d1925db" member_type: NON_VOTER last_known_addr { host: "127.0.222.60" port: 39543 } attrs { promote: true } } }
I20260809 12:10:40.794224 8145 master.cc:537] Master@127.0.222.60:39543 shutting down...
I20260809 12:10:40.803045 8309 sys_catalog.cc:455] T 00000000000000000000000000000000 P 9cd5d41083d147f788d83a438b6f7fc2 [sys.catalog]: SysCatalogTable state changed. Reason: Config change replication complete. Latest consensus state: current_term: 1 leader_uuid: "9cd5d41083d147f788d83a438b6f7fc2" committed_config { opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "9cd5d41083d147f788d83a438b6f7fc2" member_type: VOTER last_known_addr { host: "127.0.222.62" port: 41983 } } peers { permanent_uuid: "cbbb52854812417da2325b27e509812d" member_type: NON_VOTER last_known_addr { host: "127.0.222.61" port: 32831 } attrs { promote: true } } peers { permanent_uuid: "e2a7b439c3084cd680be92ac6d1925db" member_type: NON_VOTER last_known_addr { host: "127.0.222.60" port: 39543 } attrs { promote: true } } }
I20260809 12:10:40.804808 8309 sys_catalog.cc:458] T 00000000000000000000000000000000 P 9cd5d41083d147f788d83a438b6f7fc2 [sys.catalog]: This master's current role is: LEADER
I20260809 12:10:40.805218 8341 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 9cd5d41083d147f788d83a438b6f7fc2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e2a7b439c3084cd680be92ac6d1925db" member_type: NON_VOTER last_known_addr { host: "127.0.222.60" port: 39543 } attrs { promote: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 0, Time since last communication: 0.000s
I20260809 12:10:40.808629 8225 sys_catalog.cc:455] T 00000000000000000000000000000000 P cbbb52854812417da2325b27e509812d [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "9cd5d41083d147f788d83a438b6f7fc2" committed_config { opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "9cd5d41083d147f788d83a438b6f7fc2" member_type: VOTER last_known_addr { host: "127.0.222.62" port: 41983 } } peers { permanent_uuid: "cbbb52854812417da2325b27e509812d" member_type: NON_VOTER last_known_addr { host: "127.0.222.61" port: 32831 } attrs { promote: true } } peers { permanent_uuid: "e2a7b439c3084cd680be92ac6d1925db" member_type: NON_VOTER last_known_addr { host: "127.0.222.60" port: 39543 } attrs { promote: true } } }
I20260809 12:10:40.809015 8225 sys_catalog.cc:458] T 00000000000000000000000000000000 P cbbb52854812417da2325b27e509812d [sys.catalog]: This master's current role is: LEARNER
I20260809 12:10:40.810617 8342 raft_consensus.cc:1050] T 00000000000000000000000000000000 P 9cd5d41083d147f788d83a438b6f7fc2: attempting to promote NON_VOTER cbbb52854812417da2325b27e509812d to VOTER
I20260809 12:10:40.815058 8342 consensus_queue.cc:237] T 00000000000000000000000000000000 P 9cd5d41083d147f788d83a438b6f7fc2 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 1.6, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "9cd5d41083d147f788d83a438b6f7fc2" member_type: VOTER last_known_addr { host: "127.0.222.62" port: 41983 } } peers { permanent_uuid: "cbbb52854812417da2325b27e509812d" member_type: VOTER last_known_addr { host: "127.0.222.61" port: 32831 } attrs { promote: false } } peers { permanent_uuid: "e2a7b439c3084cd680be92ac6d1925db" member_type: NON_VOTER last_known_addr { host: "127.0.222.60" port: 39543 } attrs { promote: true } }
I20260809 12:10:40.817444 8193 raft_consensus.cc:1261] T 00000000000000000000000000000000 P cbbb52854812417da2325b27e509812d [term 1 LEARNER]: Refusing update from remote peer 9cd5d41083d147f788d83a438b6f7fc2: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20260809 12:10:40.818255 8342 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 9cd5d41083d147f788d83a438b6f7fc2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "cbbb52854812417da2325b27e509812d" member_type: VOTER last_known_addr { host: "127.0.222.61" port: 32831 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20260809 12:10:40.820641 8341 raft_consensus.cc:2932] T 00000000000000000000000000000000 P 9cd5d41083d147f788d83a438b6f7fc2 [term 1 LEADER]: Committing config change with OpId 1.7: config changed from index 6 to 7, cbbb52854812417da2325b27e509812d (127.0.222.61) changed from NON_VOTER to VOTER. New config: { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "9cd5d41083d147f788d83a438b6f7fc2" member_type: VOTER last_known_addr { host: "127.0.222.62" port: 41983 } } peers { permanent_uuid: "cbbb52854812417da2325b27e509812d" member_type: VOTER last_known_addr { host: "127.0.222.61" port: 32831 } attrs { promote: false } } peers { permanent_uuid: "e2a7b439c3084cd680be92ac6d1925db" member_type: NON_VOTER last_known_addr { host: "127.0.222.60" port: 39543 } attrs { promote: true } } }
I20260809 12:10:40.821384 8193 raft_consensus.cc:2932] T 00000000000000000000000000000000 P cbbb52854812417da2325b27e509812d [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, cbbb52854812417da2325b27e509812d (127.0.222.61) changed from NON_VOTER to VOTER. New config: { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "9cd5d41083d147f788d83a438b6f7fc2" member_type: VOTER last_known_addr { host: "127.0.222.62" port: 41983 } } peers { permanent_uuid: "cbbb52854812417da2325b27e509812d" member_type: VOTER last_known_addr { host: "127.0.222.61" port: 32831 } attrs { promote: false } } peers { permanent_uuid: "e2a7b439c3084cd680be92ac6d1925db" member_type: NON_VOTER last_known_addr { host: "127.0.222.60" port: 39543 } attrs { promote: true } } }
I20260809 12:10:40.825110 8341 sys_catalog.cc:455] T 00000000000000000000000000000000 P 9cd5d41083d147f788d83a438b6f7fc2 [sys.catalog]: SysCatalogTable state changed. Reason: Config change replication complete. Latest consensus state: current_term: 1 leader_uuid: "9cd5d41083d147f788d83a438b6f7fc2" committed_config { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "9cd5d41083d147f788d83a438b6f7fc2" member_type: VOTER last_known_addr { host: "127.0.222.62" port: 41983 } } peers { permanent_uuid: "cbbb52854812417da2325b27e509812d" member_type: VOTER last_known_addr { host: "127.0.222.61" port: 32831 } attrs { promote: false } } peers { permanent_uuid: "e2a7b439c3084cd680be92ac6d1925db" member_type: NON_VOTER last_known_addr { host: "127.0.222.60" port: 39543 } attrs { promote: true } } }
I20260809 12:10:40.825569 8341 sys_catalog.cc:458] T 00000000000000000000000000000000 P 9cd5d41083d147f788d83a438b6f7fc2 [sys.catalog]: This master's current role is: LEADER
I20260809 12:10:40.825516 8342 sys_catalog.cc:455] T 00000000000000000000000000000000 P 9cd5d41083d147f788d83a438b6f7fc2 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "9cd5d41083d147f788d83a438b6f7fc2" committed_config { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "9cd5d41083d147f788d83a438b6f7fc2" member_type: VOTER last_known_addr { host: "127.0.222.62" port: 41983 } } peers { permanent_uuid: "cbbb52854812417da2325b27e509812d" member_type: VOTER last_known_addr { host: "127.0.222.61" port: 32831 } attrs { promote: false } } peers { permanent_uuid: "e2a7b439c3084cd680be92ac6d1925db" member_type: NON_VOTER last_known_addr { host: "127.0.222.60" port: 39543 } attrs { promote: true } } }
I20260809 12:10:40.825846 8342 sys_catalog.cc:458] T 00000000000000000000000000000000 P 9cd5d41083d147f788d83a438b6f7fc2 [sys.catalog]: This master's current role is: LEADER
I20260809 12:10:40.826432 8225 sys_catalog.cc:455] T 00000000000000000000000000000000 P cbbb52854812417da2325b27e509812d [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "9cd5d41083d147f788d83a438b6f7fc2" committed_config { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "9cd5d41083d147f788d83a438b6f7fc2" member_type: VOTER last_known_addr { host: "127.0.222.62" port: 41983 } } peers { permanent_uuid: "cbbb52854812417da2325b27e509812d" member_type: VOTER last_known_addr { host: "127.0.222.61" port: 32831 } attrs { promote: false } } peers { permanent_uuid: "e2a7b439c3084cd680be92ac6d1925db" member_type: NON_VOTER last_known_addr { host: "127.0.222.60" port: 39543 } attrs { promote: true } } }
I20260809 12:10:40.826735 8225 sys_catalog.cc:458] T 00000000000000000000000000000000 P cbbb52854812417da2325b27e509812d [sys.catalog]: This master's current role is: FOLLOWER
I20260809 12:10:40.834532 8145 raft_consensus.cc:2229] T 00000000000000000000000000000000 P e2a7b439c3084cd680be92ac6d1925db [term 1 FOLLOWER]: Raft consensus shutting down.
W20260809 12:10:40.834992 7533 consensus_peers.cc:487] T 00000000000000000000000000000000 P 9cd5d41083d147f788d83a438b6f7fc2 -> Peer e2a7b439c3084cd680be92ac6d1925db (127.0.222.60:39543): Couldn't send request to peer e2a7b439c3084cd680be92ac6d1925db. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on Master. This is attempt 1: this message will repeat every 5th retry.
I20260809 12:10:40.835129 8145 raft_consensus.cc:2258] T 00000000000000000000000000000000 P e2a7b439c3084cd680be92ac6d1925db [term 1 FOLLOWER]: Raft consensus is shut down!
I20260809 12:10:40.835337 8145 tablet_replica.cc:330] T 00000000000000000000000000000000 P e2a7b439c3084cd680be92ac6d1925db: stopping tablet replica
I20260809 12:10:40.842149 8145 master.cc:552] Master@127.0.222.60:39543 shutdown complete.
I20260809 12:10:40.842259 8145 master_runner.cc:305] Clearing existing system tablet
I20260809 12:10:40.842733 8145 fs_manager.cc:359] Metadata directory not provided
I20260809 12:10:40.842882 8145 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260809 12:10:40.845456 8145 fs_manager.cc:709] Time spent opening directory manager: real 0.002s user 0.001s sys 0.001s
I20260809 12:10:40.847194 8348 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260809 12:10:40.847527 8145 fs_report.cc:383] FS layout report
--------------------
wal directory:
metadata directory:
1 data directories: /tmp/dist-test-taskgH5Pjm/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786277382073938-888-0/minicluster-data/master-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260809 12:10:40.847817 8145 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.001s
I20260809 12:10:40.847918 8145 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskgH5Pjm/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786277382073938-888-0/minicluster-data/master-2/data,/tmp/dist-test-taskgH5Pjm/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786277382073938-888-0/minicluster-data/master-2/wal
uuid: "e2a7b439c3084cd680be92ac6d1925db"
format_stamp: "Formatted at 2026-08-09 12:10:40 on dist-test-slave-2w3w"
server_key: "67dabfc5aa05d4447bd46bb4b26ce89c"
server_key_iv: "ca9f445d4b4bb45f49d87f7e3864c8d0"
server_key_version: "encryptionkey@0"
I20260809 12:10:40.851423 8145 ts_tablet_manager.cc:1905] T 00000000000000000000000000000000 P e2a7b439c3084cd680be92ac6d1925db: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260809 12:10:40.857235 8145 ts_tablet_manager.cc:1918] T 00000000000000000000000000000000 P e2a7b439c3084cd680be92ac6d1925db: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId unknown
I20260809 12:10:40.857436 8145 log.cc:1196] T 00000000000000000000000000000000 P e2a7b439c3084cd680be92ac6d1925db: Deleting WAL directory at /tmp/dist-test-taskgH5Pjm/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786277382073938-888-0/minicluster-data/master-2/wal/wals/00000000000000000000000000000000
I20260809 12:10:40.858031 8145 ts_tablet_manager.cc:1939] T 00000000000000000000000000000000 P e2a7b439c3084cd680be92ac6d1925db: Deleting consensus metadata
I20260809 12:10:40.858287 8145 master_runner.cc:315] Copying system tablet from 127.0.222.62:41983
I20260809 12:10:40.861469 8145 tablet_copy_client.cc:316] T 00000000000000000000000000000000 P e2a7b439c3084cd680be92ac6d1925db: tablet copy: Beginning tablet copy session from remote peer at address 127.0.222.62:41983
I20260809 12:10:40.879243 8315 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 12:10:40.862412 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33080 (local address 127.0.222.62:41983)
0809 12:10:40.862643 (+ 231us) server_negotiation.cc:206] Beginning negotiation
0809 12:10:40.862653 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0809 12:10:40.863844 (+ 1191us) server_negotiation.cc:407] Connection header received
0809 12:10:40.864113 (+ 269us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 12:10:40.864119 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 12:10:40.864199 (+ 80us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 12:10:40.864322 (+ 123us) server_negotiation.cc:226] Negotiated authn=SASL
0809 12:10:40.865853 (+ 1531us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 12:10:40.866620 (+ 767us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 12:10:40.868243 (+ 1623us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 12:10:40.869132 (+ 889us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 12:10:40.869275 (+ 143us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 12:10:40.871763 (+ 2488us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 12:10:40.871769 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 12:10:40.871774 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0809 12:10:40.871817 (+ 43us) server_negotiation.cc:922] calling sasl_server_start()
0809 12:10:40.875495 (+ 3678us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 12:10:40.876382 (+ 887us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 12:10:40.876388 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 12:10:40.876391 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0809 12:10:40.876653 (+ 262us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 12:10:40.877174 (+ 521us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 12:10:40.877179 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 12:10:40.877182 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0809 12:10:40.877587 (+ 405us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 12:10:40.877759 (+ 172us) server_negotiation.cc:1035] Waiting for connection context
0809 12:10:40.878252 (+ 493us) server_negotiation.cc:299] Negotiation successful
0809 12:10:40.878515 (+ 263us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":79}
I20260809 12:10:40.879284 8354 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 12:10:40.862408 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.222.62:41983 (local address 127.0.0.1:33080)
0809 12:10:40.863703 (+ 1295us) negotiation.cc:107] Waiting for socket to connect
0809 12:10:40.863720 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0809 12:10:40.863945 (+ 225us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 12:10:40.864348 (+ 403us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 12:10:40.864353 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 12:10:40.864975 (+ 622us) client_negotiation.cc:189] Negotiated authn=SASL
0809 12:10:40.865680 (+ 705us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 12:10:40.865697 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 12:10:40.866819 (+ 1122us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 12:10:40.866824 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 12:10:40.868078 (+ 1254us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 12:10:40.868092 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 12:10:40.869296 (+ 1204us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 12:10:40.869303 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 12:10:40.869581 (+ 278us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 12:10:40.870424 (+ 843us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0809 12:10:40.870442 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0809 12:10:40.871569 (+ 1127us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0809 12:10:40.875723 (+ 4154us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0809 12:10:40.875727 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0809 12:10:40.875731 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0809 12:10:40.876238 (+ 507us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0809 12:10:40.876822 (+ 584us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0809 12:10:40.876826 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0809 12:10:40.876828 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0809 12:10:40.877044 (+ 216us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0809 12:10:40.877719 (+ 675us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0809 12:10:40.877725 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0809 12:10:40.877935 (+ 210us) client_negotiation.cc:769] Sending connection context
0809 12:10:40.878079 (+ 144us) client_negotiation.cc:240] Negotiation successful
0809 12:10:40.878520 (+ 441us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1106,"thread_start_us":966,"threads_started":1}
I20260809 12:10:40.880379 7569 tablet_copy_service.cc:140] P 9cd5d41083d147f788d83a438b6f7fc2: Received BeginTabletCopySession request for tablet 00000000000000000000000000000000 from peer e2a7b439c3084cd680be92ac6d1925db ({username='oryx', principal='oryx/127.0.222.60@KRBTEST.COM'} at 127.0.0.1:33080)
I20260809 12:10:40.880570 7569 tablet_copy_service.cc:161] P 9cd5d41083d147f788d83a438b6f7fc2: Beginning new tablet copy session on tablet 00000000000000000000000000000000 from peer e2a7b439c3084cd680be92ac6d1925db at {username='oryx', principal='oryx/127.0.222.60@KRBTEST.COM'} at 127.0.0.1:33080: session id = e2a7b439c3084cd680be92ac6d1925db-00000000000000000000000000000000
I20260809 12:10:40.884483 7569 tablet_copy_source_session.cc:205] T 00000000000000000000000000000000 P 9cd5d41083d147f788d83a438b6f7fc2: Tablet Copy: opened 0 blocks and 1 log segments
I20260809 12:10:40.886792 8145 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total
I20260809 12:10:40.891203 888 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskgH5Pjm/build/asan/bin/kudu with pid 7611
I20260809 12:10:40.898515 8145 tablet_copy_client.cc:791] T 00000000000000000000000000000000 P e2a7b439c3084cd680be92ac6d1925db: tablet copy: Starting download of 0 data blocks...
I20260809 12:10:40.899061 8145 tablet_copy_client.cc:655] T 00000000000000000000000000000000 P e2a7b439c3084cd680be92ac6d1925db: tablet copy: Starting download of 1 WAL segments...
I20260809 12:10:40.902160 8145 tablet_copy_client.cc:531] T 00000000000000000000000000000000 P e2a7b439c3084cd680be92ac6d1925db: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20260809 12:10:40.907374 7569 tablet_copy_service.cc:342] P 9cd5d41083d147f788d83a438b6f7fc2: Request end of tablet copy session e2a7b439c3084cd680be92ac6d1925db-00000000000000000000000000000000 received from {username='oryx', principal='oryx/127.0.222.60@KRBTEST.COM'} at 127.0.0.1:33080
I20260809 12:10:40.907552 7569 tablet_copy_service.cc:433] P 9cd5d41083d147f788d83a438b6f7fc2: ending tablet copy session e2a7b439c3084cd680be92ac6d1925db-00000000000000000000000000000000 on tablet 00000000000000000000000000000000 with peer e2a7b439c3084cd680be92ac6d1925db
I20260809 12:10:40.916837 888 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskgH5Pjm/build/asan/bin/kudu
/tmp/dist-test-taskgH5Pjm/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskgH5Pjm/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786277382073938-888-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskgH5Pjm/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786277382073938-888-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskgH5Pjm/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786277382073938-888-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskgH5Pjm/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786277382073938-888-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.222.1:43733
--local_ip_for_outbound_sockets=127.0.222.1
--tserver_master_addrs=127.0.222.62:41983
--webserver_port=33359
--webserver_interface=127.0.222.1
--keytab_file=/tmp/dist-test-taskgH5Pjm/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786277382073938-888-0/krb5kdc/oryx_127.0.222.1.keytab
--principal=oryx/127.0.222.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.222.20:38069
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_master_addrs=127.0.222.62:41983,127.0.222.61:32831
--tserver_master_addrs=127.0.222.62:41983,127.0.222.61:32831,127.0.222.60:39543 with env {KRB5CCNAME=/tmp/dist-test-taskgH5Pjm/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786277382073938-888-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskgH5Pjm/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786277382073938-888-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskgH5Pjm/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786277382073938-888-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260809 12:10:40.920467 8145 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260809 12:10:40.924618 8355 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 12:10:40.927815 8356 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 12:10:40.936174 8360 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260809 12:10:40.936431 8145 server_base.cc:1034] running on GCE node
I20260809 12:10:40.936949 8145 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260809 12:10:40.937678 8145 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260809 12:10:40.938892 8145 hybrid_clock.cc:648] HybridClock initialized: now 1786277440938817 us; error 67 us; skew 500 ppm
I20260809 12:10:40.939194 8145 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 09 12:10:40 dist-test-slave-2w3w krb5kdc[7502](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786277440, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.222.60@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260809 12:10:40.941501 8145 init.cc:377] Logged in from keytab as oryx/127.0.222.60@KRBTEST.COM (short username oryx)
I20260809 12:10:40.948014 8145 webserver.cc:460] Webserver started at http://127.0.222.60:37771/ using document root <none> and password file <none>
I20260809 12:10:40.948398 8145 fs_manager.cc:359] Metadata directory not provided
I20260809 12:10:40.948541 8145 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260809 12:10:40.951911 8145 fs_manager.cc:709] Time spent opening directory manager: real 0.002s user 0.003s sys 0.000s
I20260809 12:10:40.954044 8368 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260809 12:10:40.954456 8145 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260809 12:10:40.954627 8145 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskgH5Pjm/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786277382073938-888-0/minicluster-data/master-2/data,/tmp/dist-test-taskgH5Pjm/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786277382073938-888-0/minicluster-data/master-2/wal
uuid: "e2a7b439c3084cd680be92ac6d1925db"
format_stamp: "Formatted at 2026-08-09 12:10:40 on dist-test-slave-2w3w"
server_key: "67dabfc5aa05d4447bd46bb4b26ce89c"
server_key_iv: "ca9f445d4b4bb45f49d87f7e3864c8d0"
server_key_version: "encryptionkey@0"
I20260809 12:10:40.954897 8145 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskgH5Pjm/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786277382073938-888-0/minicluster-data/master-2/wal
metadata directory: /tmp/dist-test-taskgH5Pjm/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786277382073938-888-0/minicluster-data/master-2/wal
1 data directories: /tmp/dist-test-taskgH5Pjm/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786277382073938-888-0/minicluster-data/master-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260809 12:10:40.982093 8145 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260809 12:10:40.986691 8145 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260809 12:10:41.015308 8145 rpc_server.cc:307] RPC server started. Bound to: 127.0.222.60:39543
I20260809 12:10:41.015399 8420 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.222.60:39543 every 8 connection(s)
I20260809 12:10:41.016533 8145 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskgH5Pjm/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786277382073938-888-0/minicluster-data/master-2/data/info.pb
I20260809 12:10:41.022202 8421 sys_catalog.cc:263] Verifying existing consensus state
I20260809 12:10:41.026273 8421 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P e2a7b439c3084cd680be92ac6d1925db: Bootstrap starting.
I20260809 12:10:41.045570 8421 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P e2a7b439c3084cd680be92ac6d1925db: Bootstrap replayed 1/1 log segments. Stats: ops{read=7 overwritten=0 applied=7 ignored=0} inserts{seen=3 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260809 12:10:41.046154 8421 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P e2a7b439c3084cd680be92ac6d1925db: Bootstrap complete.
I20260809 12:10:41.047297 8421 raft_consensus.cc:348] T 00000000000000000000000000000000 P e2a7b439c3084cd680be92ac6d1925db [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "9cd5d41083d147f788d83a438b6f7fc2" member_type: VOTER last_known_addr { host: "127.0.222.62" port: 41983 } } peers { permanent_uuid: "cbbb52854812417da2325b27e509812d" member_type: VOTER last_known_addr { host: "127.0.222.61" port: 32831 } attrs { promote: false } } peers { permanent_uuid: "e2a7b439c3084cd680be92ac6d1925db" member_type: NON_VOTER last_known_addr { host: "127.0.222.60" port: 39543 } attrs { promote: true } }
I20260809 12:10:41.047593 8421 raft_consensus.cc:729] T 00000000000000000000000000000000 P e2a7b439c3084cd680be92ac6d1925db [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: e2a7b439c3084cd680be92ac6d1925db, State: Initialized, Role: LEARNER
I20260809 12:10:41.047844 8421 consensus_queue.cc:260] T 00000000000000000000000000000000 P e2a7b439c3084cd680be92ac6d1925db [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 7, Last appended: 1.7, Last appended by leader: 7, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "9cd5d41083d147f788d83a438b6f7fc2" member_type: VOTER last_known_addr { host: "127.0.222.62" port: 41983 } } peers { permanent_uuid: "cbbb52854812417da2325b27e509812d" member_type: VOTER last_known_addr { host: "127.0.222.61" port: 32831 } attrs { promote: false } } peers { permanent_uuid: "e2a7b439c3084cd680be92ac6d1925db" member_type: NON_VOTER last_known_addr { host: "127.0.222.60" port: 39543 } attrs { promote: true } }
I20260809 12:10:41.054574 8424 sys_catalog.cc:455] T 00000000000000000000000000000000 P e2a7b439c3084cd680be92ac6d1925db [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 committed_config { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "9cd5d41083d147f788d83a438b6f7fc2" member_type: VOTER last_known_addr { host: "127.0.222.62" port: 41983 } } peers { permanent_uuid: "cbbb52854812417da2325b27e509812d" member_type: VOTER last_known_addr { host: "127.0.222.61" port: 32831 } attrs { promote: false } } peers { permanent_uuid: "e2a7b439c3084cd680be92ac6d1925db" member_type: NON_VOTER last_known_addr { host: "127.0.222.60" port: 39543 } attrs { promote: true } } }
I20260809 12:10:41.055194 8424 sys_catalog.cc:458] T 00000000000000000000000000000000 P e2a7b439c3084cd680be92ac6d1925db [sys.catalog]: This master's current role is: LEARNER
I20260809 12:10:41.055024 8421 sys_catalog.cc:564] T 00000000000000000000000000000000 P e2a7b439c3084cd680be92ac6d1925db [sys.catalog]: configured and running, proceeding with master startup.
I20260809 12:10:41.067986 8435 catalog_manager.cc:1259] Loaded cluster ID: 282dd341a1ea4ea895575dd472644e9f
I20260809 12:10:41.068118 8435 catalog_manager.cc:1552] T 00000000000000000000000000000000 P e2a7b439c3084cd680be92ac6d1925db: loading cluster ID for follower catalog manager: success
I20260809 12:10:41.070781 8435 catalog_manager.cc:1574] T 00000000000000000000000000000000 P e2a7b439c3084cd680be92ac6d1925db: acquiring CA information for follower catalog manager: success
I20260809 12:10:41.072693 8435 catalog_manager.cc:1602] T 00000000000000000000000000000000 P e2a7b439c3084cd680be92ac6d1925db: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260809 12:10:41.266477 8311 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 12:10:41.253031 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.222.60:39543 (local address 127.0.0.1:42466)
0809 12:10:41.253356 (+ 325us) negotiation.cc:107] Waiting for socket to connect
0809 12:10:41.253379 (+ 23us) client_negotiation.cc:174] Beginning negotiation
0809 12:10:41.253560 (+ 181us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 12:10:41.256087 (+ 2527us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 12:10:41.256095 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 12:10:41.256136 (+ 41us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0809 12:10:41.256419 (+ 283us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 12:10:41.256437 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 12:10:41.259097 (+ 2660us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 12:10:41.259105 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 12:10:41.261792 (+ 2687us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 12:10:41.261822 (+ 30us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 12:10:41.264516 (+ 2694us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 12:10:41.264525 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 12:10:41.265414 (+ 889us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 12:10:41.265510 (+ 96us) client_negotiation.cc:769] Sending connection context
0809 12:10:41.265623 (+ 113us) client_negotiation.cc:240] Negotiation successful
0809 12:10:41.265854 (+ 231us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":168}
I20260809 12:10:41.268019 8436 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 12:10:41.253144 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42466 (local address 127.0.222.60:39543)
0809 12:10:41.255579 (+ 2435us) server_negotiation.cc:206] Beginning negotiation
0809 12:10:41.255587 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0809 12:10:41.255614 (+ 27us) server_negotiation.cc:407] Connection header received
0809 12:10:41.255709 (+ 95us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 12:10:41.255713 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 12:10:41.255853 (+ 140us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 12:10:41.256018 (+ 165us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0809 12:10:41.256666 (+ 648us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 12:10:41.258845 (+ 2179us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 12:10:41.262053 (+ 3208us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 12:10:41.264264 (+ 2211us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 12:10:41.264473 (+ 209us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 12:10:41.264870 (+ 397us) server_negotiation.cc:1035] Waiting for connection context
0809 12:10:41.267009 (+ 2139us) server_negotiation.cc:299] Negotiation successful
0809 12:10:41.267234 (+ 225us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":2248,"thread_start_us":2052,"threads_started":1}
I20260809 12:10:41.269177 8396 raft_consensus.cc:1203] T 00000000000000000000000000000000 P e2a7b439c3084cd680be92ac6d1925db [term 1 LEARNER]: Deduplicated request from leader. Original: 1.6->[1.7-1.7] Dedup: 1.7->[]
I20260809 12:10:41.269908 8424 sys_catalog.cc:455] T 00000000000000000000000000000000 P e2a7b439c3084cd680be92ac6d1925db [sys.catalog]: SysCatalogTable state changed. Reason: New leader 9cd5d41083d147f788d83a438b6f7fc2. Latest consensus state: current_term: 1 leader_uuid: "9cd5d41083d147f788d83a438b6f7fc2" committed_config { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "9cd5d41083d147f788d83a438b6f7fc2" member_type: VOTER last_known_addr { host: "127.0.222.62" port: 41983 } } peers { permanent_uuid: "cbbb52854812417da2325b27e509812d" member_type: VOTER last_known_addr { host: "127.0.222.61" port: 32831 } attrs { promote: false } } peers { permanent_uuid: "e2a7b439c3084cd680be92ac6d1925db" member_type: NON_VOTER last_known_addr { host: "127.0.222.60" port: 39543 } attrs { promote: true } } }
I20260809 12:10:41.270278 8424 sys_catalog.cc:458] T 00000000000000000000000000000000 P e2a7b439c3084cd680be92ac6d1925db [sys.catalog]: This master's current role is: LEARNER
W20260809 12:10:41.383193 8357 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260809 12:10:41.383630 8357 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260809 12:10:41.383906 8357 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260809 12:10:41.393424 8357 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260809 12:10:41.393651 8357 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260809 12:10:41.393966 8357 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.222.1
I20260809 12:10:41.406524 8357 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.222.20:38069
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskgH5Pjm/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786277382073938-888-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskgH5Pjm/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786277382073938-888-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskgH5Pjm/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786277382073938-888-0/krb5kdc/oryx_127.0.222.1.keytab
--principal=oryx/127.0.222.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.222.1:43733
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskgH5Pjm/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786277382073938-888-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.222.1
--webserver_port=33359
--tserver_master_addrs=127.0.222.62:41983,127.0.222.61:32831,127.0.222.60:39543
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.222.1
--log_dir=/tmp/dist-test-taskgH5Pjm/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786277382073938-888-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 09 Aug 2026 11:06:41 UTC on ccycloud.quasar-bwoogy.root.comops.site
build id 4575991
ASAN enabled
I20260809 12:10:41.408336 8357 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260809 12:10:41.409758 8357 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260809 12:10:41.427930 8441 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 12:10:41.428414 8442 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 12:10:41.428588 8444 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260809 12:10:41.431104 8357 server_base.cc:1034] running on GCE node
I20260809 12:10:41.431869 8357 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260809 12:10:41.433118 8357 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260809 12:10:41.434427 8357 hybrid_clock.cc:648] HybridClock initialized: now 1786277441434329 us; error 55 us; skew 500 ppm
I20260809 12:10:41.434867 8357 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 09 12:10:41 dist-test-slave-2w3w krb5kdc[7502](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786277441, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.222.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260809 12:10:41.439321 8357 init.cc:377] Logged in from keytab as oryx/127.0.222.1@KRBTEST.COM (short username oryx)
I20260809 12:10:41.441344 8357 webserver.cc:460] Webserver started at http://127.0.222.1:33359/ using document root <none> and password file <none>
I20260809 12:10:41.442142 8357 fs_manager.cc:359] Metadata directory not provided
I20260809 12:10:41.442350 8357 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260809 12:10:41.450109 8357 fs_manager.cc:709] Time spent opening directory manager: real 0.005s user 0.001s sys 0.005s
I20260809 12:10:41.453995 8451 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260809 12:10:41.455734 8357 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.003s sys 0.000s
I20260809 12:10:41.455966 8357 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskgH5Pjm/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786277382073938-888-0/minicluster-data/ts-0/data,/tmp/dist-test-taskgH5Pjm/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786277382073938-888-0/minicluster-data/ts-0/wal
uuid: "df0db1a138554d00956d87f5c172ee10"
format_stamp: "Formatted at 2026-08-09 12:10:38 on dist-test-slave-2w3w"
server_key: "b6a9d17eadb03dcfff0fbce8a306b594"
server_key_iv: "c3eda1fa16fab7b7493b77c4d3a1da80"
server_key_version: "encryptionkey@0"
I20260809 12:10:41.457547 8357 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskgH5Pjm/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786277382073938-888-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskgH5Pjm/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786277382073938-888-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskgH5Pjm/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786277382073938-888-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260809 12:10:41.476727 8357 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260809 12:10:41.482177 8357 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260809 12:10:41.482602 8357 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260809 12:10:41.484146 8357 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260809 12:10:41.486198 8357 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260809 12:10:41.486352 8357 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260809 12:10:41.486507 8357 ts_tablet_manager.cc:610] Registered 0 tablets
I20260809 12:10:41.486630 8357 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260809 12:10:41.569631 8357 rpc_server.cc:307] RPC server started. Bound to: 127.0.222.1:43733
I20260809 12:10:41.569823 8570 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.222.1:43733 every 8 connection(s)
I20260809 12:10:41.571466 8357 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskgH5Pjm/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786277382073938-888-0/minicluster-data/ts-0/data/info.pb
I20260809 12:10:41.580232 888 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskgH5Pjm/build/asan/bin/kudu as pid 8357
I20260809 12:10:41.580860 888 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskgH5Pjm/build/asan/bin/kudu pid 7752
Aug 09 12:10:41 dist-test-slave-2w3w krb5kdc[7502](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786277441, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.222.1@KRBTEST.COM for oryx/127.0.222.61@KRBTEST.COM
Aug 09 12:10:41 dist-test-slave-2w3w krb5kdc[7502](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786277441, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.222.1@KRBTEST.COM for oryx/127.0.222.60@KRBTEST.COM
Aug 09 12:10:41 dist-test-slave-2w3w krb5kdc[7502](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786277437, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.0.222.2@KRBTEST.COM
Aug 09 12:10:41 dist-test-slave-2w3w krb5kdc[7502](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786277441, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.222.1@KRBTEST.COM for oryx/127.0.222.62@KRBTEST.COM
I20260809 12:10:41.617512 8576 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 12:10:41.575847 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.222.1:59527 (local address 127.0.222.61:32831)
0809 12:10:41.576611 (+ 764us) server_negotiation.cc:206] Beginning negotiation
0809 12:10:41.576618 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0809 12:10:41.581040 (+ 4422us) server_negotiation.cc:407] Connection header received
0809 12:10:41.589633 (+ 8593us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 12:10:41.589643 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 12:10:41.589786 (+ 143us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 12:10:41.589997 (+ 211us) server_negotiation.cc:226] Negotiated authn=SASL
0809 12:10:41.594132 (+ 4135us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 12:10:41.595600 (+ 1468us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 12:10:41.599551 (+ 3951us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 12:10:41.600458 (+ 907us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 12:10:41.600538 (+ 80us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 12:10:41.608122 (+ 7584us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 12:10:41.608130 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 12:10:41.608136 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0809 12:10:41.608184 (+ 48us) server_negotiation.cc:922] calling sasl_server_start()
0809 12:10:41.612702 (+ 4518us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 12:10:41.613879 (+ 1177us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 12:10:41.613886 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 12:10:41.613890 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0809 12:10:41.614071 (+ 181us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 12:10:41.614808 (+ 737us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 12:10:41.614813 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 12:10:41.614817 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0809 12:10:41.615357 (+ 540us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 12:10:41.615489 (+ 132us) server_negotiation.cc:1035] Waiting for connection context
0809 12:10:41.616737 (+ 1248us) server_negotiation.cc:299] Negotiation successful
0809 12:10:41.616980 (+ 243us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":599,"thread_start_us":449,"threads_started":1}
I20260809 12:10:41.618199 8436 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 12:10:41.575719 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.222.1:58969 (local address 127.0.222.60:39543)
0809 12:10:41.575923 (+ 204us) server_negotiation.cc:206] Beginning negotiation
0809 12:10:41.575961 (+ 38us) server_negotiation.cc:399] Waiting for connection header
0809 12:10:41.587236 (+ 11275us) server_negotiation.cc:407] Connection header received
0809 12:10:41.589553 (+ 2317us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 12:10:41.589559 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 12:10:41.589667 (+ 108us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 12:10:41.589808 (+ 141us) server_negotiation.cc:226] Negotiated authn=SASL
0809 12:10:41.593873 (+ 4065us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 12:10:41.595487 (+ 1614us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 12:10:41.600225 (+ 4738us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 12:10:41.601309 (+ 1084us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 12:10:41.601525 (+ 216us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 12:10:41.608206 (+ 6681us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 12:10:41.608214 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 12:10:41.608220 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0809 12:10:41.608277 (+ 57us) server_negotiation.cc:922] calling sasl_server_start()
0809 12:10:41.614309 (+ 6032us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 12:10:41.615394 (+ 1085us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 12:10:41.615402 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 12:10:41.615408 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0809 12:10:41.615632 (+ 224us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 12:10:41.616188 (+ 556us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 12:10:41.616193 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 12:10:41.616196 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0809 12:10:41.616624 (+ 428us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 12:10:41.616774 (+ 150us) server_negotiation.cc:1035] Waiting for connection context
0809 12:10:41.617344 (+ 570us) server_negotiation.cc:299] Negotiation successful
0809 12:10:41.617712 (+ 368us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":77}
I20260809 12:10:41.620752 8578 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 12:10:41.576773 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.222.61:32831 (local address 127.0.222.1:59527)
0809 12:10:41.580703 (+ 3930us) negotiation.cc:107] Waiting for socket to connect
0809 12:10:41.580766 (+ 63us) client_negotiation.cc:174] Beginning negotiation
0809 12:10:41.589251 (+ 8485us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 12:10:41.590404 (+ 1153us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 12:10:41.590420 (+ 16us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 12:10:41.592154 (+ 1734us) client_negotiation.cc:189] Negotiated authn=SASL
0809 12:10:41.593899 (+ 1745us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 12:10:41.593910 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 12:10:41.595882 (+ 1972us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 12:10:41.595889 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 12:10:41.599364 (+ 3475us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 12:10:41.599387 (+ 23us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 12:10:41.601150 (+ 1763us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 12:10:41.601161 (+ 11us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 12:10:41.601675 (+ 514us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 12:10:41.602763 (+ 1088us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0809 12:10:41.602796 (+ 33us) client_negotiation.cc:656] calling sasl_client_start()
0809 12:10:41.607839 (+ 5043us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0809 12:10:41.612875 (+ 5036us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0809 12:10:41.612885 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0809 12:10:41.612903 (+ 18us) client_negotiation.cc:760] calling sasl_client_step()
0809 12:10:41.613710 (+ 807us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0809 12:10:41.614231 (+ 521us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0809 12:10:41.614236 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0809 12:10:41.614239 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0809 12:10:41.614619 (+ 380us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0809 12:10:41.615641 (+ 1022us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0809 12:10:41.615649 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0809 12:10:41.616145 (+ 496us) client_negotiation.cc:769] Sending connection context
0809 12:10:41.617894 (+ 1749us) client_negotiation.cc:240] Negotiation successful
0809 12:10:41.620088 (+ 2194us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":2943,"mutex_wait_us":168,"spinlock_wait_cycles":21376,"thread_start_us":2497,"threads_started":1}
I20260809 12:10:41.620800 8579 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 12:10:41.578751 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.222.60:39543 (local address 127.0.222.1:58969)
0809 12:10:41.587036 (+ 8285us) negotiation.cc:107] Waiting for socket to connect
0809 12:10:41.587057 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0809 12:10:41.589209 (+ 2152us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 12:10:41.590401 (+ 1192us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 12:10:41.590419 (+ 18us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 12:10:41.591732 (+ 1313us) client_negotiation.cc:189] Negotiated authn=SASL
0809 12:10:41.593613 (+ 1881us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 12:10:41.593687 (+ 74us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 12:10:41.596216 (+ 2529us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 12:10:41.596221 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 12:10:41.600060 (+ 3839us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 12:10:41.600075 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 12:10:41.601829 (+ 1754us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 12:10:41.601835 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 12:10:41.602102 (+ 267us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 12:10:41.602948 (+ 846us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0809 12:10:41.603003 (+ 55us) client_negotiation.cc:656] calling sasl_client_start()
0809 12:10:41.607844 (+ 4841us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0809 12:10:41.614493 (+ 6649us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0809 12:10:41.614498 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0809 12:10:41.614501 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0809 12:10:41.615238 (+ 737us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0809 12:10:41.615774 (+ 536us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0809 12:10:41.615778 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0809 12:10:41.615781 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0809 12:10:41.616043 (+ 262us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0809 12:10:41.616856 (+ 813us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0809 12:10:41.616860 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0809 12:10:41.617036 (+ 176us) client_negotiation.cc:769] Sending connection context
0809 12:10:41.618531 (+ 1495us) client_negotiation.cc:240] Negotiation successful
0809 12:10:41.619012 (+ 481us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":8083,"mutex_wait_us":46,"thread_start_us":7873,"threads_started":1}
I20260809 12:10:41.623306 8577 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 12:10:41.577488 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.222.1:52809 (local address 127.0.222.62:41983)
0809 12:10:41.578471 (+ 983us) server_negotiation.cc:206] Beginning negotiation
0809 12:10:41.578478 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0809 12:10:41.595484 (+ 17006us) server_negotiation.cc:407] Connection header received
0809 12:10:41.595713 (+ 229us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 12:10:41.595721 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 12:10:41.595812 (+ 91us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 12:10:41.596018 (+ 206us) server_negotiation.cc:226] Negotiated authn=SASL
0809 12:10:41.598306 (+ 2288us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 12:10:41.599288 (+ 982us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 12:10:41.604018 (+ 4730us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 12:10:41.604618 (+ 600us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 12:10:41.604763 (+ 145us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 12:10:41.612486 (+ 7723us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 12:10:41.612495 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 12:10:41.612501 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0809 12:10:41.612541 (+ 40us) server_negotiation.cc:922] calling sasl_server_start()
0809 12:10:41.617358 (+ 4817us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 12:10:41.619041 (+ 1683us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 12:10:41.619049 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 12:10:41.619054 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0809 12:10:41.619247 (+ 193us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 12:10:41.620162 (+ 915us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 12:10:41.620167 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 12:10:41.620170 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0809 12:10:41.620557 (+ 387us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 12:10:41.620707 (+ 150us) server_negotiation.cc:1035] Waiting for connection context
0809 12:10:41.622485 (+ 1778us) server_negotiation.cc:299] Negotiation successful
0809 12:10:41.622803 (+ 318us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":819,"thread_start_us":622,"threads_started":1}
I20260809 12:10:41.623672 8580 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 12:10:41.578990 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.222.62:41983 (local address 127.0.222.1:52809)
0809 12:10:41.595334 (+ 16344us) negotiation.cc:107] Waiting for socket to connect
0809 12:10:41.595354 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0809 12:10:41.595539 (+ 185us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 12:10:41.596213 (+ 674us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 12:10:41.596218 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 12:10:41.597492 (+ 1274us) client_negotiation.cc:189] Negotiated authn=SASL
0809 12:10:41.598139 (+ 647us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 12:10:41.598153 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 12:10:41.600971 (+ 2818us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 12:10:41.600977 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 12:10:41.603832 (+ 2855us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 12:10:41.603862 (+ 30us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 12:10:41.604906 (+ 1044us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 12:10:41.604914 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 12:10:41.607053 (+ 2139us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 12:10:41.607867 (+ 814us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0809 12:10:41.607886 (+ 19us) client_negotiation.cc:656] calling sasl_client_start()
0809 12:10:41.612207 (+ 4321us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0809 12:10:41.618436 (+ 6229us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0809 12:10:41.618441 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0809 12:10:41.618445 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0809 12:10:41.618878 (+ 433us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0809 12:10:41.619839 (+ 961us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0809 12:10:41.619843 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0809 12:10:41.619845 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0809 12:10:41.620028 (+ 183us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0809 12:10:41.621010 (+ 982us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0809 12:10:41.621015 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0809 12:10:41.621820 (+ 805us) client_negotiation.cc:769] Sending connection context
0809 12:10:41.621983 (+ 163us) client_negotiation.cc:240] Negotiation successful
0809 12:10:41.622276 (+ 293us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":16178,"mutex_wait_us":41,"thread_start_us":15949,"threads_started":1}
I20260809 12:10:41.623737 8574 heartbeater.cc:347] Connected to a master server at 127.0.222.61:32831
I20260809 12:10:41.623821 8572 heartbeater.cc:347] Connected to a master server at 127.0.222.60:39543
I20260809 12:10:41.624214 8572 heartbeater.cc:464] Registering TS with master...
I20260809 12:10:41.624233 8574 heartbeater.cc:464] Registering TS with master...
I20260809 12:10:41.625329 8573 heartbeater.cc:347] Connected to a master server at 127.0.222.62:41983
I20260809 12:10:41.625602 8573 heartbeater.cc:464] Registering TS with master...
I20260809 12:10:41.626029 8574 heartbeater.cc:511] Master 127.0.222.61:32831 requested a full tablet report, sending...
I20260809 12:10:41.626101 8572 heartbeater.cc:511] Master 127.0.222.60:39543 requested a full tablet report, sending...
I20260809 12:10:41.626322 8582 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 12:10:41.583096 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35418 (local address 127.0.222.2:33587)
0809 12:10:41.593720 (+ 10624us) server_negotiation.cc:206] Beginning negotiation
0809 12:10:41.593733 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0809 12:10:41.593787 (+ 54us) server_negotiation.cc:407] Connection header received
0809 12:10:41.593888 (+ 101us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 12:10:41.593901 (+ 13us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 12:10:41.594192 (+ 291us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 12:10:41.595826 (+ 1634us) server_negotiation.cc:226] Negotiated authn=SASL
0809 12:10:41.599002 (+ 3176us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 12:10:41.600611 (+ 1609us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 12:10:41.603309 (+ 2698us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 12:10:41.604452 (+ 1143us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 12:10:41.605410 (+ 958us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 12:10:41.612911 (+ 7501us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 12:10:41.612924 (+ 13us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 12:10:41.612937 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0809 12:10:41.612970 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0809 12:10:41.618302 (+ 5332us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 12:10:41.620226 (+ 1924us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 12:10:41.620236 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 12:10:41.620239 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0809 12:10:41.620416 (+ 177us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 12:10:41.622899 (+ 2483us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 12:10:41.622904 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 12:10:41.622908 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0809 12:10:41.623449 (+ 541us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 12:10:41.625194 (+ 1745us) server_negotiation.cc:1035] Waiting for connection context
0809 12:10:41.625425 (+ 231us) server_negotiation.cc:299] Negotiation successful
0809 12:10:41.625758 (+ 333us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":10368,"thread_start_us":10204,"threads_started":1}
I20260809 12:10:41.626735 8573 heartbeater.cc:511] Master 127.0.222.62:41983 requested a full tablet report, sending...
I20260809 12:10:41.628197 7883 generic_service.cc:196] Checking for leaks (request via RPC)
I20260809 12:10:41.628887 8183 ts_manager.cc:194] Registered new tserver with Master: df0db1a138554d00956d87f5c172ee10 (127.0.222.1:43733)
I20260809 12:10:41.629136 8385 ts_manager.cc:194] Registered new tserver with Master: df0db1a138554d00956d87f5c172ee10 (127.0.222.1:43733)
I20260809 12:10:41.630734 7547 ts_manager.cc:194] Re-registered known tserver with Master: df0db1a138554d00956d87f5c172ee10 (127.0.222.1:43733)
I20260809 12:10:41.632334 7547 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.0.222.1@KRBTEST.COM'} at 127.0.222.1:52809
I20260809 12:10:41.670258 8309 sys_catalog.cc:455] T 00000000000000000000000000000000 P 9cd5d41083d147f788d83a438b6f7fc2 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "9cd5d41083d147f788d83a438b6f7fc2" committed_config { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "9cd5d41083d147f788d83a438b6f7fc2" member_type: VOTER last_known_addr { host: "127.0.222.62" port: 41983 } } peers { permanent_uuid: "cbbb52854812417da2325b27e509812d" member_type: VOTER last_known_addr { host: "127.0.222.61" port: 32831 } attrs { promote: false } } peers { permanent_uuid: "e2a7b439c3084cd680be92ac6d1925db" member_type: NON_VOTER last_known_addr { host: "127.0.222.60" port: 39543 } attrs { promote: true } } }
I20260809 12:10:41.670724 8309 sys_catalog.cc:458] T 00000000000000000000000000000000 P 9cd5d41083d147f788d83a438b6f7fc2 [sys.catalog]: This master's current role is: LEADER
I20260809 12:10:41.672672 8584 raft_consensus.cc:1050] T 00000000000000000000000000000000 P 9cd5d41083d147f788d83a438b6f7fc2: attempting to promote NON_VOTER e2a7b439c3084cd680be92ac6d1925db to VOTER
I20260809 12:10:41.673385 8584 consensus_queue.cc:237] T 00000000000000000000000000000000 P 9cd5d41083d147f788d83a438b6f7fc2 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 1.7, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "9cd5d41083d147f788d83a438b6f7fc2" member_type: VOTER last_known_addr { host: "127.0.222.62" port: 41983 } } peers { permanent_uuid: "cbbb52854812417da2325b27e509812d" member_type: VOTER last_known_addr { host: "127.0.222.61" port: 32831 } attrs { promote: false } } peers { permanent_uuid: "e2a7b439c3084cd680be92ac6d1925db" member_type: VOTER last_known_addr { host: "127.0.222.60" port: 39543 } attrs { promote: false } }
I20260809 12:10:41.675714 8396 raft_consensus.cc:1261] T 00000000000000000000000000000000 P e2a7b439c3084cd680be92ac6d1925db [term 1 LEARNER]: Refusing update from remote peer 9cd5d41083d147f788d83a438b6f7fc2: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 1 index: 8. (index mismatch)
I20260809 12:10:41.676079 8193 raft_consensus.cc:1261] T 00000000000000000000000000000000 P cbbb52854812417da2325b27e509812d [term 1 FOLLOWER]: Refusing update from remote peer 9cd5d41083d147f788d83a438b6f7fc2: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 1 index: 8. (index mismatch)
I20260809 12:10:41.676563 8584 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 9cd5d41083d147f788d83a438b6f7fc2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e2a7b439c3084cd680be92ac6d1925db" member_type: VOTER last_known_addr { host: "127.0.222.60" port: 39543 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20260809 12:10:41.677127 8309 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 9cd5d41083d147f788d83a438b6f7fc2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "cbbb52854812417da2325b27e509812d" member_type: VOTER last_known_addr { host: "127.0.222.61" port: 32831 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20260809 12:10:41.681653 8584 raft_consensus.cc:2932] T 00000000000000000000000000000000 P 9cd5d41083d147f788d83a438b6f7fc2 [term 1 LEADER]: Committing config change with OpId 1.8: config changed from index 7 to 8, e2a7b439c3084cd680be92ac6d1925db (127.0.222.60) changed from NON_VOTER to VOTER. New config: { opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "9cd5d41083d147f788d83a438b6f7fc2" member_type: VOTER last_known_addr { host: "127.0.222.62" port: 41983 } } peers { permanent_uuid: "cbbb52854812417da2325b27e509812d" member_type: VOTER last_known_addr { host: "127.0.222.61" port: 32831 } attrs { promote: false } } peers { permanent_uuid: "e2a7b439c3084cd680be92ac6d1925db" member_type: VOTER last_known_addr { host: "127.0.222.60" port: 39543 } attrs { promote: false } } }
I20260809 12:10:41.682466 8193 raft_consensus.cc:2932] T 00000000000000000000000000000000 P cbbb52854812417da2325b27e509812d [term 1 FOLLOWER]: Committing config change with OpId 1.8: config changed from index 7 to 8, e2a7b439c3084cd680be92ac6d1925db (127.0.222.60) changed from NON_VOTER to VOTER. New config: { opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "9cd5d41083d147f788d83a438b6f7fc2" member_type: VOTER last_known_addr { host: "127.0.222.62" port: 41983 } } peers { permanent_uuid: "cbbb52854812417da2325b27e509812d" member_type: VOTER last_known_addr { host: "127.0.222.61" port: 32831 } attrs { promote: false } } peers { permanent_uuid: "e2a7b439c3084cd680be92ac6d1925db" member_type: VOTER last_known_addr { host: "127.0.222.60" port: 39543 } attrs { promote: false } } }
I20260809 12:10:41.687273 8584 sys_catalog.cc:455] T 00000000000000000000000000000000 P 9cd5d41083d147f788d83a438b6f7fc2 [sys.catalog]: SysCatalogTable state changed. Reason: Config change replication complete. Latest consensus state: current_term: 1 leader_uuid: "9cd5d41083d147f788d83a438b6f7fc2" committed_config { opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "9cd5d41083d147f788d83a438b6f7fc2" member_type: VOTER last_known_addr { host: "127.0.222.62" port: 41983 } } peers { permanent_uuid: "cbbb52854812417da2325b27e509812d" member_type: VOTER last_known_addr { host: "127.0.222.61" port: 32831 } attrs { promote: false } } peers { permanent_uuid: "e2a7b439c3084cd680be92ac6d1925db" member_type: VOTER last_known_addr { host: "127.0.222.60" port: 39543 } attrs { promote: false } } }
I20260809 12:10:41.687475 8309 sys_catalog.cc:455] T 00000000000000000000000000000000 P 9cd5d41083d147f788d83a438b6f7fc2 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "9cd5d41083d147f788d83a438b6f7fc2" committed_config { opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "9cd5d41083d147f788d83a438b6f7fc2" member_type: VOTER last_known_addr { host: "127.0.222.62" port: 41983 } } peers { permanent_uuid: "cbbb52854812417da2325b27e509812d" member_type: VOTER last_known_addr { host: "127.0.222.61" port: 32831 } attrs { promote: false } } peers { permanent_uuid: "e2a7b439c3084cd680be92ac6d1925db" member_type: VOTER last_known_addr { host: "127.0.222.60" port: 39543 } attrs { promote: false } } }
I20260809 12:10:41.687922 8309 sys_catalog.cc:458] T 00000000000000000000000000000000 P 9cd5d41083d147f788d83a438b6f7fc2 [sys.catalog]: This master's current role is: LEADER
I20260809 12:10:41.688244 8309 sys_catalog.cc:455] T 00000000000000000000000000000000 P 9cd5d41083d147f788d83a438b6f7fc2 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "9cd5d41083d147f788d83a438b6f7fc2" committed_config { opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "9cd5d41083d147f788d83a438b6f7fc2" member_type: VOTER last_known_addr { host: "127.0.222.62" port: 41983 } } peers { permanent_uuid: "cbbb52854812417da2325b27e509812d" member_type: VOTER last_known_addr { host: "127.0.222.61" port: 32831 } attrs { promote: false } } peers { permanent_uuid: "e2a7b439c3084cd680be92ac6d1925db" member_type: VOTER last_known_addr { host: "127.0.222.60" port: 39543 } attrs { promote: false } } }
I20260809 12:10:41.688697 8309 sys_catalog.cc:458] T 00000000000000000000000000000000 P 9cd5d41083d147f788d83a438b6f7fc2 [sys.catalog]: This master's current role is: LEADER
I20260809 12:10:41.687753 8584 sys_catalog.cc:458] T 00000000000000000000000000000000 P 9cd5d41083d147f788d83a438b6f7fc2 [sys.catalog]: This master's current role is: LEADER
I20260809 12:10:41.688647 8396 raft_consensus.cc:2932] T 00000000000000000000000000000000 P e2a7b439c3084cd680be92ac6d1925db [term 1 FOLLOWER]: Committing config change with OpId 1.8: config changed from index 7 to 8, e2a7b439c3084cd680be92ac6d1925db (127.0.222.60) changed from NON_VOTER to VOTER. New config: { opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "9cd5d41083d147f788d83a438b6f7fc2" member_type: VOTER last_known_addr { host: "127.0.222.62" port: 41983 } } peers { permanent_uuid: "cbbb52854812417da2325b27e509812d" member_type: VOTER last_known_addr { host: "127.0.222.61" port: 32831 } attrs { promote: false } } peers { permanent_uuid: "e2a7b439c3084cd680be92ac6d1925db" member_type: VOTER last_known_addr { host: "127.0.222.60" port: 39543 } attrs { promote: false } } }
I20260809 12:10:41.690055 8588 sys_catalog.cc:455] T 00000000000000000000000000000000 P cbbb52854812417da2325b27e509812d [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "9cd5d41083d147f788d83a438b6f7fc2" committed_config { opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "9cd5d41083d147f788d83a438b6f7fc2" member_type: VOTER last_known_addr { host: "127.0.222.62" port: 41983 } } peers { permanent_uuid: "cbbb52854812417da2325b27e509812d" member_type: VOTER last_known_addr { host: "127.0.222.61" port: 32831 } attrs { promote: false } } peers { permanent_uuid: "e2a7b439c3084cd680be92ac6d1925db" member_type: VOTER last_known_addr { host: "127.0.222.60" port: 39543 } attrs { promote: false } } }
I20260809 12:10:41.690685 8588 sys_catalog.cc:458] T 00000000000000000000000000000000 P cbbb52854812417da2325b27e509812d [sys.catalog]: This master's current role is: FOLLOWER
I20260809 12:10:41.693820 8424 sys_catalog.cc:455] T 00000000000000000000000000000000 P e2a7b439c3084cd680be92ac6d1925db [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "9cd5d41083d147f788d83a438b6f7fc2" committed_config { opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "9cd5d41083d147f788d83a438b6f7fc2" member_type: VOTER last_known_addr { host: "127.0.222.62" port: 41983 } } peers { permanent_uuid: "cbbb52854812417da2325b27e509812d" member_type: VOTER last_known_addr { host: "127.0.222.61" port: 32831 } attrs { promote: false } } peers { permanent_uuid: "e2a7b439c3084cd680be92ac6d1925db" member_type: VOTER last_known_addr { host: "127.0.222.60" port: 39543 } attrs { promote: false } } }
I20260809 12:10:41.694273 8424 sys_catalog.cc:458] T 00000000000000000000000000000000 P e2a7b439c3084cd680be92ac6d1925db [sys.catalog]: This master's current role is: FOLLOWER
I20260809 12:10:41.723757 888 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskgH5Pjm/build/asan/bin/kudu with pid 7752
I20260809 12:10:41.744630 888 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskgH5Pjm/build/asan/bin/kudu
/tmp/dist-test-taskgH5Pjm/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskgH5Pjm/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786277382073938-888-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskgH5Pjm/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786277382073938-888-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskgH5Pjm/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786277382073938-888-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskgH5Pjm/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786277382073938-888-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.222.2:33587
--local_ip_for_outbound_sockets=127.0.222.2
--tserver_master_addrs=127.0.222.62:41983
--webserver_port=41681
--webserver_interface=127.0.222.2
--keytab_file=/tmp/dist-test-taskgH5Pjm/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786277382073938-888-0/krb5kdc/oryx_127.0.222.2.keytab
--principal=oryx/127.0.222.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.222.20:38069
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_master_addrs=127.0.222.62:41983,127.0.222.61:32831
--tserver_master_addrs=127.0.222.62:41983,127.0.222.61:32831,127.0.222.60:39543 with env {KRB5CCNAME=/tmp/dist-test-taskgH5Pjm/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786277382073938-888-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskgH5Pjm/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786277382073938-888-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskgH5Pjm/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786277382073938-888-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260809 12:10:42.091339 8589 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260809 12:10:42.091626 8589 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260809 12:10:42.091727 8589 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260809 12:10:42.099179 8589 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260809 12:10:42.099294 8589 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260809 12:10:42.099414 8589 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.222.2
I20260809 12:10:42.109007 8589 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.222.20:38069
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskgH5Pjm/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786277382073938-888-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskgH5Pjm/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786277382073938-888-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskgH5Pjm/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786277382073938-888-0/krb5kdc/oryx_127.0.222.2.keytab
--principal=oryx/127.0.222.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.222.2:33587
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskgH5Pjm/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786277382073938-888-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.222.2
--webserver_port=41681
--tserver_master_addrs=127.0.222.62:41983,127.0.222.61:32831,127.0.222.60:39543
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.222.2
--log_dir=/tmp/dist-test-taskgH5Pjm/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786277382073938-888-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 09 Aug 2026 11:06:41 UTC on ccycloud.quasar-bwoogy.root.comops.site
build id 4575991
ASAN enabled
I20260809 12:10:42.110430 8589 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260809 12:10:42.111810 8589 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260809 12:10:42.125355 8594 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 12:10:42.125339 8595 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 12:10:42.126109 8597 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260809 12:10:42.126796 8589 server_base.cc:1034] running on GCE node
I20260809 12:10:42.127498 8589 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260809 12:10:42.128486 8589 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260809 12:10:42.129711 8589 hybrid_clock.cc:648] HybridClock initialized: now 1786277442129643 us; error 62 us; skew 500 ppm
I20260809 12:10:42.130025 8589 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 09 12:10:42 dist-test-slave-2w3w krb5kdc[7502](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786277442, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.222.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260809 12:10:42.133315 8589 init.cc:377] Logged in from keytab as oryx/127.0.222.2@KRBTEST.COM (short username oryx)
I20260809 12:10:42.134874 8589 webserver.cc:460] Webserver started at http://127.0.222.2:41681/ using document root <none> and password file <none>
I20260809 12:10:42.135607 8589 fs_manager.cc:359] Metadata directory not provided
I20260809 12:10:42.135772 8589 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260809 12:10:42.141636 8589 fs_manager.cc:709] Time spent opening directory manager: real 0.004s user 0.005s sys 0.000s
I20260809 12:10:42.144490 8605 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260809 12:10:42.145834 8589 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260809 12:10:42.146018 8589 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskgH5Pjm/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786277382073938-888-0/minicluster-data/ts-1/data,/tmp/dist-test-taskgH5Pjm/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786277382073938-888-0/minicluster-data/ts-1/wal
uuid: "94413a87e0a14ef195baa2a799a9e0a8"
format_stamp: "Formatted at 2026-08-09 12:10:38 on dist-test-slave-2w3w"
server_key: "fc42e1038ea5fe088e9187d212704211"
server_key_iv: "6a728afac96572a760f842693c2562b5"
server_key_version: "encryptionkey@0"
I20260809 12:10:42.147048 8589 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskgH5Pjm/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786277382073938-888-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskgH5Pjm/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786277382073938-888-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskgH5Pjm/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786277382073938-888-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260809 12:10:42.162786 8589 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260809 12:10:42.167409 8589 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260809 12:10:42.167716 8589 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260809 12:10:42.168891 8589 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260809 12:10:42.170326 8589 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260809 12:10:42.170451 8589 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260809 12:10:42.170594 8589 ts_tablet_manager.cc:610] Registered 0 tablets
I20260809 12:10:42.170686 8589 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260809 12:10:42.204720 8589 rpc_server.cc:307] RPC server started. Bound to: 127.0.222.2:33587
I20260809 12:10:42.204777 8725 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.222.2:33587 every 8 connection(s)
I20260809 12:10:42.206144 8589 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskgH5Pjm/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786277382073938-888-0/minicluster-data/ts-1/data/info.pb
I20260809 12:10:42.212811 888 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskgH5Pjm/build/asan/bin/kudu as pid 8589
I20260809 12:10:42.213172 888 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskgH5Pjm/build/asan/bin/kudu pid 7894
Aug 09 12:10:42 dist-test-slave-2w3w krb5kdc[7502](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786277437, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.0.222.3@KRBTEST.COM
Aug 09 12:10:42 dist-test-slave-2w3w krb5kdc[7502](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786277442, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.222.2@KRBTEST.COM for oryx/127.0.222.62@KRBTEST.COM
Aug 09 12:10:42 dist-test-slave-2w3w krb5kdc[7502](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786277442, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.222.2@KRBTEST.COM for oryx/127.0.222.61@KRBTEST.COM
Aug 09 12:10:42 dist-test-slave-2w3w krb5kdc[7502](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786277442, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.222.2@KRBTEST.COM for oryx/127.0.222.60@KRBTEST.COM
I20260809 12:10:42.261953 8730 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 12:10:42.209204 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.222.2:35187 (local address 127.0.222.60:39543)
0809 12:10:42.210203 (+ 999us) server_negotiation.cc:206] Beginning negotiation
0809 12:10:42.210213 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0809 12:10:42.232129 (+ 21916us) server_negotiation.cc:407] Connection header received
0809 12:10:42.235944 (+ 3815us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 12:10:42.235950 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 12:10:42.236041 (+ 91us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 12:10:42.236127 (+ 86us) server_negotiation.cc:226] Negotiated authn=SASL
0809 12:10:42.240706 (+ 4579us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 12:10:42.241904 (+ 1198us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 12:10:42.245096 (+ 3192us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 12:10:42.245761 (+ 665us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 12:10:42.245872 (+ 111us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 12:10:42.252052 (+ 6180us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 12:10:42.252059 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 12:10:42.252079 (+ 20us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0809 12:10:42.252155 (+ 76us) server_negotiation.cc:922] calling sasl_server_start()
0809 12:10:42.256872 (+ 4717us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 12:10:42.258086 (+ 1214us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 12:10:42.258091 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 12:10:42.258095 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0809 12:10:42.258337 (+ 242us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 12:10:42.259022 (+ 685us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 12:10:42.259027 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 12:10:42.259030 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0809 12:10:42.259599 (+ 569us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 12:10:42.259785 (+ 186us) server_negotiation.cc:1035] Waiting for connection context
0809 12:10:42.261130 (+ 1345us) server_negotiation.cc:299] Negotiation successful
0809 12:10:42.261383 (+ 253us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":849,"thread_start_us":732,"threads_started":1}
I20260809 12:10:42.262089 8737 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 12:10:42.214213 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33558 (local address 127.0.222.3:37357)
0809 12:10:42.230429 (+ 16216us) server_negotiation.cc:206] Beginning negotiation
0809 12:10:42.230437 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0809 12:10:42.230466 (+ 29us) server_negotiation.cc:407] Connection header received
0809 12:10:42.230539 (+ 73us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 12:10:42.230546 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 12:10:42.230872 (+ 326us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 12:10:42.232277 (+ 1405us) server_negotiation.cc:226] Negotiated authn=SASL
0809 12:10:42.235607 (+ 3330us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 12:10:42.237879 (+ 2272us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 12:10:42.239750 (+ 1871us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 12:10:42.240587 (+ 837us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 12:10:42.240718 (+ 131us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 12:10:42.249374 (+ 8656us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 12:10:42.249391 (+ 17us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 12:10:42.249409 (+ 18us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0809 12:10:42.249456 (+ 47us) server_negotiation.cc:922] calling sasl_server_start()
0809 12:10:42.257191 (+ 7735us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 12:10:42.258222 (+ 1031us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 12:10:42.258233 (+ 11us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 12:10:42.258239 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0809 12:10:42.258419 (+ 180us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 12:10:42.259295 (+ 876us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 12:10:42.259298 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 12:10:42.259301 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0809 12:10:42.259798 (+ 497us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 12:10:42.260212 (+ 414us) server_negotiation.cc:1035] Waiting for connection context
0809 12:10:42.261010 (+ 798us) server_negotiation.cc:299] Negotiation successful
0809 12:10:42.261356 (+ 346us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":16054,"thread_start_us":16822,"threads_started":1}
I20260809 12:10:42.263543 8733 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 12:10:42.210408 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.222.2:47393 (local address 127.0.222.61:32831)
0809 12:10:42.210982 (+ 574us) server_negotiation.cc:206] Beginning negotiation
0809 12:10:42.210992 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0809 12:10:42.233514 (+ 22522us) server_negotiation.cc:407] Connection header received
0809 12:10:42.235944 (+ 2430us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 12:10:42.235950 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 12:10:42.236044 (+ 94us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 12:10:42.236126 (+ 82us) server_negotiation.cc:226] Negotiated authn=SASL
0809 12:10:42.239343 (+ 3217us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 12:10:42.240500 (+ 1157us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 12:10:42.243701 (+ 3201us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 12:10:42.244371 (+ 670us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 12:10:42.244468 (+ 97us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 12:10:42.250557 (+ 6089us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 12:10:42.250564 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 12:10:42.250580 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0809 12:10:42.250635 (+ 55us) server_negotiation.cc:922] calling sasl_server_start()
0809 12:10:42.257554 (+ 6919us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 12:10:42.258950 (+ 1396us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 12:10:42.258991 (+ 41us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 12:10:42.259003 (+ 12us) server_negotiation.cc:967] calling sasl_server_step()
0809 12:10:42.259160 (+ 157us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 12:10:42.260391 (+ 1231us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 12:10:42.260397 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 12:10:42.260400 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0809 12:10:42.260850 (+ 450us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 12:10:42.260978 (+ 128us) server_negotiation.cc:1035] Waiting for connection context
0809 12:10:42.262755 (+ 1777us) server_negotiation.cc:299] Negotiation successful
0809 12:10:42.263086 (+ 331us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":397,"thread_start_us":1028,"threads_started":1}
I20260809 12:10:42.263871 8022 generic_service.cc:196] Checking for leaks (request via RPC)
I20260809 12:10:42.263927 8734 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 12:10:42.211080 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.222.2:43953 (local address 127.0.222.62:41983)
0809 12:10:42.211634 (+ 554us) server_negotiation.cc:206] Beginning negotiation
0809 12:10:42.211643 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0809 12:10:42.234350 (+ 22707us) server_negotiation.cc:407] Connection header received
0809 12:10:42.235901 (+ 1551us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 12:10:42.235908 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 12:10:42.236020 (+ 112us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 12:10:42.236148 (+ 128us) server_negotiation.cc:226] Negotiated authn=SASL
0809 12:10:42.239342 (+ 3194us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 12:10:42.240483 (+ 1141us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 12:10:42.243679 (+ 3196us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 12:10:42.244350 (+ 671us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 12:10:42.244489 (+ 139us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 12:10:42.250020 (+ 5531us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 12:10:42.250028 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 12:10:42.250034 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0809 12:10:42.250081 (+ 47us) server_negotiation.cc:922] calling sasl_server_start()
0809 12:10:42.254030 (+ 3949us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 12:10:42.258456 (+ 4426us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 12:10:42.258461 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 12:10:42.258465 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0809 12:10:42.258749 (+ 284us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 12:10:42.259480 (+ 731us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 12:10:42.259485 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 12:10:42.259488 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0809 12:10:42.260083 (+ 595us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 12:10:42.260281 (+ 198us) server_negotiation.cc:1035] Waiting for connection context
0809 12:10:42.262862 (+ 2581us) server_negotiation.cc:299] Negotiation successful
0809 12:10:42.263126 (+ 264us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":382,"thread_start_us":268,"threads_started":1}
I20260809 12:10:42.264947 8732 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 12:10:42.209817 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.222.61:32831 (local address 127.0.222.2:47393)
0809 12:10:42.233383 (+ 23566us) negotiation.cc:107] Waiting for socket to connect
0809 12:10:42.233405 (+ 22us) client_negotiation.cc:174] Beginning negotiation
0809 12:10:42.235570 (+ 2165us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 12:10:42.236552 (+ 982us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 12:10:42.236565 (+ 13us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 12:10:42.237824 (+ 1259us) client_negotiation.cc:189] Negotiated authn=SASL
0809 12:10:42.239130 (+ 1306us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 12:10:42.239138 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 12:10:42.240698 (+ 1560us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 12:10:42.240702 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 12:10:42.243513 (+ 2811us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 12:10:42.243530 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 12:10:42.244792 (+ 1262us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 12:10:42.244798 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 12:10:42.245034 (+ 236us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 12:10:42.245889 (+ 855us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0809 12:10:42.245932 (+ 43us) client_negotiation.cc:656] calling sasl_client_start()
0809 12:10:42.250233 (+ 4301us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0809 12:10:42.257690 (+ 7457us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0809 12:10:42.257695 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0809 12:10:42.257698 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0809 12:10:42.258822 (+ 1124us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0809 12:10:42.259842 (+ 1020us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0809 12:10:42.259846 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0809 12:10:42.259849 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0809 12:10:42.260151 (+ 302us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0809 12:10:42.262212 (+ 2061us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0809 12:10:42.262216 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0809 12:10:42.262356 (+ 140us) client_negotiation.cc:769] Sending connection context
0809 12:10:42.263823 (+ 1467us) client_negotiation.cc:240] Negotiation successful
0809 12:10:42.264111 (+ 288us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":23421,"mutex_wait_us":602,"spinlock_wait_cycles":41472,"thread_start_us":23026,"threads_started":1}
I20260809 12:10:42.265502 8735 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 12:10:42.209795 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.222.60:39543 (local address 127.0.222.2:35187)
0809 12:10:42.231932 (+ 22137us) negotiation.cc:107] Waiting for socket to connect
0809 12:10:42.231994 (+ 62us) client_negotiation.cc:174] Beginning negotiation
0809 12:10:42.235570 (+ 3576us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 12:10:42.236552 (+ 982us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 12:10:42.236565 (+ 13us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 12:10:42.237800 (+ 1235us) client_negotiation.cc:189] Negotiated authn=SASL
0809 12:10:42.239100 (+ 1300us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 12:10:42.239135 (+ 35us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 12:10:42.242113 (+ 2978us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 12:10:42.242118 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 12:10:42.244904 (+ 2786us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 12:10:42.244922 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 12:10:42.246299 (+ 1377us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 12:10:42.246306 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 12:10:42.246594 (+ 288us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 12:10:42.247476 (+ 882us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0809 12:10:42.247526 (+ 50us) client_negotiation.cc:656] calling sasl_client_start()
0809 12:10:42.251763 (+ 4237us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0809 12:10:42.257071 (+ 5308us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0809 12:10:42.257075 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0809 12:10:42.257077 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0809 12:10:42.257901 (+ 824us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0809 12:10:42.258517 (+ 616us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0809 12:10:42.258522 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0809 12:10:42.258527 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0809 12:10:42.258889 (+ 362us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0809 12:10:42.259785 (+ 896us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0809 12:10:42.259794 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0809 12:10:42.260468 (+ 674us) client_negotiation.cc:769] Sending connection context
0809 12:10:42.260943 (+ 475us) client_negotiation.cc:240] Negotiation successful
0809 12:10:42.261558 (+ 615us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":21773,"mutex_wait_us":668,"thread_start_us":24455,"threads_started":1}
I20260809 12:10:42.266532 8731 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 12:10:42.211087 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.222.62:41983 (local address 127.0.222.2:43953)
0809 12:10:42.234137 (+ 23050us) negotiation.cc:107] Waiting for socket to connect
0809 12:10:42.234159 (+ 22us) client_negotiation.cc:174] Beginning negotiation
0809 12:10:42.235603 (+ 1444us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 12:10:42.236552 (+ 949us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 12:10:42.236565 (+ 13us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 12:10:42.237798 (+ 1233us) client_negotiation.cc:189] Negotiated authn=SASL
0809 12:10:42.239130 (+ 1332us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 12:10:42.239146 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 12:10:42.240698 (+ 1552us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 12:10:42.240703 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 12:10:42.243515 (+ 2812us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 12:10:42.243531 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 12:10:42.244618 (+ 1087us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 12:10:42.244626 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 12:10:42.245007 (+ 381us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 12:10:42.245944 (+ 937us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0809 12:10:42.245968 (+ 24us) client_negotiation.cc:656] calling sasl_client_start()
0809 12:10:42.249669 (+ 3701us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0809 12:10:42.254892 (+ 5223us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0809 12:10:42.254903 (+ 11us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0809 12:10:42.254924 (+ 21us) client_negotiation.cc:760] calling sasl_client_step()
0809 12:10:42.258303 (+ 3379us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0809 12:10:42.259015 (+ 712us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0809 12:10:42.259020 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0809 12:10:42.259024 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0809 12:10:42.259299 (+ 275us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0809 12:10:42.261977 (+ 2678us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0809 12:10:42.261984 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0809 12:10:42.262533 (+ 549us) client_negotiation.cc:769] Sending connection context
0809 12:10:42.262695 (+ 162us) client_negotiation.cc:240] Negotiation successful
0809 12:10:42.263039 (+ 344us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":22918,"mutex_wait_us":327,"spinlock_wait_cycles":5760,"thread_start_us":20397,"threads_started":1}
I20260809 12:10:42.267091 8726 heartbeater.cc:347] Connected to a master server at 127.0.222.60:39543
I20260809 12:10:42.267171 8728 heartbeater.cc:347] Connected to a master server at 127.0.222.61:32831
I20260809 12:10:42.267558 8726 heartbeater.cc:464] Registering TS with master...
I20260809 12:10:42.267534 8728 heartbeater.cc:464] Registering TS with master...
I20260809 12:10:42.268239 8727 heartbeater.cc:347] Connected to a master server at 127.0.222.62:41983
I20260809 12:10:42.268424 8727 heartbeater.cc:464] Registering TS with master...
I20260809 12:10:42.269672 8726 heartbeater.cc:511] Master 127.0.222.60:39543 requested a full tablet report, sending...
I20260809 12:10:42.270027 8728 heartbeater.cc:511] Master 127.0.222.61:32831 requested a full tablet report, sending...
I20260809 12:10:42.270066 8727 heartbeater.cc:511] Master 127.0.222.62:41983 requested a full tablet report, sending...
I20260809 12:10:42.271667 7547 ts_manager.cc:194] Re-registered known tserver with Master: 94413a87e0a14ef195baa2a799a9e0a8 (127.0.222.2:33587)
I20260809 12:10:42.271956 8183 ts_manager.cc:194] Registered new tserver with Master: 94413a87e0a14ef195baa2a799a9e0a8 (127.0.222.2:33587)
I20260809 12:10:42.271941 8385 ts_manager.cc:194] Registered new tserver with Master: 94413a87e0a14ef195baa2a799a9e0a8 (127.0.222.2:33587)
I20260809 12:10:42.272912 7547 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.0.222.2@KRBTEST.COM'} at 127.0.222.2:43953
I20260809 12:10:42.363795 888 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskgH5Pjm/build/asan/bin/kudu with pid 7894
I20260809 12:10:42.384008 888 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskgH5Pjm/build/asan/bin/kudu
/tmp/dist-test-taskgH5Pjm/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskgH5Pjm/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786277382073938-888-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskgH5Pjm/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786277382073938-888-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskgH5Pjm/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786277382073938-888-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskgH5Pjm/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786277382073938-888-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.222.3:37357
--local_ip_for_outbound_sockets=127.0.222.3
--tserver_master_addrs=127.0.222.62:41983
--webserver_port=35519
--webserver_interface=127.0.222.3
--keytab_file=/tmp/dist-test-taskgH5Pjm/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786277382073938-888-0/krb5kdc/oryx_127.0.222.3.keytab
--principal=oryx/127.0.222.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.222.20:38069
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_master_addrs=127.0.222.62:41983,127.0.222.61:32831
--tserver_master_addrs=127.0.222.62:41983,127.0.222.61:32831,127.0.222.60:39543 with env {KRB5CCNAME=/tmp/dist-test-taskgH5Pjm/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786277382073938-888-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskgH5Pjm/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786277382073938-888-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskgH5Pjm/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786277382073938-888-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260809 12:10:42.635437 8573 heartbeater.cc:503] Master 127.0.222.62:41983 was elected leader, sending a full tablet report...
W20260809 12:10:42.729879 8739 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260809 12:10:42.730233 8739 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260809 12:10:42.730409 8739 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260809 12:10:42.737685 8739 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260809 12:10:42.737847 8739 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260809 12:10:42.738037 8739 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.222.3
I20260809 12:10:42.748083 8739 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.222.20:38069
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskgH5Pjm/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786277382073938-888-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskgH5Pjm/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786277382073938-888-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskgH5Pjm/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786277382073938-888-0/krb5kdc/oryx_127.0.222.3.keytab
--principal=oryx/127.0.222.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.222.3:37357
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskgH5Pjm/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786277382073938-888-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.222.3
--webserver_port=35519
--tserver_master_addrs=127.0.222.62:41983,127.0.222.61:32831,127.0.222.60:39543
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.222.3
--log_dir=/tmp/dist-test-taskgH5Pjm/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786277382073938-888-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 09 Aug 2026 11:06:41 UTC on ccycloud.quasar-bwoogy.root.comops.site
build id 4575991
ASAN enabled
I20260809 12:10:42.749545 8739 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260809 12:10:42.751078 8739 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260809 12:10:42.765239 8745 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 12:10:42.765277 8747 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 12:10:42.765822 8744 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260809 12:10:42.771617 8739 server_base.cc:1034] running on GCE node
I20260809 12:10:42.772257 8739 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260809 12:10:42.773260 8739 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260809 12:10:42.774495 8739 hybrid_clock.cc:648] HybridClock initialized: now 1786277442774452 us; error 61 us; skew 500 ppm
I20260809 12:10:42.774822 8739 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 09 12:10:42 dist-test-slave-2w3w krb5kdc[7502](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786277442, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.222.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260809 12:10:42.777971 8739 init.cc:377] Logged in from keytab as oryx/127.0.222.3@KRBTEST.COM (short username oryx)
I20260809 12:10:42.779618 8739 webserver.cc:460] Webserver started at http://127.0.222.3:35519/ using document root <none> and password file <none>
I20260809 12:10:42.780407 8739 fs_manager.cc:359] Metadata directory not provided
I20260809 12:10:42.780568 8739 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260809 12:10:42.788556 8739 fs_manager.cc:709] Time spent opening directory manager: real 0.005s user 0.004s sys 0.000s
I20260809 12:10:42.792297 8755 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260809 12:10:42.793951 8739 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.004s sys 0.000s
I20260809 12:10:42.794092 8739 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskgH5Pjm/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786277382073938-888-0/minicluster-data/ts-2/data,/tmp/dist-test-taskgH5Pjm/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786277382073938-888-0/minicluster-data/ts-2/wal
uuid: "c40926a9373242b0961349a2719c22ab"
format_stamp: "Formatted at 2026-08-09 12:10:39 on dist-test-slave-2w3w"
server_key: "d550667833dac40fe63877ee3b427f59"
server_key_iv: "c0c7ae44eaf7a0b79e122a514fa08eed"
server_key_version: "encryptionkey@0"
I20260809 12:10:42.795217 8739 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskgH5Pjm/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786277382073938-888-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskgH5Pjm/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786277382073938-888-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskgH5Pjm/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786277382073938-888-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260809 12:10:42.821066 8739 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260809 12:10:42.825114 8739 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260809 12:10:42.825351 8739 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260809 12:10:42.826337 8739 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260809 12:10:42.827739 8739 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260809 12:10:42.827827 8739 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260809 12:10:42.827910 8739 ts_tablet_manager.cc:610] Registered 0 tablets
I20260809 12:10:42.827973 8739 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260809 12:10:42.862582 8739 rpc_server.cc:307] RPC server started. Bound to: 127.0.222.3:37357
I20260809 12:10:42.862710 8877 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.222.3:37357 every 8 connection(s)
I20260809 12:10:42.864212 8739 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskgH5Pjm/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786277382073938-888-0/minicluster-data/ts-2/data/info.pb
I20260809 12:10:42.871558 888 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskgH5Pjm/build/asan/bin/kudu as pid 8739
Aug 09 12:10:42 dist-test-slave-2w3w krb5kdc[7502](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786277442, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.222.3@KRBTEST.COM for oryx/127.0.222.61@KRBTEST.COM
Aug 09 12:10:42 dist-test-slave-2w3w krb5kdc[7502](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786277442, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.222.3@KRBTEST.COM for oryx/127.0.222.62@KRBTEST.COM
I20260809 12:10:42.919546 8883 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 12:10:42.867987 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.222.61:32831 (local address 127.0.222.3:33609)
0809 12:10:42.887188 (+ 19201us) negotiation.cc:107] Waiting for socket to connect
0809 12:10:42.887249 (+ 61us) client_negotiation.cc:174] Beginning negotiation
0809 12:10:42.890170 (+ 2921us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 12:10:42.893400 (+ 3230us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 12:10:42.893408 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 12:10:42.894245 (+ 837us) client_negotiation.cc:189] Negotiated authn=SASL
0809 12:10:42.894629 (+ 384us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 12:10:42.894645 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 12:10:42.896710 (+ 2065us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 12:10:42.896719 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 12:10:42.899478 (+ 2759us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 12:10:42.899500 (+ 22us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 12:10:42.901068 (+ 1568us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 12:10:42.901078 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 12:10:42.901490 (+ 412us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 12:10:42.902463 (+ 973us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0809 12:10:42.902511 (+ 48us) client_negotiation.cc:656] calling sasl_client_start()
0809 12:10:42.907832 (+ 5321us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0809 12:10:42.912317 (+ 4485us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0809 12:10:42.912327 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0809 12:10:42.912352 (+ 25us) client_negotiation.cc:760] calling sasl_client_step()
0809 12:10:42.913054 (+ 702us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0809 12:10:42.913794 (+ 740us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0809 12:10:42.913801 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0809 12:10:42.913806 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0809 12:10:42.914066 (+ 260us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0809 12:10:42.914873 (+ 807us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0809 12:10:42.914882 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0809 12:10:42.915352 (+ 470us) client_negotiation.cc:769] Sending connection context
0809 12:10:42.915699 (+ 347us) client_negotiation.cc:240] Negotiation successful
0809 12:10:42.916292 (+ 593us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":18806,"thread_start_us":18543,"threads_started":1}
Aug 09 12:10:42 dist-test-slave-2w3w krb5kdc[7502](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786277442, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.222.3@KRBTEST.COM for oryx/127.0.222.60@KRBTEST.COM
I20260809 12:10:42.920964 8882 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 12:10:42.867290 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.222.3:33609 (local address 127.0.222.61:32831)
0809 12:10:42.869238 (+ 1948us) server_negotiation.cc:206] Beginning negotiation
0809 12:10:42.869247 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0809 12:10:42.887425 (+ 18178us) server_negotiation.cc:407] Connection header received
0809 12:10:42.891104 (+ 3679us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 12:10:42.891110 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 12:10:42.891201 (+ 91us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 12:10:42.894205 (+ 3004us) server_negotiation.cc:226] Negotiated authn=SASL
0809 12:10:42.894907 (+ 702us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 12:10:42.896474 (+ 1567us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 12:10:42.899752 (+ 3278us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 12:10:42.900731 (+ 979us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 12:10:42.902047 (+ 1316us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 12:10:42.908259 (+ 6212us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 12:10:42.908265 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 12:10:42.908270 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0809 12:10:42.908345 (+ 75us) server_negotiation.cc:922] calling sasl_server_start()
0809 12:10:42.912150 (+ 3805us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 12:10:42.913468 (+ 1318us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 12:10:42.913475 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 12:10:42.913479 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0809 12:10:42.913654 (+ 175us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 12:10:42.914209 (+ 555us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 12:10:42.914213 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 12:10:42.914216 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0809 12:10:42.914594 (+ 378us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 12:10:42.914731 (+ 137us) server_negotiation.cc:1035] Waiting for connection context
0809 12:10:42.920139 (+ 5408us) server_negotiation.cc:299] Negotiation successful
0809 12:10:42.920410 (+ 271us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1780,"thread_start_us":255,"threads_started":1}
I20260809 12:10:42.923967 8880 heartbeater.cc:347] Connected to a master server at 127.0.222.61:32831
I20260809 12:10:42.924381 8880 heartbeater.cc:464] Registering TS with master...
I20260809 12:10:42.925701 8880 heartbeater.cc:511] Master 127.0.222.61:32831 requested a full tablet report, sending...
I20260809 12:10:42.927887 8183 ts_manager.cc:194] Registered new tserver with Master: c40926a9373242b0961349a2719c22ab (127.0.222.3:37357)
I20260809 12:10:42.933694 8885 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 12:10:42.869023 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.222.60:39543 (local address 127.0.222.3:33537)
0809 12:10:42.888682 (+ 19659us) negotiation.cc:107] Waiting for socket to connect
0809 12:10:42.888703 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0809 12:10:42.890168 (+ 1465us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 12:10:42.891134 (+ 966us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 12:10:42.891147 (+ 13us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 12:10:42.892301 (+ 1154us) client_negotiation.cc:189] Negotiated authn=SASL
0809 12:10:42.893363 (+ 1062us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 12:10:42.893396 (+ 33us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 12:10:42.910889 (+ 17493us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 12:10:42.910895 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 12:10:42.912499 (+ 1604us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 12:10:42.912514 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 12:10:42.915989 (+ 3475us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 12:10:42.915995 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 12:10:42.916158 (+ 163us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 12:10:42.917649 (+ 1491us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0809 12:10:42.917670 (+ 21us) client_negotiation.cc:656] calling sasl_client_start()
0809 12:10:42.920582 (+ 2912us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0809 12:10:42.929330 (+ 8748us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0809 12:10:42.929335 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0809 12:10:42.929338 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0809 12:10:42.929905 (+ 567us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0809 12:10:42.931400 (+ 1495us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0809 12:10:42.931407 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0809 12:10:42.931410 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0809 12:10:42.931591 (+ 181us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0809 12:10:42.932506 (+ 915us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0809 12:10:42.932511 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0809 12:10:42.932675 (+ 164us) client_negotiation.cc:769] Sending connection context
0809 12:10:42.932847 (+ 172us) client_negotiation.cc:240] Negotiation successful
0809 12:10:42.933193 (+ 346us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":19541,"mutex_wait_us":1,"thread_start_us":18891,"threads_started":1}
I20260809 12:10:42.934391 8886 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 12:10:42.869094 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.222.3:33537 (local address 127.0.222.60:39543)
0809 12:10:42.887468 (+ 18374us) server_negotiation.cc:206] Beginning negotiation
0809 12:10:42.887477 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0809 12:10:42.888970 (+ 1493us) server_negotiation.cc:407] Connection header received
0809 12:10:42.890624 (+ 1654us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 12:10:42.890628 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 12:10:42.890701 (+ 73us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 12:10:42.890838 (+ 137us) server_negotiation.cc:226] Negotiated authn=SASL
0809 12:10:42.905269 (+ 14431us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 12:10:42.906833 (+ 1564us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 12:10:42.913290 (+ 6457us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 12:10:42.914037 (+ 747us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 12:10:42.914156 (+ 119us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 12:10:42.920871 (+ 6715us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 12:10:42.920881 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 12:10:42.920888 (+ 7us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0809 12:10:42.920942 (+ 54us) server_negotiation.cc:922] calling sasl_server_start()
0809 12:10:42.926098 (+ 5156us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 12:10:42.930049 (+ 3951us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 12:10:42.930054 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 12:10:42.930058 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0809 12:10:42.930273 (+ 215us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 12:10:42.931735 (+ 1462us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 12:10:42.931739 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 12:10:42.931742 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0809 12:10:42.932254 (+ 512us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 12:10:42.932423 (+ 169us) server_negotiation.cc:1035] Waiting for connection context
0809 12:10:42.933720 (+ 1297us) server_negotiation.cc:299] Negotiation successful
0809 12:10:42.934055 (+ 335us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":18182,"thread_start_us":18016,"threads_started":1}
I20260809 12:10:42.934798 8884 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 12:10:42.868001 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.222.62:41983 (local address 127.0.222.3:57145)
0809 12:10:42.888029 (+ 20028us) negotiation.cc:107] Waiting for socket to connect
0809 12:10:42.888051 (+ 22us) client_negotiation.cc:174] Beginning negotiation
0809 12:10:42.890163 (+ 2112us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 12:10:42.893674 (+ 3511us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 12:10:42.893682 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 12:10:42.904695 (+ 11013us) client_negotiation.cc:189] Negotiated authn=SASL
0809 12:10:42.905024 (+ 329us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 12:10:42.905040 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 12:10:42.909032 (+ 3992us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 12:10:42.909038 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 12:10:42.910657 (+ 1619us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 12:10:42.910675 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 12:10:42.912721 (+ 2046us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 12:10:42.912727 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 12:10:42.912925 (+ 198us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 12:10:42.914464 (+ 1539us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0809 12:10:42.914485 (+ 21us) client_negotiation.cc:656] calling sasl_client_start()
0809 12:10:42.917436 (+ 2951us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0809 12:10:42.928231 (+ 10795us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0809 12:10:42.928237 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0809 12:10:42.928240 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0809 12:10:42.929145 (+ 905us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0809 12:10:42.930051 (+ 906us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0809 12:10:42.930056 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0809 12:10:42.930059 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0809 12:10:42.930272 (+ 213us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0809 12:10:42.933499 (+ 3227us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0809 12:10:42.933506 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0809 12:10:42.933695 (+ 189us) client_negotiation.cc:769] Sending connection context
0809 12:10:42.933855 (+ 160us) client_negotiation.cc:240] Negotiation successful
0809 12:10:42.934187 (+ 332us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":19840,"mutex_wait_us":59,"spinlock_wait_cycles":130304,"thread_start_us":19457,"threads_started":1}
I20260809 12:10:42.935027 8887 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 12:10:42.869491 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.222.3:57145 (local address 127.0.222.62:41983)
0809 12:10:42.888607 (+ 19116us) server_negotiation.cc:206] Beginning negotiation
0809 12:10:42.888613 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0809 12:10:42.888636 (+ 23us) server_negotiation.cc:407] Connection header received
0809 12:10:42.890624 (+ 1988us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 12:10:42.890633 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 12:10:42.890723 (+ 90us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 12:10:42.890841 (+ 118us) server_negotiation.cc:226] Negotiated authn=SASL
0809 12:10:42.905198 (+ 14357us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 12:10:42.906622 (+ 1424us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 12:10:42.910808 (+ 4186us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 12:10:42.911717 (+ 909us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 12:10:42.911867 (+ 150us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 12:10:42.917733 (+ 5866us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 12:10:42.917742 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 12:10:42.917750 (+ 8us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0809 12:10:42.917811 (+ 61us) server_negotiation.cc:922] calling sasl_server_start()
0809 12:10:42.923086 (+ 5275us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 12:10:42.929291 (+ 6205us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 12:10:42.929300 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 12:10:42.929304 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0809 12:10:42.929582 (+ 278us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 12:10:42.930391 (+ 809us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 12:10:42.930396 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 12:10:42.930400 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0809 12:10:42.931066 (+ 666us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 12:10:42.931253 (+ 187us) server_negotiation.cc:1035] Waiting for connection context
0809 12:10:42.934175 (+ 2922us) server_negotiation.cc:299] Negotiation successful
0809 12:10:42.934447 (+ 272us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":18982,"thread_start_us":18875,"threads_started":1}
I20260809 12:10:42.935652 8878 heartbeater.cc:347] Connected to a master server at 127.0.222.60:39543
I20260809 12:10:42.935843 8878 heartbeater.cc:464] Registering TS with master...
I20260809 12:10:42.936439 8878 heartbeater.cc:511] Master 127.0.222.60:39543 requested a full tablet report, sending...
I20260809 12:10:42.937265 8879 heartbeater.cc:347] Connected to a master server at 127.0.222.62:41983
I20260809 12:10:42.937463 8879 heartbeater.cc:464] Registering TS with master...
I20260809 12:10:42.938261 8385 ts_manager.cc:194] Registered new tserver with Master: c40926a9373242b0961349a2719c22ab (127.0.222.3:37357)
I20260809 12:10:42.938346 8879 heartbeater.cc:511] Master 127.0.222.62:41983 requested a full tablet report, sending...
I20260809 12:10:42.939785 7547 ts_manager.cc:194] Re-registered known tserver with Master: c40926a9373242b0961349a2719c22ab (127.0.222.3:37357)
I20260809 12:10:42.941136 7547 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.0.222.3@KRBTEST.COM'} at 127.0.222.3:57145
I20260809 12:10:42.953392 8892 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 12:10:42.891481 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33094 (local address 127.0.222.62:41983)
0809 12:10:42.905003 (+ 13522us) server_negotiation.cc:206] Beginning negotiation
0809 12:10:42.905010 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0809 12:10:42.907542 (+ 2532us) server_negotiation.cc:407] Connection header received
0809 12:10:42.907766 (+ 224us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 12:10:42.907771 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 12:10:42.907863 (+ 92us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 12:10:42.908245 (+ 382us) server_negotiation.cc:226] Negotiated authn=SASL
0809 12:10:42.916747 (+ 8502us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 12:10:42.917981 (+ 1234us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 12:10:42.926281 (+ 8300us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 12:10:42.927100 (+ 819us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 12:10:42.927233 (+ 133us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 12:10:42.944203 (+ 16970us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 12:10:42.944210 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 12:10:42.944215 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0809 12:10:42.944253 (+ 38us) server_negotiation.cc:922] calling sasl_server_start()
0809 12:10:42.949156 (+ 4903us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 12:10:42.950434 (+ 1278us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 12:10:42.950442 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 12:10:42.950446 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0809 12:10:42.950630 (+ 184us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 12:10:42.951099 (+ 469us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 12:10:42.951104 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 12:10:42.951107 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0809 12:10:42.951644 (+ 537us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 12:10:42.951961 (+ 317us) server_negotiation.cc:1035] Waiting for connection context
0809 12:10:42.952403 (+ 442us) server_negotiation.cc:299] Negotiation successful
0809 12:10:42.952703 (+ 300us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":13337,"thread_start_us":13529,"threads_started":1}
I20260809 12:10:42.958062 8895 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 12:10:42.891632 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42474 (local address 127.0.222.60:39543)
0809 12:10:42.931621 (+ 39989us) server_negotiation.cc:206] Beginning negotiation
0809 12:10:42.931625 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0809 12:10:42.931640 (+ 15us) server_negotiation.cc:407] Connection header received
0809 12:10:42.931720 (+ 80us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 12:10:42.931724 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 12:10:42.931803 (+ 79us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 12:10:42.932144 (+ 341us) server_negotiation.cc:226] Negotiated authn=SASL
0809 12:10:42.938276 (+ 6132us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 12:10:42.940291 (+ 2015us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 12:10:42.943372 (+ 3081us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 12:10:42.944284 (+ 912us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 12:10:42.944412 (+ 128us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 12:10:42.949469 (+ 5057us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 12:10:42.949475 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 12:10:42.949478 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0809 12:10:42.949511 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0809 12:10:42.954282 (+ 4771us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 12:10:42.955390 (+ 1108us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 12:10:42.955395 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 12:10:42.955399 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0809 12:10:42.955722 (+ 323us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 12:10:42.956255 (+ 533us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 12:10:42.956261 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 12:10:42.956265 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0809 12:10:42.956662 (+ 397us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 12:10:42.956963 (+ 301us) server_negotiation.cc:1035] Waiting for connection context
0809 12:10:42.957381 (+ 418us) server_negotiation.cc:299] Negotiation successful
0809 12:10:42.957619 (+ 238us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":39881,"thread_start_us":39676,"threads_started":1}
I20260809 12:10:42.961215 8896 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 12:10:42.898054 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46558 (local address 127.0.222.61:32831)
0809 12:10:42.929953 (+ 31899us) server_negotiation.cc:206] Beginning negotiation
0809 12:10:42.929963 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0809 12:10:42.929988 (+ 25us) server_negotiation.cc:407] Connection header received
0809 12:10:42.930054 (+ 66us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 12:10:42.930062 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 12:10:42.930161 (+ 99us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 12:10:42.930529 (+ 368us) server_negotiation.cc:226] Negotiated authn=SASL
0809 12:10:42.936536 (+ 6007us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 12:10:42.940360 (+ 3824us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 12:10:42.943102 (+ 2742us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 12:10:42.943846 (+ 744us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 12:10:42.943980 (+ 134us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 12:10:42.947996 (+ 4016us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 12:10:42.948002 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 12:10:42.948006 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0809 12:10:42.948040 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0809 12:10:42.955457 (+ 7417us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 12:10:42.956757 (+ 1300us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 12:10:42.956763 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 12:10:42.956767 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0809 12:10:42.957100 (+ 333us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 12:10:42.957681 (+ 581us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 12:10:42.957688 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 12:10:42.957691 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0809 12:10:42.958195 (+ 504us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 12:10:42.958598 (+ 403us) server_negotiation.cc:1035] Waiting for connection context
0809 12:10:42.960070 (+ 1472us) server_negotiation.cc:299] Negotiation successful
0809 12:10:42.960358 (+ 288us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":19413,"thread_start_us":19247,"threads_started":1}
I20260809 12:10:42.959028 7547 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:33094:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260809 12:10:42.970208 7547 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260809 12:10:43.018220 8912 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 12:10:42.995087 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.222.1:43733 (local address 127.0.0.1:49254)
0809 12:10:43.002595 (+ 7508us) negotiation.cc:107] Waiting for socket to connect
0809 12:10:43.002625 (+ 30us) client_negotiation.cc:174] Beginning negotiation
0809 12:10:43.002875 (+ 250us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 12:10:43.009713 (+ 6838us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 12:10:43.009719 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 12:10:43.009748 (+ 29us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0809 12:10:43.010595 (+ 847us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 12:10:43.010614 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 12:10:43.012355 (+ 1741us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 12:10:43.012361 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 12:10:43.014273 (+ 1912us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 12:10:43.014287 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 12:10:43.017181 (+ 2894us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 12:10:43.017189 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 12:10:43.017421 (+ 232us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 12:10:43.017458 (+ 37us) client_negotiation.cc:769] Sending connection context
0809 12:10:43.017569 (+ 111us) client_negotiation.cc:240] Negotiation successful
0809 12:10:43.017740 (+ 171us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":7314,"thread_start_us":7074,"threads_started":1}
I20260809 12:10:43.019379 8913 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 12:10:42.995266 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49254 (local address 127.0.222.1:43733)
0809 12:10:43.009077 (+ 13811us) server_negotiation.cc:206] Beginning negotiation
0809 12:10:43.009088 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0809 12:10:43.009123 (+ 35us) server_negotiation.cc:407] Connection header received
0809 12:10:43.009210 (+ 87us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 12:10:43.009216 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 12:10:43.009455 (+ 239us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 12:10:43.009738 (+ 283us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0809 12:10:43.011011 (+ 1273us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 12:10:43.012173 (+ 1162us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 12:10:43.014447 (+ 2274us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 12:10:43.016594 (+ 2147us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 12:10:43.016721 (+ 127us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 12:10:43.016971 (+ 250us) server_negotiation.cc:1035] Waiting for connection context
0809 12:10:43.018779 (+ 1808us) server_negotiation.cc:299] Negotiation successful
0809 12:10:43.019022 (+ 243us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":13624,"thread_start_us":13497,"threads_started":1}
I20260809 12:10:43.041646 8912 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 12:10:43.002861 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.222.2:33587 (local address 127.0.0.1:35430)
0809 12:10:43.018503 (+ 15642us) negotiation.cc:107] Waiting for socket to connect
0809 12:10:43.018521 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0809 12:10:43.018619 (+ 98us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 12:10:43.030772 (+ 12153us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 12:10:43.030778 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 12:10:43.030812 (+ 34us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0809 12:10:43.031202 (+ 390us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 12:10:43.031217 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 12:10:43.035128 (+ 3911us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 12:10:43.035134 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 12:10:43.038118 (+ 2984us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 12:10:43.038145 (+ 27us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 12:10:43.040615 (+ 2470us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 12:10:43.040622 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 12:10:43.040861 (+ 239us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 12:10:43.040899 (+ 38us) client_negotiation.cc:769] Sending connection context
0809 12:10:43.040970 (+ 71us) client_negotiation.cc:240] Negotiation successful
0809 12:10:43.041146 (+ 176us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":15544,"thread_start_us":34107,"threads_started":1}
I20260809 12:10:43.042060 8505 tablet_service.cc:1467] Processing CreateTablet for tablet f79823fb579a4914b2b61f7d1165d91a (DEFAULT_TABLE table=test-table [id=ec147d97ec774ba395a9d447b9aae730]), partition=RANGE (key) PARTITION UNBOUNDED
I20260809 12:10:43.043380 8917 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 12:10:43.002929 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35430 (local address 127.0.222.2:33587)
0809 12:10:43.022049 (+ 19120us) server_negotiation.cc:206] Beginning negotiation
0809 12:10:43.022059 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0809 12:10:43.022090 (+ 31us) server_negotiation.cc:407] Connection header received
0809 12:10:43.022192 (+ 102us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 12:10:43.022201 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 12:10:43.022520 (+ 319us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 12:10:43.031454 (+ 8934us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0809 12:10:43.031797 (+ 343us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 12:10:43.033897 (+ 2100us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 12:10:43.038338 (+ 4441us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 12:10:43.040412 (+ 2074us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 12:10:43.042118 (+ 1706us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 12:10:43.042470 (+ 352us) server_negotiation.cc:1035] Waiting for connection context
0809 12:10:43.042571 (+ 101us) server_negotiation.cc:299] Negotiation successful
0809 12:10:43.042820 (+ 249us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":18936,"thread_start_us":18786,"threads_started":1}
I20260809 12:10:43.043903 8505 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f79823fb579a4914b2b61f7d1165d91a. 1 dirs total, 0 dirs full, 0 dirs failed
I20260809 12:10:43.048264 8656 tablet_service.cc:1467] Processing CreateTablet for tablet f79823fb579a4914b2b61f7d1165d91a (DEFAULT_TABLE table=test-table [id=ec147d97ec774ba395a9d447b9aae730]), partition=RANGE (key) PARTITION UNBOUNDED
I20260809 12:10:43.049762 8656 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f79823fb579a4914b2b61f7d1165d91a. 1 dirs total, 0 dirs full, 0 dirs failed
I20260809 12:10:43.051167 8914 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 12:10:42.995891 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.222.3:37357 (local address 127.0.0.1:33572)
0809 12:10:43.007604 (+ 11713us) negotiation.cc:107] Waiting for socket to connect
0809 12:10:43.007631 (+ 27us) client_negotiation.cc:174] Beginning negotiation
0809 12:10:43.007869 (+ 238us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 12:10:43.038339 (+ 30470us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 12:10:43.038343 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 12:10:43.038376 (+ 33us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0809 12:10:43.039088 (+ 712us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 12:10:43.039104 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 12:10:43.042041 (+ 2937us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 12:10:43.042047 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 12:10:43.045205 (+ 3158us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 12:10:43.045220 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 12:10:43.049685 (+ 4465us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 12:10:43.049696 (+ 11us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 12:10:43.050135 (+ 439us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 12:10:43.050185 (+ 50us) client_negotiation.cc:769] Sending connection context
0809 12:10:43.050283 (+ 98us) client_negotiation.cc:240] Negotiation successful
0809 12:10:43.050445 (+ 162us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":11412,"thread_start_us":12028,"threads_started":1}
I20260809 12:10:43.051483 8915 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 12:10:42.996895 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33572 (local address 127.0.222.3:37357)
0809 12:10:43.037483 (+ 40588us) server_negotiation.cc:206] Beginning negotiation
0809 12:10:43.037494 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0809 12:10:43.037542 (+ 48us) server_negotiation.cc:407] Connection header received
0809 12:10:43.037658 (+ 116us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 12:10:43.037666 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 12:10:43.037930 (+ 264us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 12:10:43.038382 (+ 452us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0809 12:10:43.039302 (+ 920us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 12:10:43.041733 (+ 2431us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 12:10:43.045394 (+ 3661us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 12:10:43.049297 (+ 3903us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 12:10:43.049516 (+ 219us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 12:10:43.049902 (+ 386us) server_negotiation.cc:1035] Waiting for connection context
0809 12:10:43.050396 (+ 494us) server_negotiation.cc:299] Negotiation successful
0809 12:10:43.050662 (+ 266us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":38017,"thread_start_us":37667,"threads_started":1}
I20260809 12:10:43.056739 8808 tablet_service.cc:1467] Processing CreateTablet for tablet f79823fb579a4914b2b61f7d1165d91a (DEFAULT_TABLE table=test-table [id=ec147d97ec774ba395a9d447b9aae730]), partition=RANGE (key) PARTITION UNBOUNDED
I20260809 12:10:43.058877 8808 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f79823fb579a4914b2b61f7d1165d91a. 1 dirs total, 0 dirs full, 0 dirs failed
I20260809 12:10:43.073853 8918 tablet_bootstrap.cc:492] T f79823fb579a4914b2b61f7d1165d91a P 94413a87e0a14ef195baa2a799a9e0a8: Bootstrap starting.
I20260809 12:10:43.078441 8919 tablet_bootstrap.cc:492] T f79823fb579a4914b2b61f7d1165d91a P df0db1a138554d00956d87f5c172ee10: Bootstrap starting.
I20260809 12:10:43.080219 8918 tablet_bootstrap.cc:654] T f79823fb579a4914b2b61f7d1165d91a P 94413a87e0a14ef195baa2a799a9e0a8: Neither blocks nor log segments found. Creating new log.
I20260809 12:10:43.082196 8918 log.cc:824] T f79823fb579a4914b2b61f7d1165d91a P 94413a87e0a14ef195baa2a799a9e0a8: Log is configured to *not* fsync() on all Append() calls
I20260809 12:10:43.085767 8919 tablet_bootstrap.cc:654] T f79823fb579a4914b2b61f7d1165d91a P df0db1a138554d00956d87f5c172ee10: Neither blocks nor log segments found. Creating new log.
I20260809 12:10:43.085834 8918 tablet_bootstrap.cc:492] T f79823fb579a4914b2b61f7d1165d91a P 94413a87e0a14ef195baa2a799a9e0a8: No bootstrap required, opened a new log
I20260809 12:10:43.086391 8918 ts_tablet_manager.cc:1397] T f79823fb579a4914b2b61f7d1165d91a P 94413a87e0a14ef195baa2a799a9e0a8: Time spent bootstrapping tablet: real 0.013s user 0.003s sys 0.007s
I20260809 12:10:43.086557 8920 tablet_bootstrap.cc:492] T f79823fb579a4914b2b61f7d1165d91a P c40926a9373242b0961349a2719c22ab: Bootstrap starting.
I20260809 12:10:43.088256 8919 log.cc:824] T f79823fb579a4914b2b61f7d1165d91a P df0db1a138554d00956d87f5c172ee10: Log is configured to *not* fsync() on all Append() calls
I20260809 12:10:43.093636 8920 tablet_bootstrap.cc:654] T f79823fb579a4914b2b61f7d1165d91a P c40926a9373242b0961349a2719c22ab: Neither blocks nor log segments found. Creating new log.
I20260809 12:10:43.093796 8919 tablet_bootstrap.cc:492] T f79823fb579a4914b2b61f7d1165d91a P df0db1a138554d00956d87f5c172ee10: No bootstrap required, opened a new log
I20260809 12:10:43.094349 8919 ts_tablet_manager.cc:1397] T f79823fb579a4914b2b61f7d1165d91a P df0db1a138554d00956d87f5c172ee10: Time spent bootstrapping tablet: real 0.016s user 0.007s sys 0.004s
I20260809 12:10:43.096118 8920 log.cc:824] T f79823fb579a4914b2b61f7d1165d91a P c40926a9373242b0961349a2719c22ab: Log is configured to *not* fsync() on all Append() calls
I20260809 12:10:43.100517 8918 raft_consensus.cc:348] T f79823fb579a4914b2b61f7d1165d91a P 94413a87e0a14ef195baa2a799a9e0a8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df0db1a138554d00956d87f5c172ee10" member_type: VOTER last_known_addr { host: "127.0.222.1" port: 43733 } } peers { permanent_uuid: "c40926a9373242b0961349a2719c22ab" member_type: VOTER last_known_addr { host: "127.0.222.3" port: 37357 } } peers { permanent_uuid: "94413a87e0a14ef195baa2a799a9e0a8" member_type: VOTER last_known_addr { host: "127.0.222.2" port: 33587 } }
I20260809 12:10:43.101137 8918 raft_consensus.cc:374] T f79823fb579a4914b2b61f7d1165d91a P 94413a87e0a14ef195baa2a799a9e0a8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260809 12:10:43.101223 8920 tablet_bootstrap.cc:492] T f79823fb579a4914b2b61f7d1165d91a P c40926a9373242b0961349a2719c22ab: No bootstrap required, opened a new log
I20260809 12:10:43.101348 8918 raft_consensus.cc:729] T f79823fb579a4914b2b61f7d1165d91a P 94413a87e0a14ef195baa2a799a9e0a8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 94413a87e0a14ef195baa2a799a9e0a8, State: Initialized, Role: FOLLOWER
I20260809 12:10:43.101648 8920 ts_tablet_manager.cc:1397] T f79823fb579a4914b2b61f7d1165d91a P c40926a9373242b0961349a2719c22ab: Time spent bootstrapping tablet: real 0.015s user 0.003s sys 0.007s
I20260809 12:10:43.102149 8918 consensus_queue.cc:260] T f79823fb579a4914b2b61f7d1165d91a P 94413a87e0a14ef195baa2a799a9e0a8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df0db1a138554d00956d87f5c172ee10" member_type: VOTER last_known_addr { host: "127.0.222.1" port: 43733 } } peers { permanent_uuid: "c40926a9373242b0961349a2719c22ab" member_type: VOTER last_known_addr { host: "127.0.222.3" port: 37357 } } peers { permanent_uuid: "94413a87e0a14ef195baa2a799a9e0a8" member_type: VOTER last_known_addr { host: "127.0.222.2" port: 33587 } }
I20260809 12:10:43.104876 8918 ts_tablet_manager.cc:1428] T f79823fb579a4914b2b61f7d1165d91a P 94413a87e0a14ef195baa2a799a9e0a8: Time spent starting tablet: real 0.018s user 0.008s sys 0.012s
I20260809 12:10:43.105172 8727 heartbeater.cc:503] Master 127.0.222.62:41983 was elected leader, sending a full tablet report...
I20260809 12:10:43.110294 8919 raft_consensus.cc:348] T f79823fb579a4914b2b61f7d1165d91a P df0db1a138554d00956d87f5c172ee10 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df0db1a138554d00956d87f5c172ee10" member_type: VOTER last_known_addr { host: "127.0.222.1" port: 43733 } } peers { permanent_uuid: "c40926a9373242b0961349a2719c22ab" member_type: VOTER last_known_addr { host: "127.0.222.3" port: 37357 } } peers { permanent_uuid: "94413a87e0a14ef195baa2a799a9e0a8" member_type: VOTER last_known_addr { host: "127.0.222.2" port: 33587 } }
I20260809 12:10:43.111783 8919 raft_consensus.cc:374] T f79823fb579a4914b2b61f7d1165d91a P df0db1a138554d00956d87f5c172ee10 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260809 12:10:43.111908 8919 raft_consensus.cc:729] T f79823fb579a4914b2b61f7d1165d91a P df0db1a138554d00956d87f5c172ee10 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: df0db1a138554d00956d87f5c172ee10, State: Initialized, Role: FOLLOWER
I20260809 12:10:43.113125 8919 consensus_queue.cc:260] T f79823fb579a4914b2b61f7d1165d91a P df0db1a138554d00956d87f5c172ee10 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df0db1a138554d00956d87f5c172ee10" member_type: VOTER last_known_addr { host: "127.0.222.1" port: 43733 } } peers { permanent_uuid: "c40926a9373242b0961349a2719c22ab" member_type: VOTER last_known_addr { host: "127.0.222.3" port: 37357 } } peers { permanent_uuid: "94413a87e0a14ef195baa2a799a9e0a8" member_type: VOTER last_known_addr { host: "127.0.222.2" port: 33587 } }
I20260809 12:10:43.116042 8919 ts_tablet_manager.cc:1428] T f79823fb579a4914b2b61f7d1165d91a P df0db1a138554d00956d87f5c172ee10: Time spent starting tablet: real 0.022s user 0.014s sys 0.009s
I20260809 12:10:43.122532 8920 raft_consensus.cc:348] T f79823fb579a4914b2b61f7d1165d91a P c40926a9373242b0961349a2719c22ab [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df0db1a138554d00956d87f5c172ee10" member_type: VOTER last_known_addr { host: "127.0.222.1" port: 43733 } } peers { permanent_uuid: "c40926a9373242b0961349a2719c22ab" member_type: VOTER last_known_addr { host: "127.0.222.3" port: 37357 } } peers { permanent_uuid: "94413a87e0a14ef195baa2a799a9e0a8" member_type: VOTER last_known_addr { host: "127.0.222.2" port: 33587 } }
I20260809 12:10:43.123107 8920 raft_consensus.cc:374] T f79823fb579a4914b2b61f7d1165d91a P c40926a9373242b0961349a2719c22ab [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260809 12:10:43.123304 8920 raft_consensus.cc:729] T f79823fb579a4914b2b61f7d1165d91a P c40926a9373242b0961349a2719c22ab [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c40926a9373242b0961349a2719c22ab, State: Initialized, Role: FOLLOWER
I20260809 12:10:43.124003 8920 consensus_queue.cc:260] T f79823fb579a4914b2b61f7d1165d91a P c40926a9373242b0961349a2719c22ab [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df0db1a138554d00956d87f5c172ee10" member_type: VOTER last_known_addr { host: "127.0.222.1" port: 43733 } } peers { permanent_uuid: "c40926a9373242b0961349a2719c22ab" member_type: VOTER last_known_addr { host: "127.0.222.3" port: 37357 } } peers { permanent_uuid: "94413a87e0a14ef195baa2a799a9e0a8" member_type: VOTER last_known_addr { host: "127.0.222.2" port: 33587 } }
I20260809 12:10:43.125946 8879 heartbeater.cc:503] Master 127.0.222.62:41983 was elected leader, sending a full tablet report...
I20260809 12:10:43.126595 8920 ts_tablet_manager.cc:1428] T f79823fb579a4914b2b61f7d1165d91a P c40926a9373242b0961349a2719c22ab: Time spent starting tablet: real 0.025s user 0.007s sys 0.015s
W20260809 12:10:43.135376 8881 tablet.cc:2369] T f79823fb579a4914b2b61f7d1165d91a P c40926a9373242b0961349a2719c22ab: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260809 12:10:43.138271 8926 raft_consensus.cc:482] T f79823fb579a4914b2b61f7d1165d91a P c40926a9373242b0961349a2719c22ab [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260809 12:10:43.138684 8926 raft_consensus.cc:504] T f79823fb579a4914b2b61f7d1165d91a P c40926a9373242b0961349a2719c22ab [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df0db1a138554d00956d87f5c172ee10" member_type: VOTER last_known_addr { host: "127.0.222.1" port: 43733 } } peers { permanent_uuid: "c40926a9373242b0961349a2719c22ab" member_type: VOTER last_known_addr { host: "127.0.222.3" port: 37357 } } peers { permanent_uuid: "94413a87e0a14ef195baa2a799a9e0a8" member_type: VOTER last_known_addr { host: "127.0.222.2" port: 33587 } }
I20260809 12:10:43.141206 8926 leader_election.cc:288] T f79823fb579a4914b2b61f7d1165d91a P c40926a9373242b0961349a2719c22ab [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers df0db1a138554d00956d87f5c172ee10 (127.0.222.1:43733), 94413a87e0a14ef195baa2a799a9e0a8 (127.0.222.2:33587)
I20260809 12:10:43.148478 8913 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 12:10:43.141543 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.222.3:39961 (local address 127.0.222.1:43733)
0809 12:10:43.141769 (+ 226us) server_negotiation.cc:206] Beginning negotiation
0809 12:10:43.141776 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0809 12:10:43.141797 (+ 21us) server_negotiation.cc:407] Connection header received
0809 12:10:43.141993 (+ 196us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 12:10:43.141999 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 12:10:43.142082 (+ 83us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 12:10:43.142172 (+ 90us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0809 12:10:43.142586 (+ 414us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 12:10:43.143599 (+ 1013us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 12:10:43.146087 (+ 2488us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 12:10:43.147414 (+ 1327us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 12:10:43.147555 (+ 141us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 12:10:43.147656 (+ 101us) server_negotiation.cc:1035] Waiting for connection context
0809 12:10:43.148051 (+ 395us) server_negotiation.cc:299] Negotiation successful
0809 12:10:43.148169 (+ 118us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":70}
I20260809 12:10:43.148537 8884 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 12:10:43.141385 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.222.1:43733 (local address 127.0.222.3:39961)
0809 12:10:43.141652 (+ 267us) negotiation.cc:107] Waiting for socket to connect
0809 12:10:43.141675 (+ 23us) client_negotiation.cc:174] Beginning negotiation
0809 12:10:43.141803 (+ 128us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 12:10:43.142191 (+ 388us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 12:10:43.142195 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 12:10:43.142218 (+ 23us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0809 12:10:43.142435 (+ 217us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 12:10:43.142444 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 12:10:43.143736 (+ 1292us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 12:10:43.143744 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 12:10:43.145861 (+ 2117us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 12:10:43.145874 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 12:10:43.147605 (+ 1731us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 12:10:43.147611 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 12:10:43.147881 (+ 270us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 12:10:43.147905 (+ 24us) client_negotiation.cc:769] Sending connection context
0809 12:10:43.148015 (+ 110us) client_negotiation.cc:240] Negotiation successful
0809 12:10:43.148144 (+ 129us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":52}
I20260809 12:10:43.148823 8917 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 12:10:43.141981 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.222.3:45639 (local address 127.0.222.2:33587)
0809 12:10:43.142282 (+ 301us) server_negotiation.cc:206] Beginning negotiation
0809 12:10:43.142287 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0809 12:10:43.142303 (+ 16us) server_negotiation.cc:407] Connection header received
0809 12:10:43.142359 (+ 56us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 12:10:43.142363 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 12:10:43.142418 (+ 55us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 12:10:43.142523 (+ 105us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0809 12:10:43.143120 (+ 597us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 12:10:43.144182 (+ 1062us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 12:10:43.146383 (+ 2201us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 12:10:43.147624 (+ 1241us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 12:10:43.147769 (+ 145us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 12:10:43.147876 (+ 107us) server_negotiation.cc:1035] Waiting for connection context
0809 12:10:43.148358 (+ 482us) server_negotiation.cc:299] Negotiation successful
0809 12:10:43.148474 (+ 116us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":69}
I20260809 12:10:43.149447 8885 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 12:10:43.141822 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.222.2:33587 (local address 127.0.222.3:45639)
0809 12:10:43.141986 (+ 164us) negotiation.cc:107] Waiting for socket to connect
0809 12:10:43.142003 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0809 12:10:43.142109 (+ 106us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 12:10:43.142540 (+ 431us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 12:10:43.142545 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 12:10:43.142567 (+ 22us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0809 12:10:43.142905 (+ 338us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 12:10:43.142918 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 12:10:43.144353 (+ 1435us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 12:10:43.144359 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 12:10:43.146183 (+ 1824us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 12:10:43.146196 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 12:10:43.147820 (+ 1624us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 12:10:43.147827 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 12:10:43.148190 (+ 363us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 12:10:43.148214 (+ 24us) client_negotiation.cc:769] Sending connection context
0809 12:10:43.148332 (+ 118us) client_negotiation.cc:240] Negotiation successful
0809 12:10:43.148457 (+ 125us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":35}
I20260809 12:10:43.150005 8526 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f79823fb579a4914b2b61f7d1165d91a" candidate_uuid: "c40926a9373242b0961349a2719c22ab" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "df0db1a138554d00956d87f5c172ee10" is_pre_election: true
I20260809 12:10:43.150909 8526 raft_consensus.cc:2454] T f79823fb579a4914b2b61f7d1165d91a P df0db1a138554d00956d87f5c172ee10 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate c40926a9373242b0961349a2719c22ab in term 0.
I20260809 12:10:43.150695 8678 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f79823fb579a4914b2b61f7d1165d91a" candidate_uuid: "c40926a9373242b0961349a2719c22ab" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "94413a87e0a14ef195baa2a799a9e0a8" is_pre_election: true
I20260809 12:10:43.152040 8756 leader_election.cc:302] T f79823fb579a4914b2b61f7d1165d91a P c40926a9373242b0961349a2719c22ab [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: c40926a9373242b0961349a2719c22ab, df0db1a138554d00956d87f5c172ee10; no voters:
I20260809 12:10:43.151974 8678 raft_consensus.cc:2454] T f79823fb579a4914b2b61f7d1165d91a P 94413a87e0a14ef195baa2a799a9e0a8 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate c40926a9373242b0961349a2719c22ab in term 0.
I20260809 12:10:43.152616 8926 raft_consensus.cc:2781] T f79823fb579a4914b2b61f7d1165d91a P c40926a9373242b0961349a2719c22ab [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260809 12:10:43.152904 8926 raft_consensus.cc:482] T f79823fb579a4914b2b61f7d1165d91a P c40926a9373242b0961349a2719c22ab [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260809 12:10:43.153155 8926 raft_consensus.cc:3037] T f79823fb579a4914b2b61f7d1165d91a P c40926a9373242b0961349a2719c22ab [term 0 FOLLOWER]: Advancing to term 1
I20260809 12:10:43.156769 8926 raft_consensus.cc:504] T f79823fb579a4914b2b61f7d1165d91a P c40926a9373242b0961349a2719c22ab [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df0db1a138554d00956d87f5c172ee10" member_type: VOTER last_known_addr { host: "127.0.222.1" port: 43733 } } peers { permanent_uuid: "c40926a9373242b0961349a2719c22ab" member_type: VOTER last_known_addr { host: "127.0.222.3" port: 37357 } } peers { permanent_uuid: "94413a87e0a14ef195baa2a799a9e0a8" member_type: VOTER last_known_addr { host: "127.0.222.2" port: 33587 } }
I20260809 12:10:43.157640 8926 leader_election.cc:288] T f79823fb579a4914b2b61f7d1165d91a P c40926a9373242b0961349a2719c22ab [CANDIDATE]: Term 1 election: Requested vote from peers df0db1a138554d00956d87f5c172ee10 (127.0.222.1:43733), 94413a87e0a14ef195baa2a799a9e0a8 (127.0.222.2:33587)
I20260809 12:10:43.158207 8526 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f79823fb579a4914b2b61f7d1165d91a" candidate_uuid: "c40926a9373242b0961349a2719c22ab" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "df0db1a138554d00956d87f5c172ee10"
I20260809 12:10:43.158221 8678 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f79823fb579a4914b2b61f7d1165d91a" candidate_uuid: "c40926a9373242b0961349a2719c22ab" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "94413a87e0a14ef195baa2a799a9e0a8"
I20260809 12:10:43.158492 8526 raft_consensus.cc:3037] T f79823fb579a4914b2b61f7d1165d91a P df0db1a138554d00956d87f5c172ee10 [term 0 FOLLOWER]: Advancing to term 1
I20260809 12:10:43.158500 8678 raft_consensus.cc:3037] T f79823fb579a4914b2b61f7d1165d91a P 94413a87e0a14ef195baa2a799a9e0a8 [term 0 FOLLOWER]: Advancing to term 1
I20260809 12:10:43.162683 8526 raft_consensus.cc:2454] T f79823fb579a4914b2b61f7d1165d91a P df0db1a138554d00956d87f5c172ee10 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c40926a9373242b0961349a2719c22ab in term 1.
I20260809 12:10:43.163548 8756 leader_election.cc:302] T f79823fb579a4914b2b61f7d1165d91a P c40926a9373242b0961349a2719c22ab [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: c40926a9373242b0961349a2719c22ab, df0db1a138554d00956d87f5c172ee10; no voters:
I20260809 12:10:43.164093 8926 raft_consensus.cc:2781] T f79823fb579a4914b2b61f7d1165d91a P c40926a9373242b0961349a2719c22ab [term 1 FOLLOWER]: Leader election won for term 1
I20260809 12:10:43.166520 8926 raft_consensus.cc:686] T f79823fb579a4914b2b61f7d1165d91a P c40926a9373242b0961349a2719c22ab [term 1 LEADER]: Becoming Leader. State: Replica: c40926a9373242b0961349a2719c22ab, State: Running, Role: LEADER
I20260809 12:10:43.167060 8678 raft_consensus.cc:2454] T f79823fb579a4914b2b61f7d1165d91a P 94413a87e0a14ef195baa2a799a9e0a8 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c40926a9373242b0961349a2719c22ab in term 1.
I20260809 12:10:43.167716 8926 consensus_queue.cc:237] T f79823fb579a4914b2b61f7d1165d91a P c40926a9373242b0961349a2719c22ab [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df0db1a138554d00956d87f5c172ee10" member_type: VOTER last_known_addr { host: "127.0.222.1" port: 43733 } } peers { permanent_uuid: "c40926a9373242b0961349a2719c22ab" member_type: VOTER last_known_addr { host: "127.0.222.3" port: 37357 } } peers { permanent_uuid: "94413a87e0a14ef195baa2a799a9e0a8" member_type: VOTER last_known_addr { host: "127.0.222.2" port: 33587 } }
I20260809 12:10:43.175161 7547 catalog_manager.cc:5495] T f79823fb579a4914b2b61f7d1165d91a P c40926a9373242b0961349a2719c22ab reported cstate change: term changed from 0 to 1, leader changed from <none> to c40926a9373242b0961349a2719c22ab (127.0.222.3). New cstate: current_term: 1 leader_uuid: "c40926a9373242b0961349a2719c22ab" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df0db1a138554d00956d87f5c172ee10" member_type: VOTER last_known_addr { host: "127.0.222.1" port: 43733 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "c40926a9373242b0961349a2719c22ab" member_type: VOTER last_known_addr { host: "127.0.222.3" port: 37357 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "94413a87e0a14ef195baa2a799a9e0a8" member_type: VOTER last_known_addr { host: "127.0.222.2" port: 33587 } health_report { overall_health: UNKNOWN } } }
I20260809 12:10:43.202875 8915 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 12:10:43.196197 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33586 (local address 127.0.222.3:37357)
0809 12:10:43.196723 (+ 526us) server_negotiation.cc:206] Beginning negotiation
0809 12:10:43.196728 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0809 12:10:43.196742 (+ 14us) server_negotiation.cc:407] Connection header received
0809 12:10:43.196824 (+ 82us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 12:10:43.196828 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 12:10:43.196909 (+ 81us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 12:10:43.197042 (+ 133us) server_negotiation.cc:226] Negotiated authn=TOKEN
0809 12:10:43.197468 (+ 426us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 12:10:43.198655 (+ 1187us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 12:10:43.200622 (+ 1967us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 12:10:43.201131 (+ 509us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 12:10:43.201231 (+ 100us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 12:10:43.201799 (+ 568us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0809 12:10:43.202338 (+ 539us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0809 12:10:43.202449 (+ 111us) server_negotiation.cc:1035] Waiting for connection context
0809 12:10:43.202547 (+ 98us) server_negotiation.cc:299] Negotiation successful
0809 12:10:43.202613 (+ 66us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":403}
W20260809 12:10:43.211155 8729 tablet.cc:2369] T f79823fb579a4914b2b61f7d1165d91a P 94413a87e0a14ef195baa2a799a9e0a8: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260809 12:10:43.215979 8526 raft_consensus.cc:1261] T f79823fb579a4914b2b61f7d1165d91a P df0db1a138554d00956d87f5c172ee10 [term 1 FOLLOWER]: Refusing update from remote peer c40926a9373242b0961349a2719c22ab: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260809 12:10:43.216086 8678 raft_consensus.cc:1261] T f79823fb579a4914b2b61f7d1165d91a P 94413a87e0a14ef195baa2a799a9e0a8 [term 1 FOLLOWER]: Refusing update from remote peer c40926a9373242b0961349a2719c22ab: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260809 12:10:43.217311 8927 consensus_queue.cc:1035] T f79823fb579a4914b2b61f7d1165d91a P c40926a9373242b0961349a2719c22ab [LEADER]: Connected to new peer: Peer: permanent_uuid: "94413a87e0a14ef195baa2a799a9e0a8" member_type: VOTER last_known_addr { host: "127.0.222.2" port: 33587 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260809 12:10:43.218122 8926 consensus_queue.cc:1035] T f79823fb579a4914b2b61f7d1165d91a P c40926a9373242b0961349a2719c22ab [LEADER]: Connected to new peer: Peer: permanent_uuid: "df0db1a138554d00956d87f5c172ee10" member_type: VOTER last_known_addr { host: "127.0.222.1" port: 43733 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260809 12:10:43.239804 8930 mvcc.cc:205] Tried to move back new op lower bound from 7316592407399940096 to 7316592407220899840. Current Snapshot: MvccSnapshot[applied={T|T < 7316592407399940096}]
I20260809 12:10:43.242573 8932 mvcc.cc:205] Tried to move back new op lower bound from 7316592407399940096 to 7316592407220899840. Current Snapshot: MvccSnapshot[applied={T|T < 7316592407399940096}]
I20260809 12:10:43.242635 8931 mvcc.cc:205] Tried to move back new op lower bound from 7316592407399940096 to 7316592407220899840. Current Snapshot: MvccSnapshot[applied={T|T < 7316592407399940096}]
I20260809 12:10:43.277055 7548 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:33094:
table { table_name: "test-table" } modify_external_catalogs: true
I20260809 12:10:43.277426 7548 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:33094:
table { table_name: "test-table" } modify_external_catalogs: true
I20260809 12:10:43.286710 7548 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 9cd5d41083d147f788d83a438b6f7fc2: Sending DeleteTablet for 3 replicas of tablet f79823fb579a4914b2b61f7d1165d91a
I20260809 12:10:43.288296 8808 tablet_service.cc:1514] Processing DeleteTablet for tablet f79823fb579a4914b2b61f7d1165d91a with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-09 12:10:43 UTC) from {username='oryx', principal='oryx/127.0.222.62@KRBTEST.COM'} at 127.0.0.1:33572
I20260809 12:10:43.288465 8656 tablet_service.cc:1514] Processing DeleteTablet for tablet f79823fb579a4914b2b61f7d1165d91a with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-09 12:10:43 UTC) from {username='oryx', principal='oryx/127.0.222.62@KRBTEST.COM'} at 127.0.0.1:35430
I20260809 12:10:43.290130 8505 tablet_service.cc:1514] Processing DeleteTablet for tablet f79823fb579a4914b2b61f7d1165d91a with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-09 12:10:43 UTC) from {username='oryx', principal='oryx/127.0.222.62@KRBTEST.COM'} at 127.0.0.1:49254
I20260809 12:10:43.303359 8943 tablet_replica.cc:330] T f79823fb579a4914b2b61f7d1165d91a P df0db1a138554d00956d87f5c172ee10: stopping tablet replica
I20260809 12:10:43.304323 8943 raft_consensus.cc:2229] T f79823fb579a4914b2b61f7d1165d91a P df0db1a138554d00956d87f5c172ee10 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260809 12:10:43.305537 8942 tablet_replica.cc:330] T f79823fb579a4914b2b61f7d1165d91a P 94413a87e0a14ef195baa2a799a9e0a8: stopping tablet replica
I20260809 12:10:43.305516 8943 raft_consensus.cc:2258] T f79823fb579a4914b2b61f7d1165d91a P df0db1a138554d00956d87f5c172ee10 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260809 12:10:43.305912 8941 tablet_replica.cc:330] T f79823fb579a4914b2b61f7d1165d91a P c40926a9373242b0961349a2719c22ab: stopping tablet replica
I20260809 12:10:43.306041 8942 raft_consensus.cc:2229] T f79823fb579a4914b2b61f7d1165d91a P 94413a87e0a14ef195baa2a799a9e0a8 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260809 12:10:43.306475 8941 raft_consensus.cc:2229] T f79823fb579a4914b2b61f7d1165d91a P c40926a9373242b0961349a2719c22ab [term 1 LEADER]: Raft consensus shutting down.
I20260809 12:10:43.307480 8941 raft_consensus.cc:2258] T f79823fb579a4914b2b61f7d1165d91a P c40926a9373242b0961349a2719c22ab [term 1 FOLLOWER]: Raft consensus is shut down!
I20260809 12:10:43.306941 8942 raft_consensus.cc:2258] T f79823fb579a4914b2b61f7d1165d91a P 94413a87e0a14ef195baa2a799a9e0a8 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260809 12:10:43.308826 8943 ts_tablet_manager.cc:1905] T f79823fb579a4914b2b61f7d1165d91a P df0db1a138554d00956d87f5c172ee10: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260809 12:10:43.310696 8941 ts_tablet_manager.cc:1905] T f79823fb579a4914b2b61f7d1165d91a P c40926a9373242b0961349a2719c22ab: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260809 12:10:43.312922 888 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskgH5Pjm/build/asan/bin/kudu pid 7516
I20260809 12:10:43.314563 7579 generic_service.cc:196] Checking for leaks (request via RPC)
I20260809 12:10:43.320334 8943 ts_tablet_manager.cc:1918] T f79823fb579a4914b2b61f7d1165d91a P df0db1a138554d00956d87f5c172ee10: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260809 12:10:43.320658 8943 log.cc:1196] T f79823fb579a4914b2b61f7d1165d91a P df0db1a138554d00956d87f5c172ee10: Deleting WAL directory at /tmp/dist-test-taskgH5Pjm/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786277382073938-888-0/minicluster-data/ts-0/wal/wals/f79823fb579a4914b2b61f7d1165d91a
I20260809 12:10:43.320996 8942 ts_tablet_manager.cc:1905] T f79823fb579a4914b2b61f7d1165d91a P 94413a87e0a14ef195baa2a799a9e0a8: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260809 12:10:43.321934 8943 ts_tablet_manager.cc:1939] T f79823fb579a4914b2b61f7d1165d91a P df0db1a138554d00956d87f5c172ee10: Deleting consensus metadata
I20260809 12:10:43.334784 8941 ts_tablet_manager.cc:1918] T f79823fb579a4914b2b61f7d1165d91a P c40926a9373242b0961349a2719c22ab: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260809 12:10:43.334836 8942 ts_tablet_manager.cc:1918] T f79823fb579a4914b2b61f7d1165d91a P 94413a87e0a14ef195baa2a799a9e0a8: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260809 12:10:43.335070 8942 log.cc:1196] T f79823fb579a4914b2b61f7d1165d91a P 94413a87e0a14ef195baa2a799a9e0a8: Deleting WAL directory at /tmp/dist-test-taskgH5Pjm/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786277382073938-888-0/minicluster-data/ts-1/wal/wals/f79823fb579a4914b2b61f7d1165d91a
I20260809 12:10:43.336272 8942 ts_tablet_manager.cc:1939] T f79823fb579a4914b2b61f7d1165d91a P 94413a87e0a14ef195baa2a799a9e0a8: Deleting consensus metadata
I20260809 12:10:43.339720 8941 log.cc:1196] T f79823fb579a4914b2b61f7d1165d91a P c40926a9373242b0961349a2719c22ab: Deleting WAL directory at /tmp/dist-test-taskgH5Pjm/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786277382073938-888-0/minicluster-data/ts-2/wal/wals/f79823fb579a4914b2b61f7d1165d91a
I20260809 12:10:43.340869 8941 ts_tablet_manager.cc:1939] T f79823fb579a4914b2b61f7d1165d91a P c40926a9373242b0961349a2719c22ab: Deleting consensus metadata
I20260809 12:10:43.340998 8896 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 12:10:43.292016 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46574 (local address 127.0.222.61:32831)
0809 12:10:43.292267 (+ 251us) server_negotiation.cc:206] Beginning negotiation
0809 12:10:43.292277 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0809 12:10:43.309391 (+ 17114us) server_negotiation.cc:407] Connection header received
0809 12:10:43.310227 (+ 836us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 12:10:43.310236 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 12:10:43.310355 (+ 119us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 12:10:43.310439 (+ 84us) server_negotiation.cc:226] Negotiated authn=SASL
0809 12:10:43.314628 (+ 4189us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 12:10:43.316286 (+ 1658us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 12:10:43.323908 (+ 7622us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 12:10:43.324855 (+ 947us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 12:10:43.324952 (+ 97us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 12:10:43.330297 (+ 5345us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 12:10:43.330304 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 12:10:43.330309 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0809 12:10:43.330365 (+ 56us) server_negotiation.cc:922] calling sasl_server_start()
0809 12:10:43.335642 (+ 5277us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 12:10:43.337141 (+ 1499us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 12:10:43.337149 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 12:10:43.337153 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0809 12:10:43.337393 (+ 240us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 12:10:43.337989 (+ 596us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 12:10:43.337996 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 12:10:43.338000 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0809 12:10:43.338617 (+ 617us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 12:10:43.339060 (+ 443us) server_negotiation.cc:1035] Waiting for connection context
0809 12:10:43.340128 (+ 1068us) server_negotiation.cc:299] Negotiation successful
0809 12:10:43.340449 (+ 321us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":49}
I20260809 12:10:43.368260 8895 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 12:10:43.309844 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42482 (local address 127.0.222.60:39543)
0809 12:10:43.310117 (+ 273us) server_negotiation.cc:206] Beginning negotiation
0809 12:10:43.310123 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0809 12:10:43.343775 (+ 33652us) server_negotiation.cc:407] Connection header received
0809 12:10:43.344003 (+ 228us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 12:10:43.344009 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 12:10:43.344104 (+ 95us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 12:10:43.344212 (+ 108us) server_negotiation.cc:226] Negotiated authn=SASL
0809 12:10:43.347068 (+ 2856us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 12:10:43.348614 (+ 1546us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 12:10:43.351501 (+ 2887us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 12:10:43.352733 (+ 1232us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 12:10:43.352841 (+ 108us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 12:10:43.358038 (+ 5197us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 12:10:43.358045 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 12:10:43.358051 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0809 12:10:43.358094 (+ 43us) server_negotiation.cc:922] calling sasl_server_start()
0809 12:10:43.363926 (+ 5832us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 12:10:43.365206 (+ 1280us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 12:10:43.365212 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 12:10:43.365216 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0809 12:10:43.365413 (+ 197us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 12:10:43.366068 (+ 655us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 12:10:43.366074 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 12:10:43.366077 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0809 12:10:43.366466 (+ 389us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 12:10:43.366759 (+ 293us) server_negotiation.cc:1035] Waiting for connection context
0809 12:10:43.367273 (+ 514us) server_negotiation.cc:299] Negotiation successful
0809 12:10:43.367599 (+ 326us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":67}
I20260809 12:10:43.473250 7534 catalog_manager.cc:4841] TS c40926a9373242b0961349a2719c22ab (127.0.222.3:37357): tablet f79823fb579a4914b2b61f7d1165d91a (table test-table [id=ec147d97ec774ba395a9d447b9aae730]) successfully deleted
I20260809 12:10:43.474263 888 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskgH5Pjm/build/asan/bin/kudu with pid 7516
W20260809 12:10:43.501034 8609 connection.cc:535] server connection from 127.0.0.1:35430 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
I20260809 12:10:43.526204 8896 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 12:10:43.506228 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46588 (local address 127.0.222.61:32831)
0809 12:10:43.506479 (+ 251us) server_negotiation.cc:206] Beginning negotiation
0809 12:10:43.506488 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0809 12:10:43.509807 (+ 3319us) server_negotiation.cc:407] Connection header received
0809 12:10:43.510088 (+ 281us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 12:10:43.510098 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 12:10:43.510172 (+ 74us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 12:10:43.510287 (+ 115us) server_negotiation.cc:226] Negotiated authn=SASL
0809 12:10:43.512201 (+ 1914us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 12:10:43.513287 (+ 1086us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 12:10:43.515317 (+ 2030us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 12:10:43.515962 (+ 645us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 12:10:43.516098 (+ 136us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 12:10:43.518866 (+ 2768us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 12:10:43.518871 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 12:10:43.518874 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0809 12:10:43.518908 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0809 12:10:43.522489 (+ 3581us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 12:10:43.523799 (+ 1310us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 12:10:43.523812 (+ 13us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 12:10:43.523816 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0809 12:10:43.523974 (+ 158us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 12:10:43.524449 (+ 475us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 12:10:43.524455 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 12:10:43.524459 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0809 12:10:43.524746 (+ 287us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 12:10:43.525011 (+ 265us) server_negotiation.cc:1035] Waiting for connection context
0809 12:10:43.525509 (+ 498us) server_negotiation.cc:299] Negotiation successful
0809 12:10:43.525740 (+ 231us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":81}
I20260809 12:10:43.526352 8895 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 12:10:43.506764 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42488 (local address 127.0.222.60:39543)
0809 12:10:43.507032 (+ 268us) server_negotiation.cc:206] Beginning negotiation
0809 12:10:43.507041 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0809 12:10:43.509417 (+ 2376us) server_negotiation.cc:407] Connection header received
0809 12:10:43.509726 (+ 309us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 12:10:43.509732 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 12:10:43.509825 (+ 93us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 12:10:43.509905 (+ 80us) server_negotiation.cc:226] Negotiated authn=SASL
0809 12:10:43.512529 (+ 2624us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 12:10:43.513562 (+ 1033us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 12:10:43.515000 (+ 1438us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 12:10:43.515558 (+ 558us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 12:10:43.515691 (+ 133us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 12:10:43.519476 (+ 3785us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 12:10:43.519486 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 12:10:43.519490 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0809 12:10:43.519557 (+ 67us) server_negotiation.cc:922] calling sasl_server_start()
0809 12:10:43.522896 (+ 3339us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 12:10:43.523884 (+ 988us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 12:10:43.523892 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 12:10:43.523898 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0809 12:10:43.524046 (+ 148us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 12:10:43.524499 (+ 453us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 12:10:43.524504 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 12:10:43.524508 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0809 12:10:43.524891 (+ 383us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 12:10:43.525151 (+ 260us) server_negotiation.cc:1035] Waiting for connection context
0809 12:10:43.525726 (+ 575us) server_negotiation.cc:299] Negotiation successful
0809 12:10:43.525969 (+ 243us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":79}
I20260809 12:10:44.262005 8954 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 12:10:44.260620 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.222.62:41983 (local address 127.0.222.1:50409)
0809 12:10:44.261641 (+ 1021us) negotiation.cc:107] Waiting for socket to connect
0809 12:10:44.261796 (+ 155us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.222.62:41983: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":565,"thread_start_us":440,"threads_started":1}
I20260809 12:10:44.262032 8955 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 12:10:44.260782 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.222.62:41983 (local address 127.0.222.2:38941)
0809 12:10:44.261739 (+ 957us) negotiation.cc:107] Waiting for socket to connect
0809 12:10:44.261864 (+ 125us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.222.62:41983: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":847,"thread_start_us":728,"threads_started":1}
W20260809 12:10:44.262809 8727 heartbeater.cc:644] Failed to heartbeat to 127.0.222.62:41983 (0 consecutive failures): Network error: Failed to send heartbeat to master: Client connection negotiation failed: client connection to 127.0.222.62:41983: connect: Connection refused (error 111)
W20260809 12:10:44.262813 8573 heartbeater.cc:644] Failed to heartbeat to 127.0.222.62:41983 (0 consecutive failures): Network error: Failed to send heartbeat to master: Client connection negotiation failed: client connection to 127.0.222.62:41983: connect: Connection refused (error 111)
I20260809 12:10:44.269285 8956 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 12:10:44.268381 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.222.62:41983 (local address 127.0.222.3:58391)
0809 12:10:44.268857 (+ 476us) negotiation.cc:107] Waiting for socket to connect
0809 12:10:44.269051 (+ 194us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.222.62:41983: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":358,"thread_start_us":270,"threads_started":1}
W20260809 12:10:44.269661 8879 heartbeater.cc:644] Failed to heartbeat to 127.0.222.62:41983 (0 consecutive failures): Network error: Failed to send heartbeat to master: Client connection negotiation failed: client connection to 127.0.222.62:41983: connect: Connection refused (error 111)
I20260809 12:10:44.786279 8957 raft_consensus.cc:482] T 00000000000000000000000000000000 P cbbb52854812417da2325b27e509812d [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 9cd5d41083d147f788d83a438b6f7fc2)
I20260809 12:10:44.786501 8959 raft_consensus.cc:482] T 00000000000000000000000000000000 P e2a7b439c3084cd680be92ac6d1925db [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 9cd5d41083d147f788d83a438b6f7fc2)
I20260809 12:10:44.786630 8957 raft_consensus.cc:504] T 00000000000000000000000000000000 P cbbb52854812417da2325b27e509812d [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "9cd5d41083d147f788d83a438b6f7fc2" member_type: VOTER last_known_addr { host: "127.0.222.62" port: 41983 } } peers { permanent_uuid: "cbbb52854812417da2325b27e509812d" member_type: VOTER last_known_addr { host: "127.0.222.61" port: 32831 } attrs { promote: false } } peers { permanent_uuid: "e2a7b439c3084cd680be92ac6d1925db" member_type: VOTER last_known_addr { host: "127.0.222.60" port: 39543 } attrs { promote: false } }
I20260809 12:10:44.786731 8959 raft_consensus.cc:504] T 00000000000000000000000000000000 P e2a7b439c3084cd680be92ac6d1925db [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "9cd5d41083d147f788d83a438b6f7fc2" member_type: VOTER last_known_addr { host: "127.0.222.62" port: 41983 } } peers { permanent_uuid: "cbbb52854812417da2325b27e509812d" member_type: VOTER last_known_addr { host: "127.0.222.61" port: 32831 } attrs { promote: false } } peers { permanent_uuid: "e2a7b439c3084cd680be92ac6d1925db" member_type: VOTER last_known_addr { host: "127.0.222.60" port: 39543 } attrs { promote: false } }
I20260809 12:10:44.787791 8959 leader_election.cc:288] T 00000000000000000000000000000000 P e2a7b439c3084cd680be92ac6d1925db [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 9cd5d41083d147f788d83a438b6f7fc2 (127.0.222.62:41983), cbbb52854812417da2325b27e509812d (127.0.222.61:32831)
I20260809 12:10:44.789093 8960 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 12:10:44.787906 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.222.62:41983 (local address 127.0.0.1:33522)
0809 12:10:44.788493 (+ 587us) negotiation.cc:107] Waiting for socket to connect
0809 12:10:44.788724 (+ 231us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.222.62:41983: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":445,"thread_start_us":455,"threads_started":1}
W20260809 12:10:44.789563 8369 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.222.62:41983: connect: Connection refused (error 111)
I20260809 12:10:44.794275 8957 leader_election.cc:288] T 00000000000000000000000000000000 P cbbb52854812417da2325b27e509812d [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 9cd5d41083d147f788d83a438b6f7fc2 (127.0.222.62:41983), e2a7b439c3084cd680be92ac6d1925db (127.0.222.60:39543)
I20260809 12:10:44.802563 8961 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 12:10:44.801601 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.222.62:41983 (local address 127.0.0.1:33532)
0809 12:10:44.801897 (+ 296us) negotiation.cc:107] Waiting for socket to connect
0809 12:10:44.802285 (+ 388us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.222.62:41983: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":101}
I20260809 12:10:44.803974 8966 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 12:10:44.801261 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.222.62:41983 (local address 127.0.0.1:33528)
0809 12:10:44.803415 (+ 2154us) negotiation.cc:107] Waiting for socket to connect
0809 12:10:44.803639 (+ 224us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.222.62:41983: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":2027,"threads_started":1}
W20260809 12:10:44.802842 8369 leader_election.cc:334] T 00000000000000000000000000000000 P e2a7b439c3084cd680be92ac6d1925db [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 9cd5d41083d147f788d83a438b6f7fc2 (127.0.222.62:41983): Network error: Client connection negotiation failed: client connection to 127.0.222.62:41983: connect: Connection refused (error 111)
W20260809 12:10:44.806151 8167 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.222.62:41983: connect: Connection refused (error 111)
I20260809 12:10:44.808151 8966 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 12:10:44.807690 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.222.62:41983 (local address 127.0.0.1:33554)
0809 12:10:44.807879 (+ 189us) negotiation.cc:107] Waiting for socket to connect
0809 12:10:44.807985 (+ 106us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.222.62:41983: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":66}
W20260809 12:10:44.808514 8167 leader_election.cc:334] T 00000000000000000000000000000000 P cbbb52854812417da2325b27e509812d [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 9cd5d41083d147f788d83a438b6f7fc2 (127.0.222.62:41983): Network error: Client connection negotiation failed: client connection to 127.0.222.62:41983: connect: Connection refused (error 111)
I20260809 12:10:44.809113 8963 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 12:10:44.788531 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.222.60:39543 (local address 127.0.0.1:42490)
0809 12:10:44.800514 (+ 11983us) negotiation.cc:107] Waiting for socket to connect
0809 12:10:44.800540 (+ 26us) client_negotiation.cc:174] Beginning negotiation
0809 12:10:44.800718 (+ 178us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 12:10:44.802316 (+ 1598us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 12:10:44.802320 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 12:10:44.802366 (+ 46us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0809 12:10:44.802955 (+ 589us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 12:10:44.802997 (+ 42us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 12:10:44.804635 (+ 1638us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 12:10:44.804640 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 12:10:44.806812 (+ 2172us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 12:10:44.806830 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 12:10:44.808219 (+ 1389us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 12:10:44.808225 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 12:10:44.808448 (+ 223us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 12:10:44.808483 (+ 35us) client_negotiation.cc:769] Sending connection context
0809 12:10:44.808598 (+ 115us) client_negotiation.cc:240] Negotiation successful
0809 12:10:44.808751 (+ 153us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":11828,"thread_start_us":11647,"threads_started":1}
I20260809 12:10:44.809237 8965 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 12:10:44.801026 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42490 (local address 127.0.222.60:39543)
0809 12:10:44.801681 (+ 655us) server_negotiation.cc:206] Beginning negotiation
0809 12:10:44.801687 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0809 12:10:44.801724 (+ 37us) server_negotiation.cc:407] Connection header received
0809 12:10:44.801798 (+ 74us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 12:10:44.801802 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 12:10:44.801890 (+ 88us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 12:10:44.802003 (+ 113us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0809 12:10:44.803232 (+ 1229us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 12:10:44.804441 (+ 1209us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 12:10:44.807059 (+ 2618us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 12:10:44.808042 (+ 983us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 12:10:44.808192 (+ 150us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 12:10:44.808295 (+ 103us) server_negotiation.cc:1035] Waiting for connection context
0809 12:10:44.808664 (+ 369us) server_negotiation.cc:299] Negotiation successful
0809 12:10:44.808820 (+ 156us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":463,"thread_start_us":295,"threads_started":1}
I20260809 12:10:44.809283 8960 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 12:10:44.788091 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.222.61:32831 (local address 127.0.0.1:46600)
0809 12:10:44.795028 (+ 6937us) negotiation.cc:107] Waiting for socket to connect
0809 12:10:44.795055 (+ 27us) client_negotiation.cc:174] Beginning negotiation
0809 12:10:44.795415 (+ 360us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 12:10:44.801303 (+ 5888us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 12:10:44.801312 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 12:10:44.801354 (+ 42us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0809 12:10:44.801690 (+ 336us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 12:10:44.801718 (+ 28us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 12:10:44.804201 (+ 2483us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 12:10:44.804209 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 12:10:44.806447 (+ 2238us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 12:10:44.806461 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 12:10:44.808393 (+ 1932us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 12:10:44.808400 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 12:10:44.808657 (+ 257us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 12:10:44.808694 (+ 37us) client_negotiation.cc:769] Sending connection context
0809 12:10:44.808775 (+ 81us) client_negotiation.cc:240] Negotiation successful
0809 12:10:44.808900 (+ 125us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":6769,"mutex_wait_us":26,"thread_start_us":12621,"threads_started":1}
I20260809 12:10:44.809769 8962 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 12:10:44.788201 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46600 (local address 127.0.222.61:32831)
0809 12:10:44.800833 (+ 12632us) server_negotiation.cc:206] Beginning negotiation
0809 12:10:44.800841 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0809 12:10:44.800866 (+ 25us) server_negotiation.cc:407] Connection header received
0809 12:10:44.800939 (+ 73us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 12:10:44.800948 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 12:10:44.801034 (+ 86us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 12:10:44.801878 (+ 844us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0809 12:10:44.802289 (+ 411us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 12:10:44.804001 (+ 1712us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 12:10:44.806604 (+ 2603us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 12:10:44.808129 (+ 1525us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 12:10:44.808934 (+ 805us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 12:10:44.809039 (+ 105us) server_negotiation.cc:1035] Waiting for connection context
0809 12:10:44.809092 (+ 53us) server_negotiation.cc:299] Negotiation successful
0809 12:10:44.809451 (+ 359us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":12496,"thread_start_us":12743,"threads_started":1}
I20260809 12:10:44.810009 8396 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "cbbb52854812417da2325b27e509812d" candidate_term: 2 candidate_status { last_received { term: 1 index: 12 } } ignore_live_leader: false dest_uuid: "e2a7b439c3084cd680be92ac6d1925db" is_pre_election: true
I20260809 12:10:44.810348 8396 raft_consensus.cc:2454] T 00000000000000000000000000000000 P e2a7b439c3084cd680be92ac6d1925db [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate cbbb52854812417da2325b27e509812d in term 1.
I20260809 12:10:44.810315 8193 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "e2a7b439c3084cd680be92ac6d1925db" candidate_term: 2 candidate_status { last_received { term: 1 index: 12 } } ignore_live_leader: false dest_uuid: "cbbb52854812417da2325b27e509812d" is_pre_election: true
I20260809 12:10:44.810583 8193 raft_consensus.cc:2454] T 00000000000000000000000000000000 P cbbb52854812417da2325b27e509812d [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e2a7b439c3084cd680be92ac6d1925db in term 1.
I20260809 12:10:44.810937 8168 leader_election.cc:302] T 00000000000000000000000000000000 P cbbb52854812417da2325b27e509812d [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: cbbb52854812417da2325b27e509812d, e2a7b439c3084cd680be92ac6d1925db; no voters: 9cd5d41083d147f788d83a438b6f7fc2
I20260809 12:10:44.811110 8372 leader_election.cc:302] T 00000000000000000000000000000000 P e2a7b439c3084cd680be92ac6d1925db [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: cbbb52854812417da2325b27e509812d, e2a7b439c3084cd680be92ac6d1925db; no voters: 9cd5d41083d147f788d83a438b6f7fc2
I20260809 12:10:44.811447 8957 raft_consensus.cc:2781] T 00000000000000000000000000000000 P cbbb52854812417da2325b27e509812d [term 1 FOLLOWER]: Leader pre-election won for term 2
I20260809 12:10:44.811467 8959 raft_consensus.cc:2781] T 00000000000000000000000000000000 P e2a7b439c3084cd680be92ac6d1925db [term 1 FOLLOWER]: Leader pre-election won for term 2
I20260809 12:10:44.811604 8957 raft_consensus.cc:482] T 00000000000000000000000000000000 P cbbb52854812417da2325b27e509812d [term 1 FOLLOWER]: Starting leader election (detected failure of leader 9cd5d41083d147f788d83a438b6f7fc2)
I20260809 12:10:44.811614 8959 raft_consensus.cc:482] T 00000000000000000000000000000000 P e2a7b439c3084cd680be92ac6d1925db [term 1 FOLLOWER]: Starting leader election (detected failure of leader 9cd5d41083d147f788d83a438b6f7fc2)
I20260809 12:10:44.811713 8959 raft_consensus.cc:3037] T 00000000000000000000000000000000 P e2a7b439c3084cd680be92ac6d1925db [term 1 FOLLOWER]: Advancing to term 2
I20260809 12:10:44.811713 8957 raft_consensus.cc:3037] T 00000000000000000000000000000000 P cbbb52854812417da2325b27e509812d [term 1 FOLLOWER]: Advancing to term 2
I20260809 12:10:44.814821 8959 raft_consensus.cc:504] T 00000000000000000000000000000000 P e2a7b439c3084cd680be92ac6d1925db [term 2 FOLLOWER]: Starting leader election with config: opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "9cd5d41083d147f788d83a438b6f7fc2" member_type: VOTER last_known_addr { host: "127.0.222.62" port: 41983 } } peers { permanent_uuid: "cbbb52854812417da2325b27e509812d" member_type: VOTER last_known_addr { host: "127.0.222.61" port: 32831 } attrs { promote: false } } peers { permanent_uuid: "e2a7b439c3084cd680be92ac6d1925db" member_type: VOTER last_known_addr { host: "127.0.222.60" port: 39543 } attrs { promote: false } }
I20260809 12:10:44.814826 8957 raft_consensus.cc:504] T 00000000000000000000000000000000 P cbbb52854812417da2325b27e509812d [term 2 FOLLOWER]: Starting leader election with config: opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "9cd5d41083d147f788d83a438b6f7fc2" member_type: VOTER last_known_addr { host: "127.0.222.62" port: 41983 } } peers { permanent_uuid: "cbbb52854812417da2325b27e509812d" member_type: VOTER last_known_addr { host: "127.0.222.61" port: 32831 } attrs { promote: false } } peers { permanent_uuid: "e2a7b439c3084cd680be92ac6d1925db" member_type: VOTER last_known_addr { host: "127.0.222.60" port: 39543 } attrs { promote: false } }
I20260809 12:10:44.815567 8957 leader_election.cc:288] T 00000000000000000000000000000000 P cbbb52854812417da2325b27e509812d [CANDIDATE]: Term 2 election: Requested vote from peers 9cd5d41083d147f788d83a438b6f7fc2 (127.0.222.62:41983), e2a7b439c3084cd680be92ac6d1925db (127.0.222.60:39543)
I20260809 12:10:44.815567 8959 leader_election.cc:288] T 00000000000000000000000000000000 P e2a7b439c3084cd680be92ac6d1925db [CANDIDATE]: Term 2 election: Requested vote from peers 9cd5d41083d147f788d83a438b6f7fc2 (127.0.222.62:41983), cbbb52854812417da2325b27e509812d (127.0.222.61:32831)
I20260809 12:10:44.816071 8396 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "cbbb52854812417da2325b27e509812d" candidate_term: 2 candidate_status { last_received { term: 1 index: 12 } } ignore_live_leader: false dest_uuid: "e2a7b439c3084cd680be92ac6d1925db"
I20260809 12:10:44.816354 8396 raft_consensus.cc:2379] T 00000000000000000000000000000000 P e2a7b439c3084cd680be92ac6d1925db [term 2 FOLLOWER]: Leader election vote request: Denying vote to candidate cbbb52854812417da2325b27e509812d in current term 2: Already voted for candidate e2a7b439c3084cd680be92ac6d1925db in this term.
I20260809 12:10:44.816382 8960 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 12:10:44.815940 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.222.62:41983 (local address 127.0.0.1:33572)
0809 12:10:44.816126 (+ 186us) negotiation.cc:107] Waiting for socket to connect
0809 12:10:44.816215 (+ 89us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.222.62:41983: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":64}
I20260809 12:10:44.816488 8963 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 12:10:44.816011 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.222.62:41983 (local address 127.0.0.1:33570)
0809 12:10:44.816204 (+ 193us) negotiation.cc:107] Waiting for socket to connect
0809 12:10:44.816309 (+ 105us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.222.62:41983: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":72}
I20260809 12:10:44.816545 8193 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "e2a7b439c3084cd680be92ac6d1925db" candidate_term: 2 candidate_status { last_received { term: 1 index: 12 } } ignore_live_leader: false dest_uuid: "cbbb52854812417da2325b27e509812d"
I20260809 12:10:44.816823 8193 raft_consensus.cc:2379] T 00000000000000000000000000000000 P cbbb52854812417da2325b27e509812d [term 2 FOLLOWER]: Leader election vote request: Denying vote to candidate e2a7b439c3084cd680be92ac6d1925db in current term 2: Already voted for candidate cbbb52854812417da2325b27e509812d in this term.
I20260809 12:10:44.817646 8960 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 12:10:44.817266 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.222.62:41983 (local address 127.0.0.1:33584)
0809 12:10:44.817424 (+ 158us) negotiation.cc:107] Waiting for socket to connect
0809 12:10:44.817515 (+ 91us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.222.62:41983: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":43}
I20260809 12:10:44.817903 8963 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 12:10:44.817432 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.222.62:41983 (local address 127.0.0.1:33596)
0809 12:10:44.817598 (+ 166us) negotiation.cc:107] Waiting for socket to connect
0809 12:10:44.817680 (+ 82us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.222.62:41983: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":64}
W20260809 12:10:44.817903 8369 leader_election.cc:334] T 00000000000000000000000000000000 P e2a7b439c3084cd680be92ac6d1925db [CANDIDATE]: Term 2 election: RPC error from VoteRequest() call to peer 9cd5d41083d147f788d83a438b6f7fc2 (127.0.222.62:41983): Network error: Client connection negotiation failed: client connection to 127.0.222.62:41983: connect: Connection refused (error 111)
I20260809 12:10:44.818037 8369 leader_election.cc:302] T 00000000000000000000000000000000 P e2a7b439c3084cd680be92ac6d1925db [CANDIDATE]: Term 2 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: e2a7b439c3084cd680be92ac6d1925db; no voters: 9cd5d41083d147f788d83a438b6f7fc2, cbbb52854812417da2325b27e509812d
W20260809 12:10:44.818169 8167 leader_election.cc:334] T 00000000000000000000000000000000 P cbbb52854812417da2325b27e509812d [CANDIDATE]: Term 2 election: RPC error from VoteRequest() call to peer 9cd5d41083d147f788d83a438b6f7fc2 (127.0.222.62:41983): Network error: Client connection negotiation failed: client connection to 127.0.222.62:41983: connect: Connection refused (error 111)
I20260809 12:10:44.818300 8167 leader_election.cc:302] T 00000000000000000000000000000000 P cbbb52854812417da2325b27e509812d [CANDIDATE]: Term 2 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: cbbb52854812417da2325b27e509812d; no voters: 9cd5d41083d147f788d83a438b6f7fc2, e2a7b439c3084cd680be92ac6d1925db
I20260809 12:10:44.818359 8959 raft_consensus.cc:2726] T 00000000000000000000000000000000 P e2a7b439c3084cd680be92ac6d1925db [term 2 FOLLOWER]: Leader election lost for term 2. Reason: could not achieve majority
I20260809 12:10:44.818619 8957 raft_consensus.cc:2726] T 00000000000000000000000000000000 P cbbb52854812417da2325b27e509812d [term 2 FOLLOWER]: Leader election lost for term 2. Reason: could not achieve majority
I20260809 12:10:45.265434 8969 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 12:10:45.263570 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.222.62:41983 (local address 127.0.222.2:37325)
0809 12:10:45.265086 (+ 1516us) negotiation.cc:107] Waiting for socket to connect
0809 12:10:45.265201 (+ 115us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.222.62:41983: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":1374,"thread_start_us":1269,"threads_started":1}
I20260809 12:10:45.265623 8970 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 12:10:45.263640 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.222.62:41983 (local address 127.0.222.1:56871)
0809 12:10:45.265313 (+ 1673us) negotiation.cc:107] Waiting for socket to connect
0809 12:10:45.265410 (+ 97us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.222.62:41983: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":1567,"thread_start_us":1473,"threads_started":1}
I20260809 12:10:45.271097 8971 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 12:10:45.270345 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.222.62:41983 (local address 127.0.222.3:47197)
0809 12:10:45.270770 (+ 425us) negotiation.cc:107] Waiting for socket to connect
0809 12:10:45.270862 (+ 92us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.222.62:41983: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":337,"thread_start_us":237,"threads_started":1}
I20260809 12:10:46.267928 8973 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 12:10:46.266655 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.222.62:41983 (local address 127.0.222.2:37005)
0809 12:10:46.267546 (+ 891us) negotiation.cc:107] Waiting for socket to connect
0809 12:10:46.267654 (+ 108us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.222.62:41983: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":772,"thread_start_us":671,"threads_started":1}
I20260809 12:10:46.268461 8972 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 12:10:46.266616 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.222.62:41983 (local address 127.0.222.1:54387)
0809 12:10:46.268182 (+ 1566us) negotiation.cc:107] Waiting for socket to connect
0809 12:10:46.268289 (+ 107us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.222.62:41983: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":1462,"thread_start_us":1353,"threads_started":1}
I20260809 12:10:46.272723 8974 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 12:10:46.272036 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.222.62:41983 (local address 127.0.222.3:39359)
0809 12:10:46.272483 (+ 447us) negotiation.cc:107] Waiting for socket to connect
0809 12:10:46.272560 (+ 77us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.222.62:41983: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":354,"thread_start_us":232,"threads_started":1}
I20260809 12:10:46.470089 8975 raft_consensus.cc:482] T 00000000000000000000000000000000 P e2a7b439c3084cd680be92ac6d1925db [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260809 12:10:46.470333 8975 raft_consensus.cc:504] T 00000000000000000000000000000000 P e2a7b439c3084cd680be92ac6d1925db [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "9cd5d41083d147f788d83a438b6f7fc2" member_type: VOTER last_known_addr { host: "127.0.222.62" port: 41983 } } peers { permanent_uuid: "cbbb52854812417da2325b27e509812d" member_type: VOTER last_known_addr { host: "127.0.222.61" port: 32831 } attrs { promote: false } } peers { permanent_uuid: "e2a7b439c3084cd680be92ac6d1925db" member_type: VOTER last_known_addr { host: "127.0.222.60" port: 39543 } attrs { promote: false } }
I20260809 12:10:46.471101 8975 leader_election.cc:288] T 00000000000000000000000000000000 P e2a7b439c3084cd680be92ac6d1925db [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 9cd5d41083d147f788d83a438b6f7fc2 (127.0.222.62:41983), cbbb52854812417da2325b27e509812d (127.0.222.61:32831)
I20260809 12:10:46.471801 8193 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "e2a7b439c3084cd680be92ac6d1925db" candidate_term: 3 candidate_status { last_received { term: 1 index: 12 } } ignore_live_leader: false dest_uuid: "cbbb52854812417da2325b27e509812d" is_pre_election: true
I20260809 12:10:46.472170 8193 raft_consensus.cc:2454] T 00000000000000000000000000000000 P cbbb52854812417da2325b27e509812d [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e2a7b439c3084cd680be92ac6d1925db in term 2.
I20260809 12:10:46.472658 8372 leader_election.cc:302] T 00000000000000000000000000000000 P e2a7b439c3084cd680be92ac6d1925db [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: cbbb52854812417da2325b27e509812d, e2a7b439c3084cd680be92ac6d1925db; no voters:
I20260809 12:10:46.472874 8976 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 12:10:46.471333 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.222.62:41983 (local address 127.0.0.1:33834)
0809 12:10:46.472411 (+ 1078us) negotiation.cc:107] Waiting for socket to connect
0809 12:10:46.472531 (+ 120us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.222.62:41983: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":958,"thread_start_us":793,"threads_started":1}
I20260809 12:10:46.472962 8975 raft_consensus.cc:2781] T 00000000000000000000000000000000 P e2a7b439c3084cd680be92ac6d1925db [term 2 FOLLOWER]: Leader pre-election won for term 3
I20260809 12:10:46.473177 8975 raft_consensus.cc:482] T 00000000000000000000000000000000 P e2a7b439c3084cd680be92ac6d1925db [term 2 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260809 12:10:46.473320 8975 raft_consensus.cc:3037] T 00000000000000000000000000000000 P e2a7b439c3084cd680be92ac6d1925db [term 2 FOLLOWER]: Advancing to term 3
I20260809 12:10:46.474500 8976 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 12:10:46.474063 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.222.62:41983 (local address 127.0.0.1:33836)
0809 12:10:46.474232 (+ 169us) negotiation.cc:107] Waiting for socket to connect
0809 12:10:46.474340 (+ 108us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.222.62:41983: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":62}
W20260809 12:10:46.474786 8369 leader_election.cc:334] T 00000000000000000000000000000000 P e2a7b439c3084cd680be92ac6d1925db [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer 9cd5d41083d147f788d83a438b6f7fc2 (127.0.222.62:41983): Network error: Client connection negotiation failed: client connection to 127.0.222.62:41983: connect: Connection refused (error 111)
I20260809 12:10:46.475998 8975 raft_consensus.cc:504] T 00000000000000000000000000000000 P e2a7b439c3084cd680be92ac6d1925db [term 3 FOLLOWER]: Starting leader election with config: opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "9cd5d41083d147f788d83a438b6f7fc2" member_type: VOTER last_known_addr { host: "127.0.222.62" port: 41983 } } peers { permanent_uuid: "cbbb52854812417da2325b27e509812d" member_type: VOTER last_known_addr { host: "127.0.222.61" port: 32831 } attrs { promote: false } } peers { permanent_uuid: "e2a7b439c3084cd680be92ac6d1925db" member_type: VOTER last_known_addr { host: "127.0.222.60" port: 39543 } attrs { promote: false } }
I20260809 12:10:46.476576 8975 leader_election.cc:288] T 00000000000000000000000000000000 P e2a7b439c3084cd680be92ac6d1925db [CANDIDATE]: Term 3 election: Requested vote from peers 9cd5d41083d147f788d83a438b6f7fc2 (127.0.222.62:41983), cbbb52854812417da2325b27e509812d (127.0.222.61:32831)
I20260809 12:10:46.477068 8976 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 12:10:46.476689 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.222.62:41983 (local address 127.0.0.1:33850)
0809 12:10:46.476822 (+ 133us) negotiation.cc:107] Waiting for socket to connect
0809 12:10:46.476914 (+ 92us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.222.62:41983: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":35}
I20260809 12:10:46.477139 8193 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "e2a7b439c3084cd680be92ac6d1925db" candidate_term: 3 candidate_status { last_received { term: 1 index: 12 } } ignore_live_leader: false dest_uuid: "cbbb52854812417da2325b27e509812d"
I20260809 12:10:46.477416 8193 raft_consensus.cc:3037] T 00000000000000000000000000000000 P cbbb52854812417da2325b27e509812d [term 2 FOLLOWER]: Advancing to term 3
I20260809 12:10:46.478013 8976 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 12:10:46.477694 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.222.62:41983 (local address 127.0.0.1:33854)
0809 12:10:46.477832 (+ 138us) negotiation.cc:107] Waiting for socket to connect
0809 12:10:46.477906 (+ 74us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.222.62:41983: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":47}
W20260809 12:10:46.478233 8369 leader_election.cc:334] T 00000000000000000000000000000000 P e2a7b439c3084cd680be92ac6d1925db [CANDIDATE]: Term 3 election: RPC error from VoteRequest() call to peer 9cd5d41083d147f788d83a438b6f7fc2 (127.0.222.62:41983): Network error: Client connection negotiation failed: client connection to 127.0.222.62:41983: connect: Connection refused (error 111)
I20260809 12:10:46.479825 8193 raft_consensus.cc:2454] T 00000000000000000000000000000000 P cbbb52854812417da2325b27e509812d [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e2a7b439c3084cd680be92ac6d1925db in term 3.
I20260809 12:10:46.480266 8372 leader_election.cc:302] T 00000000000000000000000000000000 P e2a7b439c3084cd680be92ac6d1925db [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: cbbb52854812417da2325b27e509812d, e2a7b439c3084cd680be92ac6d1925db; no voters: 9cd5d41083d147f788d83a438b6f7fc2
I20260809 12:10:46.480679 8975 raft_consensus.cc:2781] T 00000000000000000000000000000000 P e2a7b439c3084cd680be92ac6d1925db [term 3 FOLLOWER]: Leader election won for term 3
I20260809 12:10:46.481369 8975 raft_consensus.cc:686] T 00000000000000000000000000000000 P e2a7b439c3084cd680be92ac6d1925db [term 3 LEADER]: Becoming Leader. State: Replica: e2a7b439c3084cd680be92ac6d1925db, State: Running, Role: LEADER
I20260809 12:10:46.481844 8975 consensus_queue.cc:237] T 00000000000000000000000000000000 P e2a7b439c3084cd680be92ac6d1925db [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 12, Committed index: 12, Last appended: 1.12, Last appended by leader: 12, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "9cd5d41083d147f788d83a438b6f7fc2" member_type: VOTER last_known_addr { host: "127.0.222.62" port: 41983 } } peers { permanent_uuid: "cbbb52854812417da2325b27e509812d" member_type: VOTER last_known_addr { host: "127.0.222.61" port: 32831 } attrs { promote: false } } peers { permanent_uuid: "e2a7b439c3084cd680be92ac6d1925db" member_type: VOTER last_known_addr { host: "127.0.222.60" port: 39543 } attrs { promote: false } }
I20260809 12:10:46.484179 8978 sys_catalog.cc:455] T 00000000000000000000000000000000 P e2a7b439c3084cd680be92ac6d1925db [sys.catalog]: SysCatalogTable state changed. Reason: New leader e2a7b439c3084cd680be92ac6d1925db. Latest consensus state: current_term: 3 leader_uuid: "e2a7b439c3084cd680be92ac6d1925db" committed_config { opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "9cd5d41083d147f788d83a438b6f7fc2" member_type: VOTER last_known_addr { host: "127.0.222.62" port: 41983 } } peers { permanent_uuid: "cbbb52854812417da2325b27e509812d" member_type: VOTER last_known_addr { host: "127.0.222.61" port: 32831 } attrs { promote: false } } peers { permanent_uuid: "e2a7b439c3084cd680be92ac6d1925db" member_type: VOTER last_known_addr { host: "127.0.222.60" port: 39543 } attrs { promote: false } } }
I20260809 12:10:46.484416 8978 sys_catalog.cc:458] T 00000000000000000000000000000000 P e2a7b439c3084cd680be92ac6d1925db [sys.catalog]: This master's current role is: LEADER
I20260809 12:10:46.485086 8980 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260809 12:10:46.489113 8980 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260809 12:10:46.489976 8980 catalog_manager.cc:1259] Loaded cluster ID: 282dd341a1ea4ea895575dd472644e9f
I20260809 12:10:46.490082 8980 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260809 12:10:46.491351 8980 catalog_manager.cc:1504] Loading token signing keys...
I20260809 12:10:46.492197 8980 catalog_manager.cc:5879] T 00000000000000000000000000000000 P e2a7b439c3084cd680be92ac6d1925db: Loaded TSK: 0
I20260809 12:10:46.493116 8980 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260809 12:10:46.567863 8384 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:42488:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260809 12:10:46.574934 8384 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260809 12:10:46.578923 8976 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 12:10:46.578483 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.222.62:41983 (local address 127.0.0.1:33884)
0809 12:10:46.578694 (+ 211us) negotiation.cc:107] Waiting for socket to connect
0809 12:10:46.578781 (+ 87us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.222.62:41983: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":101}
I20260809 12:10:46.579121 8193 raft_consensus.cc:1261] T 00000000000000000000000000000000 P cbbb52854812417da2325b27e509812d [term 3 FOLLOWER]: Refusing update from remote peer e2a7b439c3084cd680be92ac6d1925db: Log matching property violated. Preceding OpId in replica: term: 1 index: 12. Preceding OpId from leader: term: 3 index: 14. (index mismatch)
I20260809 12:10:46.579980 8976 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 12:10:46.579621 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.222.62:41983 (local address 127.0.0.1:33894)
0809 12:10:46.579808 (+ 187us) negotiation.cc:107] Waiting for socket to connect
0809 12:10:46.579872 (+ 64us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.222.62:41983: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":49}
I20260809 12:10:46.579918 8978 consensus_queue.cc:1035] T 00000000000000000000000000000000 P e2a7b439c3084cd680be92ac6d1925db [LEADER]: Connected to new peer: Peer: permanent_uuid: "cbbb52854812417da2325b27e509812d" member_type: VOTER last_known_addr { host: "127.0.222.61" port: 32831 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 13, Last known committed idx: 12, Time since last communication: 0.000s
W20260809 12:10:46.580255 8369 consensus_peers.cc:487] T 00000000000000000000000000000000 P e2a7b439c3084cd680be92ac6d1925db -> Peer 9cd5d41083d147f788d83a438b6f7fc2 (127.0.222.62:41983): Couldn't send request to peer 9cd5d41083d147f788d83a438b6f7fc2. Status: Network error: Client connection negotiation failed: client connection to 127.0.222.62:41983: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20260809 12:10:46.583083 8982 sys_catalog.cc:455] T 00000000000000000000000000000000 P cbbb52854812417da2325b27e509812d [sys.catalog]: SysCatalogTable state changed. Reason: New leader e2a7b439c3084cd680be92ac6d1925db. Latest consensus state: current_term: 3 leader_uuid: "e2a7b439c3084cd680be92ac6d1925db" committed_config { opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "9cd5d41083d147f788d83a438b6f7fc2" member_type: VOTER last_known_addr { host: "127.0.222.62" port: 41983 } } peers { permanent_uuid: "cbbb52854812417da2325b27e509812d" member_type: VOTER last_known_addr { host: "127.0.222.61" port: 32831 } attrs { promote: false } } peers { permanent_uuid: "e2a7b439c3084cd680be92ac6d1925db" member_type: VOTER last_known_addr { host: "127.0.222.60" port: 39543 } attrs { promote: false } } }
I20260809 12:10:46.583341 8982 sys_catalog.cc:458] T 00000000000000000000000000000000 P cbbb52854812417da2325b27e509812d [sys.catalog]: This master's current role is: FOLLOWER
I20260809 12:10:46.585439 8982 sys_catalog.cc:455] T 00000000000000000000000000000000 P cbbb52854812417da2325b27e509812d [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 3 leader_uuid: "e2a7b439c3084cd680be92ac6d1925db" committed_config { opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "9cd5d41083d147f788d83a438b6f7fc2" member_type: VOTER last_known_addr { host: "127.0.222.62" port: 41983 } } peers { permanent_uuid: "cbbb52854812417da2325b27e509812d" member_type: VOTER last_known_addr { host: "127.0.222.61" port: 32831 } attrs { promote: false } } peers { permanent_uuid: "e2a7b439c3084cd680be92ac6d1925db" member_type: VOTER last_known_addr { host: "127.0.222.60" port: 39543 } attrs { promote: false } } }
I20260809 12:10:46.585826 8982 sys_catalog.cc:458] T 00000000000000000000000000000000 P cbbb52854812417da2325b27e509812d [sys.catalog]: This master's current role is: FOLLOWER
I20260809 12:10:46.592371 8975 sys_catalog.cc:455] T 00000000000000000000000000000000 P e2a7b439c3084cd680be92ac6d1925db [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 3 leader_uuid: "e2a7b439c3084cd680be92ac6d1925db" committed_config { opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "9cd5d41083d147f788d83a438b6f7fc2" member_type: VOTER last_known_addr { host: "127.0.222.62" port: 41983 } } peers { permanent_uuid: "cbbb52854812417da2325b27e509812d" member_type: VOTER last_known_addr { host: "127.0.222.61" port: 32831 } attrs { promote: false } } peers { permanent_uuid: "e2a7b439c3084cd680be92ac6d1925db" member_type: VOTER last_known_addr { host: "127.0.222.60" port: 39543 } attrs { promote: false } } }
I20260809 12:10:46.592880 8975 sys_catalog.cc:458] T 00000000000000000000000000000000 P e2a7b439c3084cd680be92ac6d1925db [sys.catalog]: This master's current role is: LEADER
I20260809 12:10:46.627170 8992 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 12:10:46.600149 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.222.2:33587 (local address 127.0.0.1:35432)
0809 12:10:46.612331 (+ 12182us) negotiation.cc:107] Waiting for socket to connect
0809 12:10:46.612367 (+ 36us) client_negotiation.cc:174] Beginning negotiation
0809 12:10:46.612793 (+ 426us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 12:10:46.620371 (+ 7578us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 12:10:46.620376 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 12:10:46.620406 (+ 30us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0809 12:10:46.620891 (+ 485us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 12:10:46.620906 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 12:10:46.622335 (+ 1429us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 12:10:46.622343 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 12:10:46.625229 (+ 2886us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 12:10:46.625246 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 12:10:46.626460 (+ 1214us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 12:10:46.626464 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 12:10:46.626579 (+ 115us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 12:10:46.626604 (+ 25us) client_negotiation.cc:769] Sending connection context
0809 12:10:46.626677 (+ 73us) client_negotiation.cc:240] Negotiation successful
0809 12:10:46.626773 (+ 96us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":12004,"mutex_wait_us":109,"thread_start_us":11832,"threads_started":1}
I20260809 12:10:46.627269 8996 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 12:10:46.601210 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35432 (local address 127.0.222.2:33587)
0809 12:10:46.619436 (+ 18226us) server_negotiation.cc:206] Beginning negotiation
0809 12:10:46.619441 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0809 12:10:46.619462 (+ 21us) server_negotiation.cc:407] Connection header received
0809 12:10:46.619604 (+ 142us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 12:10:46.619612 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 12:10:46.619703 (+ 91us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 12:10:46.620081 (+ 378us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0809 12:10:46.621210 (+ 1129us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 12:10:46.622176 (+ 966us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 12:10:46.625432 (+ 3256us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 12:10:46.626295 (+ 863us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 12:10:46.626434 (+ 139us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 12:10:46.626558 (+ 124us) server_negotiation.cc:1035] Waiting for connection context
0809 12:10:46.626732 (+ 174us) server_negotiation.cc:299] Negotiation successful
0809 12:10:46.626897 (+ 165us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":18103,"thread_start_us":17972,"threads_started":1}
I20260809 12:10:46.628328 8656 tablet_service.cc:1467] Processing CreateTablet for tablet ae5ac5988e024e7689a379e9c5a70c21 (DEFAULT_TABLE table=test-table [id=3a58cbf1ca2446f38efb53e00bb5f763]), partition=RANGE (key) PARTITION UNBOUNDED
I20260809 12:10:46.628703 8995 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 12:10:46.600642 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.222.3:37357 (local address 127.0.0.1:33598)
0809 12:10:46.620548 (+ 19906us) negotiation.cc:107] Waiting for socket to connect
0809 12:10:46.620574 (+ 26us) client_negotiation.cc:174] Beginning negotiation
0809 12:10:46.620764 (+ 190us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 12:10:46.621649 (+ 885us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 12:10:46.621655 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 12:10:46.621705 (+ 50us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0809 12:10:46.622233 (+ 528us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 12:10:46.622247 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 12:10:46.623736 (+ 1489us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 12:10:46.623740 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 12:10:46.626190 (+ 2450us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 12:10:46.626199 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 12:10:46.627857 (+ 1658us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 12:10:46.627861 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 12:10:46.628043 (+ 182us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 12:10:46.628075 (+ 32us) client_negotiation.cc:769] Sending connection context
0809 12:10:46.628177 (+ 102us) client_negotiation.cc:240] Negotiation successful
0809 12:10:46.628293 (+ 116us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":19716,"mutex_wait_us":175,"spinlock_wait_cycles":7680,"thread_start_us":19537,"threads_started":1}
I20260809 12:10:46.628733 8994 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 12:10:46.600727 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33598 (local address 127.0.222.3:37357)
0809 12:10:46.612713 (+ 11986us) server_negotiation.cc:206] Beginning negotiation
0809 12:10:46.612718 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0809 12:10:46.621110 (+ 8392us) server_negotiation.cc:407] Connection header received
0809 12:10:46.621202 (+ 92us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 12:10:46.621207 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 12:10:46.621291 (+ 84us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 12:10:46.621401 (+ 110us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0809 12:10:46.622455 (+ 1054us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 12:10:46.623598 (+ 1143us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 12:10:46.626360 (+ 2762us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 12:10:46.627661 (+ 1301us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 12:10:46.627807 (+ 146us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 12:10:46.627932 (+ 125us) server_negotiation.cc:1035] Waiting for connection context
0809 12:10:46.628219 (+ 287us) server_negotiation.cc:299] Negotiation successful
0809 12:10:46.628369 (+ 150us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":11855,"thread_start_us":11702,"threads_started":1}
I20260809 12:10:46.628875 8656 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ae5ac5988e024e7689a379e9c5a70c21. 1 dirs total, 0 dirs full, 0 dirs failed
I20260809 12:10:46.630604 8991 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 12:10:46.599897 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49260 (local address 127.0.222.1:43733)
0809 12:10:46.621271 (+ 21374us) server_negotiation.cc:206] Beginning negotiation
0809 12:10:46.621278 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0809 12:10:46.621296 (+ 18us) server_negotiation.cc:407] Connection header received
0809 12:10:46.621384 (+ 88us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 12:10:46.621389 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 12:10:46.621476 (+ 87us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 12:10:46.622292 (+ 816us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0809 12:10:46.622590 (+ 298us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 12:10:46.623424 (+ 834us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 12:10:46.628040 (+ 4616us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 12:10:46.629700 (+ 1660us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 12:10:46.629830 (+ 130us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 12:10:46.629916 (+ 86us) server_negotiation.cc:1035] Waiting for connection context
0809 12:10:46.630240 (+ 324us) server_negotiation.cc:299] Negotiation successful
0809 12:10:46.630343 (+ 103us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":21232,"thread_start_us":21094,"threads_started":1}
I20260809 12:10:46.630658 8976 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 12:10:46.599680 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.222.1:43733 (local address 127.0.0.1:49260)
0809 12:10:46.599844 (+ 164us) negotiation.cc:107] Waiting for socket to connect
0809 12:10:46.599866 (+ 22us) client_negotiation.cc:174] Beginning negotiation
0809 12:10:46.600257 (+ 391us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 12:10:46.621650 (+ 21393us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 12:10:46.621655 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 12:10:46.621707 (+ 52us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0809 12:10:46.622079 (+ 372us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 12:10:46.622096 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 12:10:46.623581 (+ 1485us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 12:10:46.623587 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 12:10:46.626177 (+ 2590us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 12:10:46.626193 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 12:10:46.629907 (+ 3714us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 12:10:46.629913 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 12:10:46.630083 (+ 170us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 12:10:46.630116 (+ 33us) client_negotiation.cc:769] Sending connection context
0809 12:10:46.630210 (+ 94us) client_negotiation.cc:240] Negotiation successful
0809 12:10:46.630354 (+ 144us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":32,"mutex_wait_us":136,"spinlock_wait_cycles":10624}
I20260809 12:10:46.631562 8505 tablet_service.cc:1467] Processing CreateTablet for tablet ae5ac5988e024e7689a379e9c5a70c21 (DEFAULT_TABLE table=test-table [id=3a58cbf1ca2446f38efb53e00bb5f763]), partition=RANGE (key) PARTITION UNBOUNDED
I20260809 12:10:46.632001 8505 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ae5ac5988e024e7689a379e9c5a70c21. 1 dirs total, 0 dirs full, 0 dirs failed
I20260809 12:10:46.634928 8808 tablet_service.cc:1467] Processing CreateTablet for tablet ae5ac5988e024e7689a379e9c5a70c21 (DEFAULT_TABLE table=test-table [id=3a58cbf1ca2446f38efb53e00bb5f763]), partition=RANGE (key) PARTITION UNBOUNDED
I20260809 12:10:46.635426 8808 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ae5ac5988e024e7689a379e9c5a70c21. 1 dirs total, 0 dirs full, 0 dirs failed
I20260809 12:10:46.636304 8997 tablet_bootstrap.cc:492] T ae5ac5988e024e7689a379e9c5a70c21 P 94413a87e0a14ef195baa2a799a9e0a8: Bootstrap starting.
I20260809 12:10:46.638285 8997 tablet_bootstrap.cc:654] T ae5ac5988e024e7689a379e9c5a70c21 P 94413a87e0a14ef195baa2a799a9e0a8: Neither blocks nor log segments found. Creating new log.
I20260809 12:10:46.638458 8998 tablet_bootstrap.cc:492] T ae5ac5988e024e7689a379e9c5a70c21 P df0db1a138554d00956d87f5c172ee10: Bootstrap starting.
I20260809 12:10:46.640352 8998 tablet_bootstrap.cc:654] T ae5ac5988e024e7689a379e9c5a70c21 P df0db1a138554d00956d87f5c172ee10: Neither blocks nor log segments found. Creating new log.
I20260809 12:10:46.641537 8997 tablet_bootstrap.cc:492] T ae5ac5988e024e7689a379e9c5a70c21 P 94413a87e0a14ef195baa2a799a9e0a8: No bootstrap required, opened a new log
I20260809 12:10:46.641687 8997 ts_tablet_manager.cc:1397] T ae5ac5988e024e7689a379e9c5a70c21 P 94413a87e0a14ef195baa2a799a9e0a8: Time spent bootstrapping tablet: real 0.005s user 0.002s sys 0.002s
I20260809 12:10:46.642575 9002 tablet_bootstrap.cc:492] T ae5ac5988e024e7689a379e9c5a70c21 P c40926a9373242b0961349a2719c22ab: Bootstrap starting.
I20260809 12:10:46.642565 8997 raft_consensus.cc:348] T ae5ac5988e024e7689a379e9c5a70c21 P 94413a87e0a14ef195baa2a799a9e0a8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df0db1a138554d00956d87f5c172ee10" member_type: VOTER last_known_addr { host: "127.0.222.1" port: 43733 } } peers { permanent_uuid: "94413a87e0a14ef195baa2a799a9e0a8" member_type: VOTER last_known_addr { host: "127.0.222.2" port: 33587 } } peers { permanent_uuid: "c40926a9373242b0961349a2719c22ab" member_type: VOTER last_known_addr { host: "127.0.222.3" port: 37357 } }
I20260809 12:10:46.642731 8998 tablet_bootstrap.cc:492] T ae5ac5988e024e7689a379e9c5a70c21 P df0db1a138554d00956d87f5c172ee10: No bootstrap required, opened a new log
I20260809 12:10:46.642880 8998 ts_tablet_manager.cc:1397] T ae5ac5988e024e7689a379e9c5a70c21 P df0db1a138554d00956d87f5c172ee10: Time spent bootstrapping tablet: real 0.005s user 0.003s sys 0.000s
I20260809 12:10:46.642885 8997 raft_consensus.cc:374] T ae5ac5988e024e7689a379e9c5a70c21 P 94413a87e0a14ef195baa2a799a9e0a8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260809 12:10:46.643034 8997 raft_consensus.cc:729] T ae5ac5988e024e7689a379e9c5a70c21 P 94413a87e0a14ef195baa2a799a9e0a8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 94413a87e0a14ef195baa2a799a9e0a8, State: Initialized, Role: FOLLOWER
I20260809 12:10:46.643335 8997 consensus_queue.cc:260] T ae5ac5988e024e7689a379e9c5a70c21 P 94413a87e0a14ef195baa2a799a9e0a8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df0db1a138554d00956d87f5c172ee10" member_type: VOTER last_known_addr { host: "127.0.222.1" port: 43733 } } peers { permanent_uuid: "94413a87e0a14ef195baa2a799a9e0a8" member_type: VOTER last_known_addr { host: "127.0.222.2" port: 33587 } } peers { permanent_uuid: "c40926a9373242b0961349a2719c22ab" member_type: VOTER last_known_addr { host: "127.0.222.3" port: 37357 } }
I20260809 12:10:46.643716 8998 raft_consensus.cc:348] T ae5ac5988e024e7689a379e9c5a70c21 P df0db1a138554d00956d87f5c172ee10 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df0db1a138554d00956d87f5c172ee10" member_type: VOTER last_known_addr { host: "127.0.222.1" port: 43733 } } peers { permanent_uuid: "94413a87e0a14ef195baa2a799a9e0a8" member_type: VOTER last_known_addr { host: "127.0.222.2" port: 33587 } } peers { permanent_uuid: "c40926a9373242b0961349a2719c22ab" member_type: VOTER last_known_addr { host: "127.0.222.3" port: 37357 } }
I20260809 12:10:46.644049 8998 raft_consensus.cc:374] T ae5ac5988e024e7689a379e9c5a70c21 P df0db1a138554d00956d87f5c172ee10 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260809 12:10:46.644204 8998 raft_consensus.cc:729] T ae5ac5988e024e7689a379e9c5a70c21 P df0db1a138554d00956d87f5c172ee10 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: df0db1a138554d00956d87f5c172ee10, State: Initialized, Role: FOLLOWER
I20260809 12:10:46.644505 8998 consensus_queue.cc:260] T ae5ac5988e024e7689a379e9c5a70c21 P df0db1a138554d00956d87f5c172ee10 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df0db1a138554d00956d87f5c172ee10" member_type: VOTER last_known_addr { host: "127.0.222.1" port: 43733 } } peers { permanent_uuid: "94413a87e0a14ef195baa2a799a9e0a8" member_type: VOTER last_known_addr { host: "127.0.222.2" port: 33587 } } peers { permanent_uuid: "c40926a9373242b0961349a2719c22ab" member_type: VOTER last_known_addr { host: "127.0.222.3" port: 37357 } }
I20260809 12:10:46.645032 9002 tablet_bootstrap.cc:654] T ae5ac5988e024e7689a379e9c5a70c21 P c40926a9373242b0961349a2719c22ab: Neither blocks nor log segments found. Creating new log.
I20260809 12:10:46.645236 8973 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 12:10:46.644743 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.222.62:41983 (local address 127.0.222.2:36827)
0809 12:10:46.644964 (+ 221us) negotiation.cc:107] Waiting for socket to connect
0809 12:10:46.645082 (+ 118us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.222.62:41983: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":117}
I20260809 12:10:46.647053 8998 ts_tablet_manager.cc:1428] T ae5ac5988e024e7689a379e9c5a70c21 P df0db1a138554d00956d87f5c172ee10: Time spent starting tablet: real 0.004s user 0.004s sys 0.000s
I20260809 12:10:46.648526 8972 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 12:10:46.647898 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.222.62:41983 (local address 127.0.222.1:54603)
0809 12:10:46.648260 (+ 362us) negotiation.cc:107] Waiting for socket to connect
0809 12:10:46.648366 (+ 106us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.222.62:41983: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":241}
I20260809 12:10:46.648759 8997 ts_tablet_manager.cc:1428] T ae5ac5988e024e7689a379e9c5a70c21 P 94413a87e0a14ef195baa2a799a9e0a8: Time spent starting tablet: real 0.007s user 0.001s sys 0.002s
I20260809 12:10:46.648808 9002 tablet_bootstrap.cc:492] T ae5ac5988e024e7689a379e9c5a70c21 P c40926a9373242b0961349a2719c22ab: No bootstrap required, opened a new log
I20260809 12:10:46.648949 9002 ts_tablet_manager.cc:1397] T ae5ac5988e024e7689a379e9c5a70c21 P c40926a9373242b0961349a2719c22ab: Time spent bootstrapping tablet: real 0.006s user 0.004s sys 0.000s
I20260809 12:10:46.648973 8726 heartbeater.cc:503] Master 127.0.222.60:39543 was elected leader, sending a full tablet report...
I20260809 12:10:46.649794 9002 raft_consensus.cc:348] T ae5ac5988e024e7689a379e9c5a70c21 P c40926a9373242b0961349a2719c22ab [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df0db1a138554d00956d87f5c172ee10" member_type: VOTER last_known_addr { host: "127.0.222.1" port: 43733 } } peers { permanent_uuid: "94413a87e0a14ef195baa2a799a9e0a8" member_type: VOTER last_known_addr { host: "127.0.222.2" port: 33587 } } peers { permanent_uuid: "c40926a9373242b0961349a2719c22ab" member_type: VOTER last_known_addr { host: "127.0.222.3" port: 37357 } }
I20260809 12:10:46.650028 9002 raft_consensus.cc:374] T ae5ac5988e024e7689a379e9c5a70c21 P c40926a9373242b0961349a2719c22ab [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260809 12:10:46.650138 9002 raft_consensus.cc:729] T ae5ac5988e024e7689a379e9c5a70c21 P c40926a9373242b0961349a2719c22ab [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c40926a9373242b0961349a2719c22ab, State: Initialized, Role: FOLLOWER
I20260809 12:10:46.650578 9002 consensus_queue.cc:260] T ae5ac5988e024e7689a379e9c5a70c21 P c40926a9373242b0961349a2719c22ab [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df0db1a138554d00956d87f5c172ee10" member_type: VOTER last_known_addr { host: "127.0.222.1" port: 43733 } } peers { permanent_uuid: "94413a87e0a14ef195baa2a799a9e0a8" member_type: VOTER last_known_addr { host: "127.0.222.2" port: 33587 } } peers { permanent_uuid: "c40926a9373242b0961349a2719c22ab" member_type: VOTER last_known_addr { host: "127.0.222.3" port: 37357 } }
I20260809 12:10:46.651923 8572 heartbeater.cc:503] Master 127.0.222.60:39543 was elected leader, sending a full tablet report...
I20260809 12:10:46.652336 8973 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 12:10:46.651820 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.222.62:41983 (local address 127.0.222.2:53869)
0809 12:10:46.652049 (+ 229us) negotiation.cc:107] Waiting for socket to connect
0809 12:10:46.652162 (+ 113us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.222.62:41983: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":111}
I20260809 12:10:46.653333 9002 ts_tablet_manager.cc:1428] T ae5ac5988e024e7689a379e9c5a70c21 P c40926a9373242b0961349a2719c22ab: Time spent starting tablet: real 0.004s user 0.002s sys 0.002s
I20260809 12:10:46.654135 8974 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 12:10:46.653637 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.222.62:41983 (local address 127.0.222.3:58689)
0809 12:10:46.653889 (+ 252us) negotiation.cc:107] Waiting for socket to connect
0809 12:10:46.653984 (+ 95us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.222.62:41983: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":125}
I20260809 12:10:46.655197 8972 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 12:10:46.654626 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.222.62:41983 (local address 127.0.222.1:39179)
0809 12:10:46.654909 (+ 283us) negotiation.cc:107] Waiting for socket to connect
0809 12:10:46.655024 (+ 115us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.222.62:41983: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":178}
I20260809 12:10:46.655361 8878 heartbeater.cc:503] Master 127.0.222.60:39543 was elected leader, sending a full tablet report...
I20260809 12:10:46.656082 8974 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 12:10:46.655659 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.222.62:41983 (local address 127.0.222.3:42699)
0809 12:10:46.655821 (+ 162us) negotiation.cc:107] Waiting for socket to connect
0809 12:10:46.655930 (+ 109us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.222.62:41983: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":50}
W20260809 12:10:46.831625 8575 tablet.cc:2369] T ae5ac5988e024e7689a379e9c5a70c21 P df0db1a138554d00956d87f5c172ee10: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260809 12:10:46.882745 9004 raft_consensus.cc:482] T ae5ac5988e024e7689a379e9c5a70c21 P df0db1a138554d00956d87f5c172ee10 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260809 12:10:46.883080 9004 raft_consensus.cc:504] T ae5ac5988e024e7689a379e9c5a70c21 P df0db1a138554d00956d87f5c172ee10 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df0db1a138554d00956d87f5c172ee10" member_type: VOTER last_known_addr { host: "127.0.222.1" port: 43733 } } peers { permanent_uuid: "94413a87e0a14ef195baa2a799a9e0a8" member_type: VOTER last_known_addr { host: "127.0.222.2" port: 33587 } } peers { permanent_uuid: "c40926a9373242b0961349a2719c22ab" member_type: VOTER last_known_addr { host: "127.0.222.3" port: 37357 } }
I20260809 12:10:46.884912 9004 leader_election.cc:288] T ae5ac5988e024e7689a379e9c5a70c21 P df0db1a138554d00956d87f5c172ee10 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 94413a87e0a14ef195baa2a799a9e0a8 (127.0.222.2:33587), c40926a9373242b0961349a2719c22ab (127.0.222.3:37357)
I20260809 12:10:46.893270 8994 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 12:10:46.885448 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.222.1:54227 (local address 127.0.222.3:37357)
0809 12:10:46.885683 (+ 235us) server_negotiation.cc:206] Beginning negotiation
0809 12:10:46.885692 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0809 12:10:46.886185 (+ 493us) server_negotiation.cc:407] Connection header received
0809 12:10:46.886346 (+ 161us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 12:10:46.886350 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 12:10:46.886415 (+ 65us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 12:10:46.886487 (+ 72us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0809 12:10:46.887101 (+ 614us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 12:10:46.888095 (+ 994us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 12:10:46.890661 (+ 2566us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 12:10:46.891978 (+ 1317us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 12:10:46.892112 (+ 134us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 12:10:46.892216 (+ 104us) server_negotiation.cc:1035] Waiting for connection context
0809 12:10:46.892861 (+ 645us) server_negotiation.cc:299] Negotiation successful
0809 12:10:46.892995 (+ 134us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":73}
I20260809 12:10:46.893254 8996 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 12:10:46.885273 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.222.1:53431 (local address 127.0.222.2:33587)
0809 12:10:46.885512 (+ 239us) server_negotiation.cc:206] Beginning negotiation
0809 12:10:46.885519 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0809 12:10:46.885608 (+ 89us) server_negotiation.cc:407] Connection header received
0809 12:10:46.885912 (+ 304us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 12:10:46.885918 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 12:10:46.886019 (+ 101us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 12:10:46.886101 (+ 82us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0809 12:10:46.886742 (+ 641us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 12:10:46.887857 (+ 1115us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 12:10:46.890661 (+ 2804us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 12:10:46.891958 (+ 1297us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 12:10:46.892112 (+ 154us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 12:10:46.892217 (+ 105us) server_negotiation.cc:1035] Waiting for connection context
0809 12:10:46.892833 (+ 616us) server_negotiation.cc:299] Negotiation successful
0809 12:10:46.892979 (+ 146us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":75}
I20260809 12:10:46.893353 8972 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 12:10:46.885190 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.222.2:33587 (local address 127.0.222.1:53431)
0809 12:10:46.885497 (+ 307us) negotiation.cc:107] Waiting for socket to connect
0809 12:10:46.885526 (+ 29us) client_negotiation.cc:174] Beginning negotiation
0809 12:10:46.885699 (+ 173us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 12:10:46.886166 (+ 467us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 12:10:46.886170 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 12:10:46.886219 (+ 49us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0809 12:10:46.886579 (+ 360us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 12:10:46.886589 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 12:10:46.888001 (+ 1412us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 12:10:46.888004 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 12:10:46.890458 (+ 2454us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 12:10:46.890475 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 12:10:46.892248 (+ 1773us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 12:10:46.892261 (+ 13us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 12:10:46.892561 (+ 300us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 12:10:46.892668 (+ 107us) client_negotiation.cc:769] Sending connection context
0809 12:10:46.892771 (+ 103us) client_negotiation.cc:240] Negotiation successful
0809 12:10:46.892928 (+ 157us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":168,"mutex_wait_us":165,"spinlock_wait_cycles":12544}
I20260809 12:10:46.893430 9007 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 12:10:46.885241 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.222.3:37357 (local address 127.0.222.1:54227)
0809 12:10:46.886040 (+ 799us) negotiation.cc:107] Waiting for socket to connect
0809 12:10:46.886055 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0809 12:10:46.886214 (+ 159us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 12:10:46.886556 (+ 342us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 12:10:46.886561 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 12:10:46.886585 (+ 24us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0809 12:10:46.886896 (+ 311us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 12:10:46.886904 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 12:10:46.888220 (+ 1316us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 12:10:46.888225 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 12:10:46.890458 (+ 2233us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 12:10:46.890475 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 12:10:46.892279 (+ 1804us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 12:10:46.892283 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 12:10:46.892562 (+ 279us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 12:10:46.892668 (+ 106us) client_negotiation.cc:769] Sending connection context
0809 12:10:46.892771 (+ 103us) client_negotiation.cc:240] Negotiation successful
0809 12:10:46.892929 (+ 158us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":700,"mutex_wait_us":154,"spinlock_wait_cycles":1408,"thread_start_us":499,"threads_started":1}
I20260809 12:10:46.894256 8677 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ae5ac5988e024e7689a379e9c5a70c21" candidate_uuid: "df0db1a138554d00956d87f5c172ee10" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "94413a87e0a14ef195baa2a799a9e0a8" is_pre_election: true
I20260809 12:10:46.894639 8677 raft_consensus.cc:2454] T ae5ac5988e024e7689a379e9c5a70c21 P 94413a87e0a14ef195baa2a799a9e0a8 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate df0db1a138554d00956d87f5c172ee10 in term 0.
I20260809 12:10:46.894501 8829 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ae5ac5988e024e7689a379e9c5a70c21" candidate_uuid: "df0db1a138554d00956d87f5c172ee10" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c40926a9373242b0961349a2719c22ab" is_pre_election: true
I20260809 12:10:46.895054 8829 raft_consensus.cc:2454] T ae5ac5988e024e7689a379e9c5a70c21 P c40926a9373242b0961349a2719c22ab [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate df0db1a138554d00956d87f5c172ee10 in term 0.
I20260809 12:10:46.895294 8452 leader_election.cc:302] T ae5ac5988e024e7689a379e9c5a70c21 P df0db1a138554d00956d87f5c172ee10 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 94413a87e0a14ef195baa2a799a9e0a8, df0db1a138554d00956d87f5c172ee10; no voters:
I20260809 12:10:46.895705 9004 raft_consensus.cc:2781] T ae5ac5988e024e7689a379e9c5a70c21 P df0db1a138554d00956d87f5c172ee10 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260809 12:10:46.895860 9004 raft_consensus.cc:482] T ae5ac5988e024e7689a379e9c5a70c21 P df0db1a138554d00956d87f5c172ee10 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260809 12:10:46.895974 9004 raft_consensus.cc:3037] T ae5ac5988e024e7689a379e9c5a70c21 P df0db1a138554d00956d87f5c172ee10 [term 0 FOLLOWER]: Advancing to term 1
I20260809 12:10:46.898214 9004 raft_consensus.cc:504] T ae5ac5988e024e7689a379e9c5a70c21 P df0db1a138554d00956d87f5c172ee10 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df0db1a138554d00956d87f5c172ee10" member_type: VOTER last_known_addr { host: "127.0.222.1" port: 43733 } } peers { permanent_uuid: "94413a87e0a14ef195baa2a799a9e0a8" member_type: VOTER last_known_addr { host: "127.0.222.2" port: 33587 } } peers { permanent_uuid: "c40926a9373242b0961349a2719c22ab" member_type: VOTER last_known_addr { host: "127.0.222.3" port: 37357 } }
I20260809 12:10:46.898856 9004 leader_election.cc:288] T ae5ac5988e024e7689a379e9c5a70c21 P df0db1a138554d00956d87f5c172ee10 [CANDIDATE]: Term 1 election: Requested vote from peers 94413a87e0a14ef195baa2a799a9e0a8 (127.0.222.2:33587), c40926a9373242b0961349a2719c22ab (127.0.222.3:37357)
I20260809 12:10:46.899399 8677 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ae5ac5988e024e7689a379e9c5a70c21" candidate_uuid: "df0db1a138554d00956d87f5c172ee10" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "94413a87e0a14ef195baa2a799a9e0a8"
I20260809 12:10:46.899497 8829 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ae5ac5988e024e7689a379e9c5a70c21" candidate_uuid: "df0db1a138554d00956d87f5c172ee10" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c40926a9373242b0961349a2719c22ab"
I20260809 12:10:46.899623 8677 raft_consensus.cc:3037] T ae5ac5988e024e7689a379e9c5a70c21 P 94413a87e0a14ef195baa2a799a9e0a8 [term 0 FOLLOWER]: Advancing to term 1
I20260809 12:10:46.899724 8829 raft_consensus.cc:3037] T ae5ac5988e024e7689a379e9c5a70c21 P c40926a9373242b0961349a2719c22ab [term 0 FOLLOWER]: Advancing to term 1
I20260809 12:10:46.901854 8677 raft_consensus.cc:2454] T ae5ac5988e024e7689a379e9c5a70c21 P 94413a87e0a14ef195baa2a799a9e0a8 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate df0db1a138554d00956d87f5c172ee10 in term 1.
I20260809 12:10:46.901950 8829 raft_consensus.cc:2454] T ae5ac5988e024e7689a379e9c5a70c21 P c40926a9373242b0961349a2719c22ab [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate df0db1a138554d00956d87f5c172ee10 in term 1.
I20260809 12:10:46.902470 8455 leader_election.cc:302] T ae5ac5988e024e7689a379e9c5a70c21 P df0db1a138554d00956d87f5c172ee10 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 3 yes votes; 0 no votes. yes voters: 94413a87e0a14ef195baa2a799a9e0a8, c40926a9373242b0961349a2719c22ab, df0db1a138554d00956d87f5c172ee10; no voters:
I20260809 12:10:46.902884 9004 raft_consensus.cc:2781] T ae5ac5988e024e7689a379e9c5a70c21 P df0db1a138554d00956d87f5c172ee10 [term 1 FOLLOWER]: Leader election won for term 1
I20260809 12:10:46.903481 9004 raft_consensus.cc:686] T ae5ac5988e024e7689a379e9c5a70c21 P df0db1a138554d00956d87f5c172ee10 [term 1 LEADER]: Becoming Leader. State: Replica: df0db1a138554d00956d87f5c172ee10, State: Running, Role: LEADER
I20260809 12:10:46.903934 9004 consensus_queue.cc:237] T ae5ac5988e024e7689a379e9c5a70c21 P df0db1a138554d00956d87f5c172ee10 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df0db1a138554d00956d87f5c172ee10" member_type: VOTER last_known_addr { host: "127.0.222.1" port: 43733 } } peers { permanent_uuid: "94413a87e0a14ef195baa2a799a9e0a8" member_type: VOTER last_known_addr { host: "127.0.222.2" port: 33587 } } peers { permanent_uuid: "c40926a9373242b0961349a2719c22ab" member_type: VOTER last_known_addr { host: "127.0.222.3" port: 37357 } }
I20260809 12:10:46.904865 8972 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 12:10:46.904276 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.222.62:41983 (local address 127.0.222.1:34659)
0809 12:10:46.904558 (+ 282us) negotiation.cc:107] Waiting for socket to connect
0809 12:10:46.904653 (+ 95us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.222.62:41983: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":125}
I20260809 12:10:46.907755 8384 catalog_manager.cc:5495] T ae5ac5988e024e7689a379e9c5a70c21 P df0db1a138554d00956d87f5c172ee10 reported cstate change: term changed from 0 to 1, leader changed from <none> to df0db1a138554d00956d87f5c172ee10 (127.0.222.1). New cstate: current_term: 1 leader_uuid: "df0db1a138554d00956d87f5c172ee10" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df0db1a138554d00956d87f5c172ee10" member_type: VOTER last_known_addr { host: "127.0.222.1" port: 43733 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "94413a87e0a14ef195baa2a799a9e0a8" member_type: VOTER last_known_addr { host: "127.0.222.2" port: 33587 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "c40926a9373242b0961349a2719c22ab" member_type: VOTER last_known_addr { host: "127.0.222.3" port: 37357 } health_report { overall_health: UNKNOWN } } }
I20260809 12:10:46.925326 8991 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 12:10:46.920527 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49272 (local address 127.0.222.1:43733)
0809 12:10:46.920706 (+ 179us) server_negotiation.cc:206] Beginning negotiation
0809 12:10:46.920710 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0809 12:10:46.920729 (+ 19us) server_negotiation.cc:407] Connection header received
0809 12:10:46.920842 (+ 113us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 12:10:46.920846 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 12:10:46.920924 (+ 78us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 12:10:46.921058 (+ 134us) server_negotiation.cc:226] Negotiated authn=TOKEN
0809 12:10:46.921378 (+ 320us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 12:10:46.922085 (+ 707us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 12:10:46.923396 (+ 1311us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 12:10:46.923831 (+ 435us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 12:10:46.923927 (+ 96us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 12:10:46.924272 (+ 345us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0809 12:10:46.924835 (+ 563us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0809 12:10:46.924952 (+ 117us) server_negotiation.cc:1035] Waiting for connection context
0809 12:10:46.925044 (+ 92us) server_negotiation.cc:299] Negotiation successful
0809 12:10:46.925113 (+ 69us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":54}
I20260809 12:10:46.929785 8677 raft_consensus.cc:1261] T ae5ac5988e024e7689a379e9c5a70c21 P 94413a87e0a14ef195baa2a799a9e0a8 [term 1 FOLLOWER]: Refusing update from remote peer df0db1a138554d00956d87f5c172ee10: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260809 12:10:46.930205 8829 raft_consensus.cc:1261] T ae5ac5988e024e7689a379e9c5a70c21 P c40926a9373242b0961349a2719c22ab [term 1 FOLLOWER]: Refusing update from remote peer df0db1a138554d00956d87f5c172ee10: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260809 12:10:46.930665 9008 consensus_queue.cc:1035] T ae5ac5988e024e7689a379e9c5a70c21 P df0db1a138554d00956d87f5c172ee10 [LEADER]: Connected to new peer: Peer: permanent_uuid: "94413a87e0a14ef195baa2a799a9e0a8" member_type: VOTER last_known_addr { host: "127.0.222.2" port: 33587 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260809 12:10:46.931095 9004 consensus_queue.cc:1035] T ae5ac5988e024e7689a379e9c5a70c21 P df0db1a138554d00956d87f5c172ee10 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c40926a9373242b0961349a2719c22ab" member_type: VOTER last_known_addr { host: "127.0.222.3" port: 37357 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260809 12:10:46.933845 8973 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 12:10:46.933359 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.222.62:41983 (local address 127.0.222.2:41809)
0809 12:10:46.933557 (+ 198us) negotiation.cc:107] Waiting for socket to connect
0809 12:10:46.933666 (+ 109us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.222.62:41983: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":53}
I20260809 12:10:46.934330 8974 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 12:10:46.933767 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.222.62:41983 (local address 127.0.222.3:59543)
0809 12:10:46.934029 (+ 262us) negotiation.cc:107] Waiting for socket to connect
0809 12:10:46.934143 (+ 114us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.222.62:41983: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":141}
I20260809 12:10:46.944371 8973 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 12:10:46.943572 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.222.62:41983 (local address 127.0.222.2:34951)
0809 12:10:46.944036 (+ 464us) negotiation.cc:107] Waiting for socket to connect
0809 12:10:46.944136 (+ 100us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.222.62:41983: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":320}
I20260809 12:10:46.944975 8974 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 12:10:46.944454 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.222.62:41983 (local address 127.0.222.3:34253)
0809 12:10:46.944683 (+ 229us) negotiation.cc:107] Waiting for socket to connect
0809 12:10:46.944781 (+ 98us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.222.62:41983: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":128}
I20260809 12:10:46.948438 8972 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 12:10:46.945667 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.222.62:41983 (local address 127.0.222.1:37191)
0809 12:10:46.948157 (+ 2490us) negotiation.cc:107] Waiting for socket to connect
0809 12:10:46.948271 (+ 114us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.222.62:41983: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":2349}
I20260809 12:10:46.953709 8972 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 12:10:46.952472 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.222.62:41983 (local address 127.0.222.1:45673)
0809 12:10:46.953332 (+ 860us) negotiation.cc:107] Waiting for socket to connect
0809 12:10:46.953504 (+ 172us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.222.62:41983: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":687}
I20260809 12:10:46.958760 8384 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:42488:
table { table_name: "test-table" } modify_external_catalogs: true
I20260809 12:10:46.959120 8384 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:42488:
table { table_name: "test-table" } modify_external_catalogs: true
I20260809 12:10:46.965413 8384 catalog_manager.cc:5782] T 00000000000000000000000000000000 P e2a7b439c3084cd680be92ac6d1925db: Sending DeleteTablet for 3 replicas of tablet ae5ac5988e024e7689a379e9c5a70c21
I20260809 12:10:46.966616 8656 tablet_service.cc:1514] Processing DeleteTablet for tablet ae5ac5988e024e7689a379e9c5a70c21 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-09 12:10:46 UTC) from {username='oryx', principal='oryx/127.0.222.60@KRBTEST.COM'} at 127.0.0.1:35432
I20260809 12:10:46.966889 8505 tablet_service.cc:1514] Processing DeleteTablet for tablet ae5ac5988e024e7689a379e9c5a70c21 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-09 12:10:46 UTC) from {username='oryx', principal='oryx/127.0.222.60@KRBTEST.COM'} at 127.0.0.1:49260
I20260809 12:10:46.967535 9020 tablet_replica.cc:330] T ae5ac5988e024e7689a379e9c5a70c21 P 94413a87e0a14ef195baa2a799a9e0a8: stopping tablet replica
I20260809 12:10:46.967844 9020 raft_consensus.cc:2229] T ae5ac5988e024e7689a379e9c5a70c21 P 94413a87e0a14ef195baa2a799a9e0a8 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260809 12:10:46.967895 8808 tablet_service.cc:1514] Processing DeleteTablet for tablet ae5ac5988e024e7689a379e9c5a70c21 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-09 12:10:46 UTC) from {username='oryx', principal='oryx/127.0.222.60@KRBTEST.COM'} at 127.0.0.1:33598
I20260809 12:10:46.968225 9020 raft_consensus.cc:2258] T ae5ac5988e024e7689a379e9c5a70c21 P 94413a87e0a14ef195baa2a799a9e0a8 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260809 12:10:46.969657 9020 ts_tablet_manager.cc:1905] T ae5ac5988e024e7689a379e9c5a70c21 P 94413a87e0a14ef195baa2a799a9e0a8: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260809 12:10:46.974031 888 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskgH5Pjm/build/asan/bin/kudu pid 8357
I20260809 12:10:46.979579 9022 tablet_replica.cc:330] T ae5ac5988e024e7689a379e9c5a70c21 P c40926a9373242b0961349a2719c22ab: stopping tablet replica
I20260809 12:10:46.979907 9022 raft_consensus.cc:2229] T ae5ac5988e024e7689a379e9c5a70c21 P c40926a9373242b0961349a2719c22ab [term 1 FOLLOWER]: Raft consensus shutting down.
I20260809 12:10:46.980309 9022 raft_consensus.cc:2258] T ae5ac5988e024e7689a379e9c5a70c21 P c40926a9373242b0961349a2719c22ab [term 1 FOLLOWER]: Raft consensus is shut down!
I20260809 12:10:46.981706 9022 ts_tablet_manager.cc:1905] T ae5ac5988e024e7689a379e9c5a70c21 P c40926a9373242b0961349a2719c22ab: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260809 12:10:46.983479 9020 ts_tablet_manager.cc:1918] T ae5ac5988e024e7689a379e9c5a70c21 P 94413a87e0a14ef195baa2a799a9e0a8: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260809 12:10:46.983713 9020 log.cc:1196] T ae5ac5988e024e7689a379e9c5a70c21 P 94413a87e0a14ef195baa2a799a9e0a8: Deleting WAL directory at /tmp/dist-test-taskgH5Pjm/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786277382073938-888-0/minicluster-data/ts-1/wal/wals/ae5ac5988e024e7689a379e9c5a70c21
I20260809 12:10:46.984354 9020 ts_tablet_manager.cc:1939] T ae5ac5988e024e7689a379e9c5a70c21 P 94413a87e0a14ef195baa2a799a9e0a8: Deleting consensus metadata
I20260809 12:10:46.985786 8369 catalog_manager.cc:4841] TS 94413a87e0a14ef195baa2a799a9e0a8 (127.0.222.2:33587): tablet ae5ac5988e024e7689a379e9c5a70c21 (table test-table [id=3a58cbf1ca2446f38efb53e00bb5f763]) successfully deleted
I20260809 12:10:46.985860 9021 tablet_replica.cc:330] T ae5ac5988e024e7689a379e9c5a70c21 P df0db1a138554d00956d87f5c172ee10: stopping tablet replica
I20260809 12:10:46.986150 9021 raft_consensus.cc:2229] T ae5ac5988e024e7689a379e9c5a70c21 P df0db1a138554d00956d87f5c172ee10 [term 1 LEADER]: Raft consensus shutting down.
I20260809 12:10:46.986943 9021 raft_consensus.cc:2258] T ae5ac5988e024e7689a379e9c5a70c21 P df0db1a138554d00956d87f5c172ee10 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260809 12:10:46.988430 9021 ts_tablet_manager.cc:1905] T ae5ac5988e024e7689a379e9c5a70c21 P df0db1a138554d00956d87f5c172ee10: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260809 12:10:46.989883 9022 ts_tablet_manager.cc:1918] T ae5ac5988e024e7689a379e9c5a70c21 P c40926a9373242b0961349a2719c22ab: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260809 12:10:46.990104 9022 log.cc:1196] T ae5ac5988e024e7689a379e9c5a70c21 P c40926a9373242b0961349a2719c22ab: Deleting WAL directory at /tmp/dist-test-taskgH5Pjm/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786277382073938-888-0/minicluster-data/ts-2/wal/wals/ae5ac5988e024e7689a379e9c5a70c21
I20260809 12:10:46.990888 9022 ts_tablet_manager.cc:1939] T ae5ac5988e024e7689a379e9c5a70c21 P c40926a9373242b0961349a2719c22ab: Deleting consensus metadata
I20260809 12:10:46.992280 8371 catalog_manager.cc:4841] TS c40926a9373242b0961349a2719c22ab (127.0.222.3:37357): tablet ae5ac5988e024e7689a379e9c5a70c21 (table test-table [id=3a58cbf1ca2446f38efb53e00bb5f763]) successfully deleted
I20260809 12:10:46.996476 9021 ts_tablet_manager.cc:1918] T ae5ac5988e024e7689a379e9c5a70c21 P df0db1a138554d00956d87f5c172ee10: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260809 12:10:46.996686 9021 log.cc:1196] T ae5ac5988e024e7689a379e9c5a70c21 P df0db1a138554d00956d87f5c172ee10: Deleting WAL directory at /tmp/dist-test-taskgH5Pjm/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786277382073938-888-0/minicluster-data/ts-0/wal/wals/ae5ac5988e024e7689a379e9c5a70c21
I20260809 12:10:46.997252 9021 ts_tablet_manager.cc:1939] T ae5ac5988e024e7689a379e9c5a70c21 P df0db1a138554d00956d87f5c172ee10: Deleting consensus metadata
I20260809 12:10:46.998400 8369 catalog_manager.cc:4841] TS df0db1a138554d00956d87f5c172ee10 (127.0.222.1:43733): tablet ae5ac5988e024e7689a379e9c5a70c21 (table test-table [id=3a58cbf1ca2446f38efb53e00bb5f763]) successfully deleted
I20260809 12:10:47.006649 8991 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 12:10:46.975635 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49280 (local address 127.0.222.1:43733)
0809 12:10:46.975895 (+ 260us) server_negotiation.cc:206] Beginning negotiation
0809 12:10:46.975904 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0809 12:10:46.989585 (+ 13681us) server_negotiation.cc:407] Connection header received
0809 12:10:46.989807 (+ 222us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 12:10:46.989813 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 12:10:46.989895 (+ 82us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 12:10:46.990007 (+ 112us) server_negotiation.cc:226] Negotiated authn=SASL
0809 12:10:46.991828 (+ 1821us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 12:10:46.993316 (+ 1488us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 12:10:46.995503 (+ 2187us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 12:10:46.996182 (+ 679us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 12:10:46.996329 (+ 147us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 12:10:47.000202 (+ 3873us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 12:10:47.000215 (+ 13us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 12:10:47.000232 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0809 12:10:47.000274 (+ 42us) server_negotiation.cc:922] calling sasl_server_start()
0809 12:10:47.003562 (+ 3288us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 12:10:47.004478 (+ 916us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 12:10:47.004495 (+ 17us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 12:10:47.004498 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0809 12:10:47.004647 (+ 149us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 12:10:47.005085 (+ 438us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 12:10:47.005088 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 12:10:47.005091 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0809 12:10:47.005446 (+ 355us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 12:10:47.005709 (+ 263us) server_negotiation.cc:1035] Waiting for connection context
0809 12:10:47.006124 (+ 415us) server_negotiation.cc:299] Negotiation successful
0809 12:10:47.006299 (+ 175us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":87}
I20260809 12:10:47.007463 8566 generic_service.cc:196] Checking for leaks (request via RPC)
I20260809 12:10:47.126420 888 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskgH5Pjm/build/asan/bin/kudu with pid 8357
I20260809 12:10:47.145526 9025 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 12:10:47.144787 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.222.62:41983 (local address 127.0.0.1:33900)
0809 12:10:47.145225 (+ 438us) negotiation.cc:107] Waiting for socket to connect
0809 12:10:47.145338 (+ 113us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.222.62:41983: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":338,"thread_start_us":238,"threads_started":1}
I20260809 12:10:47.146853 9025 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 12:10:47.146524 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.222.62:41983 (local address 127.0.0.1:33914)
0809 12:10:47.146661 (+ 137us) negotiation.cc:107] Waiting for socket to connect
0809 12:10:47.146729 (+ 68us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.222.62:41983: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":53}
I20260809 12:10:47.149771 888 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskgH5Pjm/build/asan/bin/kudu pid 8589
I20260809 12:10:47.163847 8996 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 12:10:47.150487 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35440 (local address 127.0.222.2:33587)
0809 12:10:47.150715 (+ 228us) server_negotiation.cc:206] Beginning negotiation
0809 12:10:47.150719 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0809 12:10:47.150737 (+ 18us) server_negotiation.cc:407] Connection header received
0809 12:10:47.150826 (+ 89us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 12:10:47.150831 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 12:10:47.150893 (+ 62us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 12:10:47.151045 (+ 152us) server_negotiation.cc:226] Negotiated authn=SASL
0809 12:10:47.152393 (+ 1348us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 12:10:47.153535 (+ 1142us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 12:10:47.154812 (+ 1277us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 12:10:47.155462 (+ 650us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 12:10:47.155573 (+ 111us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 12:10:47.158187 (+ 2614us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 12:10:47.158202 (+ 15us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 12:10:47.158218 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0809 12:10:47.158254 (+ 36us) server_negotiation.cc:922] calling sasl_server_start()
0809 12:10:47.161379 (+ 3125us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 12:10:47.162005 (+ 626us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 12:10:47.162012 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 12:10:47.162014 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0809 12:10:47.162147 (+ 133us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 12:10:47.162462 (+ 315us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 12:10:47.162465 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 12:10:47.162467 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0809 12:10:47.162800 (+ 333us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 12:10:47.163145 (+ 345us) server_negotiation.cc:1035] Waiting for connection context
0809 12:10:47.163319 (+ 174us) server_negotiation.cc:299] Negotiation successful
0809 12:10:47.163501 (+ 182us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":127}
I20260809 12:10:47.164528 8721 generic_service.cc:196] Checking for leaks (request via RPC)
I20260809 12:10:47.259196 888 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskgH5Pjm/build/asan/bin/kudu with pid 8589
I20260809 12:10:47.280706 888 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskgH5Pjm/build/asan/bin/kudu pid 8739
I20260809 12:10:47.293928 8994 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 12:10:47.281484 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33600 (local address 127.0.222.3:37357)
0809 12:10:47.281702 (+ 218us) server_negotiation.cc:206] Beginning negotiation
0809 12:10:47.281707 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0809 12:10:47.281725 (+ 18us) server_negotiation.cc:407] Connection header received
0809 12:10:47.281854 (+ 129us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 12:10:47.281858 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 12:10:47.281922 (+ 64us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 12:10:47.282020 (+ 98us) server_negotiation.cc:226] Negotiated authn=SASL
0809 12:10:47.283445 (+ 1425us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 12:10:47.284191 (+ 746us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 12:10:47.285329 (+ 1138us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 12:10:47.285716 (+ 387us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 12:10:47.285804 (+ 88us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 12:10:47.288371 (+ 2567us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 12:10:47.288386 (+ 15us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 12:10:47.288401 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0809 12:10:47.288433 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0809 12:10:47.291492 (+ 3059us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 12:10:47.292092 (+ 600us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 12:10:47.292099 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 12:10:47.292101 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0809 12:10:47.292221 (+ 120us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 12:10:47.292545 (+ 324us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 12:10:47.292549 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 12:10:47.292551 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0809 12:10:47.292921 (+ 370us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 12:10:47.293156 (+ 235us) server_negotiation.cc:1035] Waiting for connection context
0809 12:10:47.293397 (+ 241us) server_negotiation.cc:299] Negotiation successful
0809 12:10:47.293569 (+ 172us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":72}
I20260809 12:10:47.294641 8872 generic_service.cc:196] Checking for leaks (request via RPC)
I20260809 12:10:47.419086 888 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskgH5Pjm/build/asan/bin/kudu with pid 8739
I20260809 12:10:47.444965 888 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskgH5Pjm/build/asan/bin/kudu pid 8034
I20260809 12:10:47.446215 8216 generic_service.cc:196] Checking for leaks (request via RPC)
I20260809 12:10:47.566895 888 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskgH5Pjm/build/asan/bin/kudu with pid 8034
I20260809 12:10:47.591768 888 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskgH5Pjm/build/asan/bin/kudu pid 8145
I20260809 12:10:47.592978 8416 generic_service.cc:196] Checking for leaks (request via RPC)
I20260809 12:10:47.718104 888 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskgH5Pjm/build/asan/bin/kudu with pid 8145
2026-08-09T12:10:47Z chronyd exiting
[ OK ] SecurityITest.TestNonDefaultPrincipalMultipleMaster (12047 ms)
[ RUN ] SecurityITest.TestNonDefaultPrincipal
Loading random data
Initializing database '/tmp/dist-test-taskgH5Pjm/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786277382073938-888-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 09 12:10:47 dist-test-slave-2w3w krb5kdc[9036](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 09 12:10:47 dist-test-slave-2w3w krb5kdc[9036](info): set up 2 sockets
Aug 09 12:10:47 dist-test-slave-2w3w krb5kdc[9036](info): commencing operation
krb5kdc: starting...
W20260809 12:10:48.983377 888 mini_kdc.cc:121] Time spent starting KDC: real 1.214s user 0.003s sys 0.034s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskgH5Pjm/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786277382073938-888-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskgH5Pjm/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786277382073938-888-0/krb5kdc/test-user.keytab.
Aug 09 12:10:49 dist-test-slave-2w3w krb5kdc[9036](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786277449, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-09T12:10:49Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-09T12:10:49Z Disabled control of system clock
WARNING: no policy specified for oryx/127.0.222.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.0.222.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.0.222.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskgH5Pjm/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786277382073938-888-0/krb5kdc/oryx_127.0.222.62.keytab.
Entry for principal oryx/127.0.222.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskgH5Pjm/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786277382073938-888-0/krb5kdc/oryx_127.0.222.62.keytab.
I20260809 12:10:49.262122 888 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskgH5Pjm/build/asan/bin/kudu
/tmp/dist-test-taskgH5Pjm/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskgH5Pjm/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786277382073938-888-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskgH5Pjm/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786277382073938-888-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskgH5Pjm/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786277382073938-888-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskgH5Pjm/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786277382073938-888-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.222.62:36963
--webserver_interface=127.0.222.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskgH5Pjm/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786277382073938-888-0/krb5kdc/oryx_127.0.222.62.keytab
--principal=oryx/127.0.222.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.222.20:45275
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.222.62:36963
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--txn_manager_enabled=true with env {KRB5CCNAME=/tmp/dist-test-taskgH5Pjm/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786277382073938-888-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskgH5Pjm/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786277382073938-888-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskgH5Pjm/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786277382073938-888-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260809 12:10:49.600728 9050 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260809 12:10:49.601066 9050 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260809 12:10:49.601224 9050 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260809 12:10:49.608455 9050 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260809 12:10:49.608561 9050 flags.cc:425] Enabled experimental flag: --txn_manager_enabled=true
W20260809 12:10:49.608675 9050 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260809 12:10:49.608784 9050 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260809 12:10:49.608882 9050 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260809 12:10:49.608999 9050 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260809 12:10:49.619004 9050 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.222.20:45275
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskgH5Pjm/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786277382073938-888-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskgH5Pjm/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786277382073938-888-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.222.62:36963
--txn_manager_enabled=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskgH5Pjm/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786277382073938-888-0/krb5kdc/oryx_127.0.222.62.keytab
--principal=oryx/127.0.222.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.222.62:36963
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskgH5Pjm/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786277382073938-888-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.222.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskgH5Pjm/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786277382073938-888-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 09 Aug 2026 11:06:41 UTC on ccycloud.quasar-bwoogy.root.comops.site
build id 4575991
ASAN enabled
I20260809 12:10:49.620362 9050 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260809 12:10:49.621712 9050 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260809 12:10:49.634423 9055 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 12:10:49.634454 9056 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 12:10:49.634388 9058 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260809 12:10:49.635146 9050 server_base.cc:1034] running on GCE node
I20260809 12:10:49.635927 9050 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260809 12:10:49.637305 9050 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260809 12:10:49.638550 9050 hybrid_clock.cc:648] HybridClock initialized: now 1786277449638489 us; error 38 us; skew 500 ppm
I20260809 12:10:49.638845 9050 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 09 12:10:49 dist-test-slave-2w3w krb5kdc[9036](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786277449, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.222.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260809 12:10:49.642058 9050 init.cc:377] Logged in from keytab as oryx/127.0.222.62@KRBTEST.COM (short username oryx)
I20260809 12:10:49.643606 9050 webserver.cc:460] Webserver started at http://127.0.222.62:42355/ using document root <none> and password file <none>
I20260809 12:10:49.644289 9050 fs_manager.cc:359] Metadata directory not provided
I20260809 12:10:49.644419 9050 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260809 12:10:49.644757 9050 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260809 12:10:49.647776 9050 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskgH5Pjm/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786277382073938-888-0/minicluster-data/master-0/data/instance:
uuid: "7873beb5ce214b9d8d238a51db1accb1"
format_stamp: "Formatted at 2026-08-09 12:10:49 on dist-test-slave-2w3w"
server_key: "834d6d2c23ef3719f37eebf996a6448e"
server_key_iv: "0a87731a7d9b2045c8f76e48395580f2"
server_key_version: "encryptionkey@0"
I20260809 12:10:49.648582 9050 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskgH5Pjm/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786277382073938-888-0/minicluster-data/master-0/wal/instance:
uuid: "7873beb5ce214b9d8d238a51db1accb1"
format_stamp: "Formatted at 2026-08-09 12:10:49 on dist-test-slave-2w3w"
server_key: "834d6d2c23ef3719f37eebf996a6448e"
server_key_iv: "0a87731a7d9b2045c8f76e48395580f2"
server_key_version: "encryptionkey@0"
I20260809 12:10:49.653757 9050 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.000s
I20260809 12:10:49.657471 9067 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260809 12:10:49.658708 9050 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260809 12:10:49.658905 9050 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskgH5Pjm/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786277382073938-888-0/minicluster-data/master-0/data,/tmp/dist-test-taskgH5Pjm/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786277382073938-888-0/minicluster-data/master-0/wal
uuid: "7873beb5ce214b9d8d238a51db1accb1"
format_stamp: "Formatted at 2026-08-09 12:10:49 on dist-test-slave-2w3w"
server_key: "834d6d2c23ef3719f37eebf996a6448e"
server_key_iv: "0a87731a7d9b2045c8f76e48395580f2"
server_key_version: "encryptionkey@0"
I20260809 12:10:49.659179 9050 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskgH5Pjm/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786277382073938-888-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskgH5Pjm/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786277382073938-888-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskgH5Pjm/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786277382073938-888-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260809 12:10:49.684917 9050 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260809 12:10:49.689092 9050 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260809 12:10:49.689412 9050 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260809 12:10:49.712641 9050 rpc_server.cc:307] RPC server started. Bound to: 127.0.222.62:36963
I20260809 12:10:49.712674 9132 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.222.62:36963 every 8 connection(s)
I20260809 12:10:49.714059 9050 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskgH5Pjm/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786277382073938-888-0/minicluster-data/master-0/data/info.pb
I20260809 12:10:49.717849 9133 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260809 12:10:49.718947 888 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskgH5Pjm/build/asan/bin/kudu as pid 9050
I20260809 12:10:49.719192 888 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskgH5Pjm/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786277382073938-888-0/minicluster-data/master-0/wal/instance
I20260809 12:10:49.720118 888 external_mini_cluster.cc:1428] Setting key a967470609c51d33d954c1d3bc8c6ea4
I20260809 12:10:49.732640 9133 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 7873beb5ce214b9d8d238a51db1accb1: Bootstrap starting.
Aug 09 12:10:49 dist-test-slave-2w3w krb5kdc[9036](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786277449, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.0.222.62@KRBTEST.COM
I20260809 12:10:49.736617 9133 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 7873beb5ce214b9d8d238a51db1accb1: Neither blocks nor log segments found. Creating new log.
I20260809 12:10:49.737720 9133 log.cc:824] T 00000000000000000000000000000000 P 7873beb5ce214b9d8d238a51db1accb1: Log is configured to *not* fsync() on all Append() calls
I20260809 12:10:49.741014 9133 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 7873beb5ce214b9d8d238a51db1accb1: No bootstrap required, opened a new log
I20260809 12:10:49.743824 9136 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 12:10:49.722275 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53604 (local address 127.0.222.62:36963)
0809 12:10:49.723111 (+ 836us) server_negotiation.cc:206] Beginning negotiation
0809 12:10:49.723125 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0809 12:10:49.723164 (+ 39us) server_negotiation.cc:407] Connection header received
0809 12:10:49.724771 (+ 1607us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 12:10:49.724793 (+ 22us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 12:10:49.725169 (+ 376us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 12:10:49.725499 (+ 330us) server_negotiation.cc:226] Negotiated authn=SASL
0809 12:10:49.727276 (+ 1777us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 12:10:49.728599 (+ 1323us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 12:10:49.730415 (+ 1816us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 12:10:49.731147 (+ 732us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 12:10:49.731274 (+ 127us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 12:10:49.736178 (+ 4904us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 12:10:49.736186 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 12:10:49.736200 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0809 12:10:49.736251 (+ 51us) server_negotiation.cc:922] calling sasl_server_start()
0809 12:10:49.739476 (+ 3225us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 12:10:49.740299 (+ 823us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 12:10:49.740305 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 12:10:49.740318 (+ 13us) server_negotiation.cc:967] calling sasl_server_step()
0809 12:10:49.740500 (+ 182us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 12:10:49.740972 (+ 472us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 12:10:49.740976 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 12:10:49.740978 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0809 12:10:49.741588 (+ 610us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 12:10:49.741882 (+ 294us) server_negotiation.cc:1035] Waiting for connection context
0809 12:10:49.742297 (+ 415us) server_negotiation.cc:299] Negotiation successful
0809 12:10:49.742653 (+ 356us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":592,"thread_start_us":387,"threads_started":1}
I20260809 12:10:49.749270 9133 raft_consensus.cc:348] T 00000000000000000000000000000000 P 7873beb5ce214b9d8d238a51db1accb1 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7873beb5ce214b9d8d238a51db1accb1" member_type: VOTER last_known_addr { host: "127.0.222.62" port: 36963 } }
I20260809 12:10:49.749657 9133 raft_consensus.cc:374] T 00000000000000000000000000000000 P 7873beb5ce214b9d8d238a51db1accb1 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260809 12:10:49.749773 9133 raft_consensus.cc:729] T 00000000000000000000000000000000 P 7873beb5ce214b9d8d238a51db1accb1 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7873beb5ce214b9d8d238a51db1accb1, State: Initialized, Role: FOLLOWER
I20260809 12:10:49.750339 9133 consensus_queue.cc:260] T 00000000000000000000000000000000 P 7873beb5ce214b9d8d238a51db1accb1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7873beb5ce214b9d8d238a51db1accb1" member_type: VOTER last_known_addr { host: "127.0.222.62" port: 36963 } }
I20260809 12:10:49.750594 9133 raft_consensus.cc:388] T 00000000000000000000000000000000 P 7873beb5ce214b9d8d238a51db1accb1 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260809 12:10:49.750747 9133 raft_consensus.cc:482] T 00000000000000000000000000000000 P 7873beb5ce214b9d8d238a51db1accb1 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260809 12:10:49.750922 9133 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 7873beb5ce214b9d8d238a51db1accb1 [term 0 FOLLOWER]: Advancing to term 1
I20260809 12:10:49.753187 9133 raft_consensus.cc:504] T 00000000000000000000000000000000 P 7873beb5ce214b9d8d238a51db1accb1 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7873beb5ce214b9d8d238a51db1accb1" member_type: VOTER last_known_addr { host: "127.0.222.62" port: 36963 } }
I20260809 12:10:49.753702 9133 leader_election.cc:302] T 00000000000000000000000000000000 P 7873beb5ce214b9d8d238a51db1accb1 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 7873beb5ce214b9d8d238a51db1accb1; no voters:
I20260809 12:10:49.754269 9133 leader_election.cc:288] T 00000000000000000000000000000000 P 7873beb5ce214b9d8d238a51db1accb1 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260809 12:10:49.754418 9138 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 7873beb5ce214b9d8d238a51db1accb1 [term 1 FOLLOWER]: Leader election won for term 1
I20260809 12:10:49.755764 9138 raft_consensus.cc:686] T 00000000000000000000000000000000 P 7873beb5ce214b9d8d238a51db1accb1 [term 1 LEADER]: Becoming Leader. State: Replica: 7873beb5ce214b9d8d238a51db1accb1, State: Running, Role: LEADER
I20260809 12:10:49.756354 9138 consensus_queue.cc:237] T 00000000000000000000000000000000 P 7873beb5ce214b9d8d238a51db1accb1 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7873beb5ce214b9d8d238a51db1accb1" member_type: VOTER last_known_addr { host: "127.0.222.62" port: 36963 } }
I20260809 12:10:49.756669 9133 sys_catalog.cc:564] T 00000000000000000000000000000000 P 7873beb5ce214b9d8d238a51db1accb1 [sys.catalog]: configured and running, proceeding with master startup.
I20260809 12:10:49.759218 9139 sys_catalog.cc:455] T 00000000000000000000000000000000 P 7873beb5ce214b9d8d238a51db1accb1 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "7873beb5ce214b9d8d238a51db1accb1" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7873beb5ce214b9d8d238a51db1accb1" member_type: VOTER last_known_addr { host: "127.0.222.62" port: 36963 } } }
I20260809 12:10:49.759627 9139 sys_catalog.cc:458] T 00000000000000000000000000000000 P 7873beb5ce214b9d8d238a51db1accb1 [sys.catalog]: This master's current role is: LEADER
I20260809 12:10:49.760200 9140 sys_catalog.cc:455] T 00000000000000000000000000000000 P 7873beb5ce214b9d8d238a51db1accb1 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 7873beb5ce214b9d8d238a51db1accb1. Latest consensus state: current_term: 1 leader_uuid: "7873beb5ce214b9d8d238a51db1accb1" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7873beb5ce214b9d8d238a51db1accb1" member_type: VOTER last_known_addr { host: "127.0.222.62" port: 36963 } } }
I20260809 12:10:49.760602 9140 sys_catalog.cc:458] T 00000000000000000000000000000000 P 7873beb5ce214b9d8d238a51db1accb1 [sys.catalog]: This master's current role is: LEADER
I20260809 12:10:49.769356 9149 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260809 12:10:49.774691 9149 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260809 12:10:49.783420 9149 catalog_manager.cc:1347] Generated new cluster ID: 04706f3e2b2044f1a35a603b9e91f27e
I20260809 12:10:49.783543 9149 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260809 12:10:49.794159 9149 catalog_manager.cc:1370] Generated new certificate authority record
I20260809 12:10:49.795950 9149 catalog_manager.cc:1504] Loading token signing keys...
I20260809 12:10:49.812757 9149 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 7873beb5ce214b9d8d238a51db1accb1: Generated new TSK 0
I20260809 12:10:49.813628 9149 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for oryx/127.0.222.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.0.222.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.0.222.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskgH5Pjm/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786277382073938-888-0/krb5kdc/oryx_127.0.222.1.keytab.
Entry for principal oryx/127.0.222.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskgH5Pjm/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786277382073938-888-0/krb5kdc/oryx_127.0.222.1.keytab.
I20260809 12:10:49.892397 888 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskgH5Pjm/build/asan/bin/kudu
/tmp/dist-test-taskgH5Pjm/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskgH5Pjm/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786277382073938-888-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskgH5Pjm/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786277382073938-888-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskgH5Pjm/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786277382073938-888-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskgH5Pjm/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786277382073938-888-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.222.1:0
--local_ip_for_outbound_sockets=127.0.222.1
--webserver_interface=127.0.222.1
--webserver_port=0
--tserver_master_addrs=127.0.222.62:36963
--keytab_file=/tmp/dist-test-taskgH5Pjm/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786277382073938-888-0/krb5kdc/oryx_127.0.222.1.keytab
--principal=oryx/127.0.222.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.222.20:45275
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true with env {KRB5CCNAME=/tmp/dist-test-taskgH5Pjm/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786277382073938-888-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskgH5Pjm/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786277382073938-888-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskgH5Pjm/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786277382073938-888-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260809 12:10:50.246206 9164 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260809 12:10:50.246493 9164 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260809 12:10:50.246593 9164 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260809 12:10:50.254032 9164 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260809 12:10:50.254141 9164 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260809 12:10:50.254226 9164 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260809 12:10:50.254321 9164 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.222.1
I20260809 12:10:50.264976 9164 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.222.20:45275
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskgH5Pjm/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786277382073938-888-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskgH5Pjm/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786277382073938-888-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskgH5Pjm/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786277382073938-888-0/krb5kdc/oryx_127.0.222.1.keytab
--principal=oryx/127.0.222.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.222.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskgH5Pjm/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786277382073938-888-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.222.1
--webserver_port=0
--enable_txn_system_client_init=true
--tserver_master_addrs=127.0.222.62:36963
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.222.1
--log_dir=/tmp/dist-test-taskgH5Pjm/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786277382073938-888-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 09 Aug 2026 11:06:41 UTC on ccycloud.quasar-bwoogy.root.comops.site
build id 4575991
ASAN enabled
I20260809 12:10:50.266249 9164 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260809 12:10:50.267819 9164 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260809 12:10:50.280699 9169 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 12:10:50.281064 9172 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 12:10:50.281194 9170 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260809 12:10:50.281873 9164 server_base.cc:1034] running on GCE node
I20260809 12:10:50.282485 9164 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260809 12:10:50.283421 9164 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260809 12:10:50.284647 9164 hybrid_clock.cc:648] HybridClock initialized: now 1786277450284577 us; error 38 us; skew 500 ppm
I20260809 12:10:50.284963 9164 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 09 12:10:50 dist-test-slave-2w3w krb5kdc[9036](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786277450, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.222.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260809 12:10:50.288259 9164 init.cc:377] Logged in from keytab as oryx/127.0.222.1@KRBTEST.COM (short username oryx)
I20260809 12:10:50.289691 9164 webserver.cc:460] Webserver started at http://127.0.222.1:38439/ using document root <none> and password file <none>
I20260809 12:10:50.290421 9164 fs_manager.cc:359] Metadata directory not provided
I20260809 12:10:50.290566 9164 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260809 12:10:50.290899 9164 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260809 12:10:50.293862 9164 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskgH5Pjm/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786277382073938-888-0/minicluster-data/ts-0/data/instance:
uuid: "af75f3a57f6f48fba26736ad36aff045"
format_stamp: "Formatted at 2026-08-09 12:10:50 on dist-test-slave-2w3w"
server_key: "498c1369b797daa7011b945516cdfb0c"
server_key_iv: "4308736056d83d9e291ad2fe78a87988"
server_key_version: "encryptionkey@0"
I20260809 12:10:50.294701 9164 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskgH5Pjm/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786277382073938-888-0/minicluster-data/ts-0/wal/instance:
uuid: "af75f3a57f6f48fba26736ad36aff045"
format_stamp: "Formatted at 2026-08-09 12:10:50 on dist-test-slave-2w3w"
server_key: "498c1369b797daa7011b945516cdfb0c"
server_key_iv: "4308736056d83d9e291ad2fe78a87988"
server_key_version: "encryptionkey@0"
I20260809 12:10:50.300024 9164 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.004s sys 0.000s
I20260809 12:10:50.303557 9179 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260809 12:10:50.304876 9164 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260809 12:10:50.305068 9164 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskgH5Pjm/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786277382073938-888-0/minicluster-data/ts-0/data,/tmp/dist-test-taskgH5Pjm/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786277382073938-888-0/minicluster-data/ts-0/wal
uuid: "af75f3a57f6f48fba26736ad36aff045"
format_stamp: "Formatted at 2026-08-09 12:10:50 on dist-test-slave-2w3w"
server_key: "498c1369b797daa7011b945516cdfb0c"
server_key_iv: "4308736056d83d9e291ad2fe78a87988"
server_key_version: "encryptionkey@0"
I20260809 12:10:50.305299 9164 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskgH5Pjm/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786277382073938-888-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskgH5Pjm/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786277382073938-888-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskgH5Pjm/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786277382073938-888-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260809 12:10:50.325853 9164 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260809 12:10:50.329968 9164 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260809 12:10:50.330215 9164 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260809 12:10:50.333042 9164 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260809 12:10:50.333179 9164 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260809 12:10:50.333391 9164 ts_tablet_manager.cc:610] Registered 0 tablets
I20260809 12:10:50.333487 9164 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Aug 09 12:10:50 dist-test-slave-2w3w krb5kdc[9036](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786277450, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.222.1@KRBTEST.COM for oryx/127.0.222.62@KRBTEST.COM
I20260809 12:10:50.358595 9191 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 12:10:50.334135 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.222.62:36963 (local address 127.0.222.1:37401)
0809 12:10:50.335110 (+ 975us) negotiation.cc:107] Waiting for socket to connect
0809 12:10:50.335163 (+ 53us) client_negotiation.cc:174] Beginning negotiation
0809 12:10:50.337035 (+ 1872us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 12:10:50.337787 (+ 752us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 12:10:50.337799 (+ 12us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 12:10:50.338612 (+ 813us) client_negotiation.cc:189] Negotiated authn=SASL
0809 12:10:50.339564 (+ 952us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 12:10:50.339603 (+ 39us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 12:10:50.341396 (+ 1793us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 12:10:50.341409 (+ 13us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 12:10:50.343739 (+ 2330us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 12:10:50.343764 (+ 25us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 12:10:50.345081 (+ 1317us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 12:10:50.345093 (+ 12us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 12:10:50.345710 (+ 617us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 12:10:50.347092 (+ 1382us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0809 12:10:50.347250 (+ 158us) client_negotiation.cc:656] calling sasl_client_start()
0809 12:10:50.349592 (+ 2342us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0809 12:10:50.353601 (+ 4009us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0809 12:10:50.353618 (+ 17us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0809 12:10:50.353631 (+ 13us) client_negotiation.cc:760] calling sasl_client_step()
0809 12:10:50.354039 (+ 408us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0809 12:10:50.354521 (+ 482us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0809 12:10:50.354525 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0809 12:10:50.354527 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0809 12:10:50.354719 (+ 192us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0809 12:10:50.355511 (+ 792us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0809 12:10:50.355523 (+ 12us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0809 12:10:50.355972 (+ 449us) client_negotiation.cc:769] Sending connection context
0809 12:10:50.356384 (+ 412us) client_negotiation.cc:240] Negotiation successful
0809 12:10:50.357027 (+ 643us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":655,"thread_start_us":369,"threads_started":1}
I20260809 12:10:50.359951 9188 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 12:10:50.333436 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.222.1:37401 (local address 127.0.222.62:36963)
0809 12:10:50.334111 (+ 675us) server_negotiation.cc:206] Beginning negotiation
0809 12:10:50.334118 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0809 12:10:50.335306 (+ 1188us) server_negotiation.cc:407] Connection header received
0809 12:10:50.337332 (+ 2026us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 12:10:50.337337 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 12:10:50.337430 (+ 93us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 12:10:50.337549 (+ 119us) server_negotiation.cc:226] Negotiated authn=SASL
0809 12:10:50.339801 (+ 2252us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 12:10:50.341167 (+ 1366us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 12:10:50.343972 (+ 2805us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 12:10:50.344716 (+ 744us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 12:10:50.344868 (+ 152us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 12:10:50.349838 (+ 4970us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 12:10:50.349843 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 12:10:50.349847 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0809 12:10:50.349882 (+ 35us) server_negotiation.cc:922] calling sasl_server_start()
0809 12:10:50.353431 (+ 3549us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 12:10:50.354184 (+ 753us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 12:10:50.354192 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 12:10:50.354199 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0809 12:10:50.354396 (+ 197us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 12:10:50.354867 (+ 471us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 12:10:50.354875 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 12:10:50.354881 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0809 12:10:50.355323 (+ 442us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 12:10:50.355492 (+ 169us) server_negotiation.cc:1035] Waiting for connection context
0809 12:10:50.359388 (+ 3896us) server_negotiation.cc:299] Negotiation successful
0809 12:10:50.359662 (+ 274us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":490,"thread_start_us":351,"threads_started":1}
I20260809 12:10:50.384337 9256 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 12:10:50.370319 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.222.62:36963 (local address 127.0.222.1:37127)
0809 12:10:50.371151 (+ 832us) negotiation.cc:107] Waiting for socket to connect
0809 12:10:50.371183 (+ 32us) client_negotiation.cc:174] Beginning negotiation
0809 12:10:50.371370 (+ 187us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 12:10:50.371720 (+ 350us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 12:10:50.371724 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 12:10:50.372205 (+ 481us) client_negotiation.cc:189] Negotiated authn=SASL
0809 12:10:50.372502 (+ 297us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 12:10:50.372513 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 12:10:50.373519 (+ 1006us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 12:10:50.373525 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 12:10:50.374643 (+ 1118us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 12:10:50.374656 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 12:10:50.375757 (+ 1101us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 12:10:50.375763 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 12:10:50.375923 (+ 160us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 12:10:50.376784 (+ 861us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0809 12:10:50.376813 (+ 29us) client_negotiation.cc:656] calling sasl_client_start()
0809 12:10:50.378000 (+ 1187us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0809 12:10:50.381536 (+ 3536us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0809 12:10:50.381542 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0809 12:10:50.381546 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0809 12:10:50.381975 (+ 429us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0809 12:10:50.382376 (+ 401us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0809 12:10:50.382381 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0809 12:10:50.382385 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0809 12:10:50.382521 (+ 136us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0809 12:10:50.383159 (+ 638us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0809 12:10:50.383165 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0809 12:10:50.383419 (+ 254us) client_negotiation.cc:769] Sending connection context
0809 12:10:50.383624 (+ 205us) client_negotiation.cc:240] Negotiation successful
0809 12:10:50.384005 (+ 381us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":599,"thread_start_us":454,"threads_started":1}
I20260809 12:10:50.384660 9188 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 12:10:50.370231 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.222.1:37127 (local address 127.0.222.62:36963)
0809 12:10:50.370408 (+ 177us) server_negotiation.cc:206] Beginning negotiation
0809 12:10:50.370412 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0809 12:10:50.371294 (+ 882us) server_negotiation.cc:407] Connection header received
0809 12:10:50.371501 (+ 207us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 12:10:50.371513 (+ 12us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 12:10:50.371577 (+ 64us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 12:10:50.371682 (+ 105us) server_negotiation.cc:226] Negotiated authn=SASL
0809 12:10:50.372657 (+ 975us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 12:10:50.373375 (+ 718us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 12:10:50.374805 (+ 1430us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 12:10:50.375311 (+ 506us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 12:10:50.375478 (+ 167us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 12:10:50.378272 (+ 2794us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 12:10:50.378277 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 12:10:50.378280 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0809 12:10:50.378318 (+ 38us) server_negotiation.cc:922] calling sasl_server_start()
0809 12:10:50.381399 (+ 3081us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 12:10:50.382101 (+ 702us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 12:10:50.382107 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 12:10:50.382111 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0809 12:10:50.382248 (+ 137us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 12:10:50.382634 (+ 386us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 12:10:50.382639 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 12:10:50.382643 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0809 12:10:50.382936 (+ 293us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 12:10:50.383205 (+ 269us) server_negotiation.cc:1035] Waiting for connection context
0809 12:10:50.383780 (+ 575us) server_negotiation.cc:299] Negotiation successful
0809 12:10:50.384139 (+ 359us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":59}
I20260809 12:10:50.397492 9164 rpc_server.cc:307] RPC server started. Bound to: 127.0.222.1:41951
I20260809 12:10:50.397511 9311 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.222.1:41951 every 8 connection(s)
I20260809 12:10:50.398765 9164 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskgH5Pjm/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786277382073938-888-0/minicluster-data/ts-0/data/info.pb
I20260809 12:10:50.401114 9312 heartbeater.cc:347] Connected to a master server at 127.0.222.62:36963
I20260809 12:10:50.401378 9312 heartbeater.cc:464] Registering TS with master...
I20260809 12:10:50.402045 9312 heartbeater.cc:511] Master 127.0.222.62:36963 requested a full tablet report, sending...
I20260809 12:10:50.404096 9085 ts_manager.cc:194] Registered new tserver with Master: af75f3a57f6f48fba26736ad36aff045 (127.0.222.1:41951)
I20260809 12:10:50.404727 888 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskgH5Pjm/build/asan/bin/kudu as pid 9164
I20260809 12:10:50.404950 888 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskgH5Pjm/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786277382073938-888-0/minicluster-data/ts-0/wal/instance
I20260809 12:10:50.405797 888 external_mini_cluster.cc:1428] Setting key 63a639439dbdf08d2b31be7f3ce7d126
I20260809 12:10:50.406325 9085 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.0.222.1@KRBTEST.COM'} at 127.0.222.1:37401
WARNING: no policy specified for oryx/127.0.222.2@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.0.222.2@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.0.222.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskgH5Pjm/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786277382073938-888-0/krb5kdc/oryx_127.0.222.2.keytab.
Entry for principal oryx/127.0.222.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskgH5Pjm/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786277382073938-888-0/krb5kdc/oryx_127.0.222.2.keytab.
I20260809 12:10:50.466041 888 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskgH5Pjm/build/asan/bin/kudu
/tmp/dist-test-taskgH5Pjm/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskgH5Pjm/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786277382073938-888-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskgH5Pjm/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786277382073938-888-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskgH5Pjm/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786277382073938-888-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskgH5Pjm/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786277382073938-888-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.222.2:0
--local_ip_for_outbound_sockets=127.0.222.2
--webserver_interface=127.0.222.2
--webserver_port=0
--tserver_master_addrs=127.0.222.62:36963
--keytab_file=/tmp/dist-test-taskgH5Pjm/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786277382073938-888-0/krb5kdc/oryx_127.0.222.2.keytab
--principal=oryx/127.0.222.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.222.20:45275
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true with env {KRB5CCNAME=/tmp/dist-test-taskgH5Pjm/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786277382073938-888-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskgH5Pjm/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786277382073938-888-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskgH5Pjm/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786277382073938-888-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260809 12:10:50.810370 9317 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260809 12:10:50.810680 9317 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260809 12:10:50.811033 9317 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260809 12:10:50.819412 9317 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260809 12:10:50.819530 9317 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260809 12:10:50.819715 9317 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260809 12:10:50.819856 9317 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.222.2
I20260809 12:10:50.829875 9317 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.222.20:45275
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskgH5Pjm/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786277382073938-888-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskgH5Pjm/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786277382073938-888-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskgH5Pjm/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786277382073938-888-0/krb5kdc/oryx_127.0.222.2.keytab
--principal=oryx/127.0.222.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.222.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskgH5Pjm/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786277382073938-888-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.222.2
--webserver_port=0
--enable_txn_system_client_init=true
--tserver_master_addrs=127.0.222.62:36963
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.222.2
--log_dir=/tmp/dist-test-taskgH5Pjm/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786277382073938-888-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 09 Aug 2026 11:06:41 UTC on ccycloud.quasar-bwoogy.root.comops.site
build id 4575991
ASAN enabled
I20260809 12:10:50.831436 9317 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260809 12:10:50.832870 9317 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260809 12:10:50.846457 9322 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 12:10:50.846426 9323 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 12:10:50.846513 9325 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260809 12:10:50.847368 9317 server_base.cc:1034] running on GCE node
I20260809 12:10:50.847981 9317 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260809 12:10:50.848893 9317 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260809 12:10:50.850143 9317 hybrid_clock.cc:648] HybridClock initialized: now 1786277450850079 us; error 40 us; skew 500 ppm
I20260809 12:10:50.850468 9317 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 09 12:10:50 dist-test-slave-2w3w krb5kdc[9036](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786277450, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.222.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260809 12:10:50.860096 9317 init.cc:377] Logged in from keytab as oryx/127.0.222.2@KRBTEST.COM (short username oryx)
I20260809 12:10:50.861774 9317 webserver.cc:460] Webserver started at http://127.0.222.2:36419/ using document root <none> and password file <none>
I20260809 12:10:50.862533 9317 fs_manager.cc:359] Metadata directory not provided
I20260809 12:10:50.862627 9317 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260809 12:10:50.863091 9317 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260809 12:10:50.866225 9317 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskgH5Pjm/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786277382073938-888-0/minicluster-data/ts-1/data/instance:
uuid: "cd3a4ffa0f4544a1b74b74fe5b6e3711"
format_stamp: "Formatted at 2026-08-09 12:10:50 on dist-test-slave-2w3w"
server_key: "82559220d60e5daa0fdefa4902967d7d"
server_key_iv: "0e567e9fe7df18093f63b8c7e0e5d59e"
server_key_version: "encryptionkey@0"
I20260809 12:10:50.867149 9317 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskgH5Pjm/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786277382073938-888-0/minicluster-data/ts-1/wal/instance:
uuid: "cd3a4ffa0f4544a1b74b74fe5b6e3711"
format_stamp: "Formatted at 2026-08-09 12:10:50 on dist-test-slave-2w3w"
server_key: "82559220d60e5daa0fdefa4902967d7d"
server_key_iv: "0e567e9fe7df18093f63b8c7e0e5d59e"
server_key_version: "encryptionkey@0"
I20260809 12:10:50.872457 9317 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.000s sys 0.004s
I20260809 12:10:50.875813 9334 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260809 12:10:50.877153 9317 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.001s
I20260809 12:10:50.877278 9317 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskgH5Pjm/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786277382073938-888-0/minicluster-data/ts-1/data,/tmp/dist-test-taskgH5Pjm/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786277382073938-888-0/minicluster-data/ts-1/wal
uuid: "cd3a4ffa0f4544a1b74b74fe5b6e3711"
format_stamp: "Formatted at 2026-08-09 12:10:50 on dist-test-slave-2w3w"
server_key: "82559220d60e5daa0fdefa4902967d7d"
server_key_iv: "0e567e9fe7df18093f63b8c7e0e5d59e"
server_key_version: "encryptionkey@0"
I20260809 12:10:50.877480 9317 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskgH5Pjm/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786277382073938-888-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskgH5Pjm/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786277382073938-888-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskgH5Pjm/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786277382073938-888-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260809 12:10:50.910625 9317 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260809 12:10:50.914898 9317 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260809 12:10:50.915169 9317 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260809 12:10:50.917799 9317 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260809 12:10:50.917937 9317 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260809 12:10:50.918118 9317 ts_tablet_manager.cc:610] Registered 0 tablets
I20260809 12:10:50.918196 9317 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Aug 09 12:10:50 dist-test-slave-2w3w krb5kdc[9036](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786277450, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.222.2@KRBTEST.COM for oryx/127.0.222.62@KRBTEST.COM
I20260809 12:10:50.943982 9343 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 12:10:50.918656 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.222.2:47435 (local address 127.0.222.62:36963)
0809 12:10:50.919239 (+ 583us) server_negotiation.cc:206] Beginning negotiation
0809 12:10:50.919248 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0809 12:10:50.920482 (+ 1234us) server_negotiation.cc:407] Connection header received
0809 12:10:50.922600 (+ 2118us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 12:10:50.922606 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 12:10:50.922697 (+ 91us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 12:10:50.922838 (+ 141us) server_negotiation.cc:226] Negotiated authn=SASL
0809 12:10:50.925798 (+ 2960us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 12:10:50.926850 (+ 1052us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 12:10:50.929621 (+ 2771us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 12:10:50.930356 (+ 735us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 12:10:50.930557 (+ 201us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 12:10:50.935634 (+ 5077us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 12:10:50.935644 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 12:10:50.935652 (+ 8us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0809 12:10:50.935699 (+ 47us) server_negotiation.cc:922] calling sasl_server_start()
0809 12:10:50.939417 (+ 3718us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 12:10:50.940543 (+ 1126us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 12:10:50.940551 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 12:10:50.940557 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0809 12:10:50.940722 (+ 165us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 12:10:50.941438 (+ 716us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 12:10:50.941443 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 12:10:50.941447 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0809 12:10:50.941884 (+ 437us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 12:10:50.942051 (+ 167us) server_negotiation.cc:1035] Waiting for connection context
0809 12:10:50.943320 (+ 1269us) server_negotiation.cc:299] Negotiation successful
0809 12:10:50.943588 (+ 268us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":454,"thread_start_us":334,"threads_started":1}
I20260809 12:10:50.945096 9344 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 12:10:50.919386 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.222.62:36963 (local address 127.0.222.2:47435)
0809 12:10:50.920296 (+ 910us) negotiation.cc:107] Waiting for socket to connect
0809 12:10:50.920343 (+ 47us) client_negotiation.cc:174] Beginning negotiation
0809 12:10:50.922294 (+ 1951us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 12:10:50.923314 (+ 1020us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 12:10:50.923333 (+ 19us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 12:10:50.924341 (+ 1008us) client_negotiation.cc:189] Negotiated authn=SASL
0809 12:10:50.925554 (+ 1213us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 12:10:50.925585 (+ 31us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 12:10:50.927054 (+ 1469us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 12:10:50.927066 (+ 12us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 12:10:50.929413 (+ 2347us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 12:10:50.929426 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 12:10:50.930708 (+ 1282us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 12:10:50.930722 (+ 14us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 12:10:50.930941 (+ 219us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 12:10:50.931797 (+ 856us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0809 12:10:50.931842 (+ 45us) client_negotiation.cc:656] calling sasl_client_start()
0809 12:10:50.935250 (+ 3408us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0809 12:10:50.939582 (+ 4332us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0809 12:10:50.939594 (+ 12us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0809 12:10:50.939613 (+ 19us) client_negotiation.cc:760] calling sasl_client_step()
0809 12:10:50.940329 (+ 716us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0809 12:10:50.940846 (+ 517us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0809 12:10:50.940853 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0809 12:10:50.940864 (+ 11us) client_negotiation.cc:760] calling sasl_client_step()
0809 12:10:50.941295 (+ 431us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0809 12:10:50.942022 (+ 727us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0809 12:10:50.942030 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0809 12:10:50.942642 (+ 612us) client_negotiation.cc:769] Sending connection context
0809 12:10:50.943196 (+ 554us) client_negotiation.cc:240] Negotiation successful
0809 12:10:50.943807 (+ 611us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":618,"thread_start_us":398,"threads_started":1}
I20260809 12:10:50.969506 9317 rpc_server.cc:307] RPC server started. Bound to: 127.0.222.2:43017
I20260809 12:10:50.969604 9462 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.222.2:43017 every 8 connection(s)
I20260809 12:10:50.970889 9317 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskgH5Pjm/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786277382073938-888-0/minicluster-data/ts-1/data/info.pb
I20260809 12:10:50.971565 9343 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 12:10:50.953515 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.222.2:39935 (local address 127.0.222.62:36963)
0809 12:10:50.953978 (+ 463us) server_negotiation.cc:206] Beginning negotiation
0809 12:10:50.953984 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0809 12:10:50.954342 (+ 358us) server_negotiation.cc:407] Connection header received
0809 12:10:50.954592 (+ 250us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 12:10:50.954601 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 12:10:50.954697 (+ 96us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 12:10:50.954831 (+ 134us) server_negotiation.cc:226] Negotiated authn=SASL
0809 12:10:50.956438 (+ 1607us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 12:10:50.957748 (+ 1310us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 12:10:50.960190 (+ 2442us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 12:10:50.960761 (+ 571us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 12:10:50.960981 (+ 220us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 12:10:50.964320 (+ 3339us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 12:10:50.964329 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 12:10:50.964337 (+ 8us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0809 12:10:50.964389 (+ 52us) server_negotiation.cc:922] calling sasl_server_start()
0809 12:10:50.968264 (+ 3875us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 12:10:50.969269 (+ 1005us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 12:10:50.969278 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 12:10:50.969285 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0809 12:10:50.969472 (+ 187us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 12:10:50.969954 (+ 482us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 12:10:50.969959 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 12:10:50.969963 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0809 12:10:50.970306 (+ 343us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 12:10:50.970469 (+ 163us) server_negotiation.cc:1035] Waiting for connection context
0809 12:10:50.970949 (+ 480us) server_negotiation.cc:299] Negotiation successful
0809 12:10:50.971215 (+ 266us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":308}
I20260809 12:10:50.971693 9425 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 12:10:50.953543 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.222.62:36963 (local address 127.0.222.2:39935)
0809 12:10:50.954220 (+ 677us) negotiation.cc:107] Waiting for socket to connect
0809 12:10:50.954242 (+ 22us) client_negotiation.cc:174] Beginning negotiation
0809 12:10:50.954408 (+ 166us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 12:10:50.954883 (+ 475us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 12:10:50.954890 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 12:10:50.955705 (+ 815us) client_negotiation.cc:189] Negotiated authn=SASL
0809 12:10:50.956250 (+ 545us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 12:10:50.956267 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 12:10:50.957952 (+ 1685us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 12:10:50.957957 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 12:10:50.959985 (+ 2028us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 12:10:50.960007 (+ 22us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 12:10:50.961066 (+ 1059us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 12:10:50.961075 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 12:10:50.961365 (+ 290us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 12:10:50.962238 (+ 873us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0809 12:10:50.962271 (+ 33us) client_negotiation.cc:656] calling sasl_client_start()
0809 12:10:50.964057 (+ 1786us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0809 12:10:50.968413 (+ 4356us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0809 12:10:50.968420 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0809 12:10:50.968427 (+ 7us) client_negotiation.cc:760] calling sasl_client_step()
0809 12:10:50.969126 (+ 699us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0809 12:10:50.969633 (+ 507us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0809 12:10:50.969641 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0809 12:10:50.969647 (+ 6us) client_negotiation.cc:760] calling sasl_client_step()
0809 12:10:50.969811 (+ 164us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0809 12:10:50.970460 (+ 649us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0809 12:10:50.970464 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0809 12:10:50.970666 (+ 202us) client_negotiation.cc:769] Sending connection context
0809 12:10:50.970824 (+ 158us) client_negotiation.cc:240] Negotiation successful
0809 12:10:50.971215 (+ 391us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":533,"thread_start_us":448,"threads_started":1}
I20260809 12:10:50.973798 9463 heartbeater.cc:347] Connected to a master server at 127.0.222.62:36963
I20260809 12:10:50.974040 888 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskgH5Pjm/build/asan/bin/kudu as pid 9317
I20260809 12:10:50.974088 9463 heartbeater.cc:464] Registering TS with master...
I20260809 12:10:50.974229 888 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskgH5Pjm/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786277382073938-888-0/minicluster-data/ts-1/wal/instance
I20260809 12:10:50.974860 888 external_mini_cluster.cc:1428] Setting key a87fb80afc24778025f4d06328bc5757
I20260809 12:10:50.975016 9463 heartbeater.cc:511] Master 127.0.222.62:36963 requested a full tablet report, sending...
I20260809 12:10:50.976570 9085 ts_manager.cc:194] Registered new tserver with Master: cd3a4ffa0f4544a1b74b74fe5b6e3711 (127.0.222.2:43017)
I20260809 12:10:50.977424 9085 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.0.222.2@KRBTEST.COM'} at 127.0.222.2:47435
WARNING: no policy specified for oryx/127.0.222.3@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.0.222.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.0.222.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskgH5Pjm/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786277382073938-888-0/krb5kdc/oryx_127.0.222.3.keytab.
Entry for principal oryx/127.0.222.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskgH5Pjm/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786277382073938-888-0/krb5kdc/oryx_127.0.222.3.keytab.
I20260809 12:10:51.036096 888 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskgH5Pjm/build/asan/bin/kudu
/tmp/dist-test-taskgH5Pjm/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskgH5Pjm/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786277382073938-888-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskgH5Pjm/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786277382073938-888-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskgH5Pjm/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786277382073938-888-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskgH5Pjm/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786277382073938-888-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.222.3:0
--local_ip_for_outbound_sockets=127.0.222.3
--webserver_interface=127.0.222.3
--webserver_port=0
--tserver_master_addrs=127.0.222.62:36963
--keytab_file=/tmp/dist-test-taskgH5Pjm/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786277382073938-888-0/krb5kdc/oryx_127.0.222.3.keytab
--principal=oryx/127.0.222.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.222.20:45275
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true with env {KRB5CCNAME=/tmp/dist-test-taskgH5Pjm/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786277382073938-888-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskgH5Pjm/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786277382073938-888-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskgH5Pjm/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786277382073938-888-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260809 12:10:51.382885 9468 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260809 12:10:51.383283 9468 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260809 12:10:51.383463 9468 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260809 12:10:51.390921 9468 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260809 12:10:51.391145 9468 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260809 12:10:51.391309 9468 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260809 12:10:51.391454 9468 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.222.3
I20260809 12:10:51.404199 9468 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.222.20:45275
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskgH5Pjm/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786277382073938-888-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskgH5Pjm/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786277382073938-888-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskgH5Pjm/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786277382073938-888-0/krb5kdc/oryx_127.0.222.3.keytab
--principal=oryx/127.0.222.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.222.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskgH5Pjm/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786277382073938-888-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.222.3
--webserver_port=0
--enable_txn_system_client_init=true
--tserver_master_addrs=127.0.222.62:36963
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.222.3
--log_dir=/tmp/dist-test-taskgH5Pjm/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786277382073938-888-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 09 Aug 2026 11:06:41 UTC on ccycloud.quasar-bwoogy.root.comops.site
build id 4575991
ASAN enabled
I20260809 12:10:51.405423 9468 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260809 12:10:51.406793 9468 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260809 12:10:51.409715 9312 heartbeater.cc:503] Master 127.0.222.62:36963 was elected leader, sending a full tablet report...
W20260809 12:10:51.420615 9473 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 12:10:51.420641 9476 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 12:10:51.420675 9474 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260809 12:10:51.421550 9468 server_base.cc:1034] running on GCE node
I20260809 12:10:51.422215 9468 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260809 12:10:51.423233 9468 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260809 12:10:51.424494 9468 hybrid_clock.cc:648] HybridClock initialized: now 1786277451424418 us; error 42 us; skew 500 ppm
I20260809 12:10:51.424878 9468 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 09 12:10:51 dist-test-slave-2w3w krb5kdc[9036](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786277451, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.222.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260809 12:10:51.428258 9468 init.cc:377] Logged in from keytab as oryx/127.0.222.3@KRBTEST.COM (short username oryx)
I20260809 12:10:51.429718 9468 webserver.cc:460] Webserver started at http://127.0.222.3:34125/ using document root <none> and password file <none>
I20260809 12:10:51.430460 9468 fs_manager.cc:359] Metadata directory not provided
I20260809 12:10:51.430580 9468 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260809 12:10:51.430941 9468 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260809 12:10:51.434170 9468 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskgH5Pjm/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786277382073938-888-0/minicluster-data/ts-2/data/instance:
uuid: "040356c8db9f4f8e93ae96f8fcaecf29"
format_stamp: "Formatted at 2026-08-09 12:10:51 on dist-test-slave-2w3w"
server_key: "5f092ba569bdb6f2a4149c046aa1e07f"
server_key_iv: "9ceae9c579fdafdf6923d8bdd21c13e8"
server_key_version: "encryptionkey@0"
I20260809 12:10:51.435066 9468 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskgH5Pjm/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786277382073938-888-0/minicluster-data/ts-2/wal/instance:
uuid: "040356c8db9f4f8e93ae96f8fcaecf29"
format_stamp: "Formatted at 2026-08-09 12:10:51 on dist-test-slave-2w3w"
server_key: "5f092ba569bdb6f2a4149c046aa1e07f"
server_key_iv: "9ceae9c579fdafdf6923d8bdd21c13e8"
server_key_version: "encryptionkey@0"
I20260809 12:10:51.440333 9468 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.004s sys 0.000s
I20260809 12:10:51.444168 9484 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260809 12:10:51.445536 9468 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.001s
I20260809 12:10:51.445720 9468 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskgH5Pjm/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786277382073938-888-0/minicluster-data/ts-2/data,/tmp/dist-test-taskgH5Pjm/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786277382073938-888-0/minicluster-data/ts-2/wal
uuid: "040356c8db9f4f8e93ae96f8fcaecf29"
format_stamp: "Formatted at 2026-08-09 12:10:51 on dist-test-slave-2w3w"
server_key: "5f092ba569bdb6f2a4149c046aa1e07f"
server_key_iv: "9ceae9c579fdafdf6923d8bdd21c13e8"
server_key_version: "encryptionkey@0"
I20260809 12:10:51.446003 9468 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskgH5Pjm/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786277382073938-888-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskgH5Pjm/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786277382073938-888-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskgH5Pjm/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786277382073938-888-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260809 12:10:51.468832 9468 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260809 12:10:51.473372 9468 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260809 12:10:51.473659 9468 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260809 12:10:51.476524 9468 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260809 12:10:51.476634 9468 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260809 12:10:51.476842 9468 ts_tablet_manager.cc:610] Registered 0 tablets
I20260809 12:10:51.476944 9468 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Aug 09 12:10:51 dist-test-slave-2w3w krb5kdc[9036](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786277451, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.222.3@KRBTEST.COM for oryx/127.0.222.62@KRBTEST.COM
I20260809 12:10:51.505374 9493 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 12:10:51.477022 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.222.3:54153 (local address 127.0.222.62:36963)
0809 12:10:51.480619 (+ 3597us) server_negotiation.cc:206] Beginning negotiation
0809 12:10:51.480627 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0809 12:10:51.480652 (+ 25us) server_negotiation.cc:407] Connection header received
0809 12:10:51.482364 (+ 1712us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 12:10:51.482370 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 12:10:51.482461 (+ 91us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 12:10:51.482580 (+ 119us) server_negotiation.cc:226] Negotiated authn=SASL
0809 12:10:51.484794 (+ 2214us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 12:10:51.486396 (+ 1602us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 12:10:51.490055 (+ 3659us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 12:10:51.490807 (+ 752us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 12:10:51.490932 (+ 125us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 12:10:51.496829 (+ 5897us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 12:10:51.496839 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 12:10:51.496844 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0809 12:10:51.496888 (+ 44us) server_negotiation.cc:922] calling sasl_server_start()
0809 12:10:51.501059 (+ 4171us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 12:10:51.502077 (+ 1018us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 12:10:51.502086 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 12:10:51.502092 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0809 12:10:51.502287 (+ 195us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 12:10:51.503060 (+ 773us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 12:10:51.503067 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 12:10:51.503073 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0809 12:10:51.503516 (+ 443us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 12:10:51.503704 (+ 188us) server_negotiation.cc:1035] Waiting for connection context
0809 12:10:51.504461 (+ 757us) server_negotiation.cc:299] Negotiation successful
0809 12:10:51.504787 (+ 326us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":3420,"thread_start_us":3244,"threads_started":1}
I20260809 12:10:51.505837 9494 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 12:10:51.477620 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.222.62:36963 (local address 127.0.222.3:54153)
0809 12:10:51.479785 (+ 2165us) negotiation.cc:107] Waiting for socket to connect
0809 12:10:51.479868 (+ 83us) client_negotiation.cc:174] Beginning negotiation
0809 12:10:51.482054 (+ 2186us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 12:10:51.482876 (+ 822us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 12:10:51.482890 (+ 14us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 12:10:51.483761 (+ 871us) client_negotiation.cc:189] Negotiated authn=SASL
0809 12:10:51.484581 (+ 820us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 12:10:51.484601 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 12:10:51.486701 (+ 2100us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 12:10:51.486710 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 12:10:51.489821 (+ 3111us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 12:10:51.489843 (+ 22us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 12:10:51.491360 (+ 1517us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 12:10:51.491372 (+ 12us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 12:10:51.491642 (+ 270us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 12:10:51.492560 (+ 918us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0809 12:10:51.492592 (+ 32us) client_negotiation.cc:656] calling sasl_client_start()
0809 12:10:51.496450 (+ 3858us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0809 12:10:51.501236 (+ 4786us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0809 12:10:51.501247 (+ 11us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0809 12:10:51.501270 (+ 23us) client_negotiation.cc:760] calling sasl_client_step()
0809 12:10:51.501904 (+ 634us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0809 12:10:51.502480 (+ 576us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0809 12:10:51.502492 (+ 12us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0809 12:10:51.502498 (+ 6us) client_negotiation.cc:760] calling sasl_client_step()
0809 12:10:51.502850 (+ 352us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0809 12:10:51.503671 (+ 821us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0809 12:10:51.503679 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0809 12:10:51.504037 (+ 358us) client_negotiation.cc:769] Sending connection context
0809 12:10:51.504269 (+ 232us) client_negotiation.cc:240] Negotiation successful
0809 12:10:51.504669 (+ 400us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1767,"thread_start_us":1489,"threads_started":1}
I20260809 12:10:51.529910 9574 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 12:10:51.514700 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.222.62:36963 (local address 127.0.222.3:47139)
0809 12:10:51.515712 (+ 1012us) negotiation.cc:107] Waiting for socket to connect
0809 12:10:51.515740 (+ 28us) client_negotiation.cc:174] Beginning negotiation
0809 12:10:51.515924 (+ 184us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 12:10:51.516421 (+ 497us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 12:10:51.516426 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 12:10:51.517210 (+ 784us) client_negotiation.cc:189] Negotiated authn=SASL
0809 12:10:51.517775 (+ 565us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 12:10:51.517794 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 12:10:51.519137 (+ 1343us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 12:10:51.519143 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 12:10:51.520730 (+ 1587us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 12:10:51.520746 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 12:10:51.521760 (+ 1014us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 12:10:51.521774 (+ 14us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 12:10:51.522048 (+ 274us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 12:10:51.522841 (+ 793us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0809 12:10:51.522860 (+ 19us) client_negotiation.cc:656] calling sasl_client_start()
0809 12:10:51.523719 (+ 859us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0809 12:10:51.526987 (+ 3268us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0809 12:10:51.526995 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0809 12:10:51.527000 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0809 12:10:51.527451 (+ 451us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0809 12:10:51.527867 (+ 416us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0809 12:10:51.527872 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0809 12:10:51.527876 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0809 12:10:51.528009 (+ 133us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0809 12:10:51.528562 (+ 553us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0809 12:10:51.528566 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0809 12:10:51.528849 (+ 283us) client_negotiation.cc:769] Sending connection context
0809 12:10:51.529028 (+ 179us) client_negotiation.cc:240] Negotiation successful
0809 12:10:51.529354 (+ 326us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":800,"thread_start_us":597,"threads_started":1}
I20260809 12:10:51.530864 9493 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 12:10:51.514705 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.222.3:47139 (local address 127.0.222.62:36963)
0809 12:10:51.514944 (+ 239us) server_negotiation.cc:206] Beginning negotiation
0809 12:10:51.514953 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0809 12:10:51.515861 (+ 908us) server_negotiation.cc:407] Connection header received
0809 12:10:51.516172 (+ 311us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 12:10:51.516181 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 12:10:51.516276 (+ 95us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 12:10:51.516391 (+ 115us) server_negotiation.cc:226] Negotiated authn=SASL
0809 12:10:51.517994 (+ 1603us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 12:10:51.518935 (+ 941us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 12:10:51.520901 (+ 1966us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 12:10:51.521539 (+ 638us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 12:10:51.521703 (+ 164us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 12:10:51.523941 (+ 2238us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 12:10:51.523946 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 12:10:51.523949 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0809 12:10:51.523983 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0809 12:10:51.526829 (+ 2846us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 12:10:51.527581 (+ 752us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 12:10:51.527588 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 12:10:51.527592 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0809 12:10:51.527730 (+ 138us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 12:10:51.528132 (+ 402us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 12:10:51.528139 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 12:10:51.528146 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0809 12:10:51.528426 (+ 280us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 12:10:51.528599 (+ 173us) server_negotiation.cc:1035] Waiting for connection context
0809 12:10:51.530313 (+ 1714us) server_negotiation.cc:299] Negotiation successful
0809 12:10:51.530583 (+ 270us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":66}
I20260809 12:10:51.533895 9468 rpc_server.cc:307] RPC server started. Bound to: 127.0.222.3:33697
I20260809 12:10:51.533949 9610 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.222.3:33697 every 8 connection(s)
I20260809 12:10:51.535457 9468 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskgH5Pjm/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786277382073938-888-0/minicluster-data/ts-2/data/info.pb
I20260809 12:10:51.537511 9611 heartbeater.cc:347] Connected to a master server at 127.0.222.62:36963
I20260809 12:10:51.537782 9611 heartbeater.cc:464] Registering TS with master...
I20260809 12:10:51.538442 9611 heartbeater.cc:511] Master 127.0.222.62:36963 requested a full tablet report, sending...
I20260809 12:10:51.539892 9085 ts_manager.cc:194] Registered new tserver with Master: 040356c8db9f4f8e93ae96f8fcaecf29 (127.0.222.3:33697)
I20260809 12:10:51.540758 9085 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.0.222.3@KRBTEST.COM'} at 127.0.222.3:54153
I20260809 12:10:51.543350 888 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskgH5Pjm/build/asan/bin/kudu as pid 9468
I20260809 12:10:51.543537 888 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskgH5Pjm/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786277382073938-888-0/minicluster-data/ts-2/wal/instance
I20260809 12:10:51.543987 888 external_mini_cluster.cc:1428] Setting key 7523018f43979cd88e3eb62e408bca55
I20260809 12:10:51.546857 888 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
Aug 09 12:10:51 dist-test-slave-2w3w krb5kdc[9036](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786277451, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-user@KRBTEST.COM:
Aug 09 12:10:51 dist-test-slave-2w3w krb5kdc[9036](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, test-user@KRBTEST.COM for kudu/127.0.222.62@KRBTEST.COM, Server not found in Kerberos database
Aug 09 12:10:51 dist-test-slave-2w3w krb5kdc[9036](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, test-user@KRBTEST.COM for kudu/127.0.222.62@KRBTEST.COM, Server not found in Kerberos database
W20260809 12:10:51.608254 9493 negotiation.cc:337] Failed RPC negotiation. Trace:
0809 12:10:51.597637 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53616 (local address 127.0.222.62:36963)
0809 12:10:51.597837 (+ 200us) server_negotiation.cc:206] Beginning negotiation
0809 12:10:51.597841 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0809 12:10:51.598084 (+ 243us) server_negotiation.cc:407] Connection header received
0809 12:10:51.598247 (+ 163us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 12:10:51.598250 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 12:10:51.598341 (+ 91us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 12:10:51.598439 (+ 98us) server_negotiation.cc:226] Negotiated authn=SASL
0809 12:10:51.599991 (+ 1552us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 12:10:51.600711 (+ 720us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 12:10:51.601903 (+ 1192us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 12:10:51.602322 (+ 419us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 12:10:51.602415 (+ 93us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 12:10:51.607970 (+ 5555us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:53616: BlockingRecv error: recv got EOF from 127.0.0.1:53616 (error 108)
Metrics: {"server-negotiator.queue_time_us":71}
Aug 09 12:10:51 dist-test-slave-2w3w krb5kdc[9036](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786277451, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for oryx/127.0.222.62@KRBTEST.COM
I20260809 12:10:51.624994 9493 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 12:10:51.612223 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53632 (local address 127.0.222.62:36963)
0809 12:10:51.612440 (+ 217us) server_negotiation.cc:206] Beginning negotiation
0809 12:10:51.612449 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0809 12:10:51.612628 (+ 179us) server_negotiation.cc:407] Connection header received
0809 12:10:51.612746 (+ 118us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 12:10:51.612749 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 12:10:51.612811 (+ 62us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 12:10:51.612896 (+ 85us) server_negotiation.cc:226] Negotiated authn=SASL
0809 12:10:51.613993 (+ 1097us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 12:10:51.614730 (+ 737us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 12:10:51.616015 (+ 1285us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 12:10:51.616432 (+ 417us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 12:10:51.616531 (+ 99us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 12:10:51.620271 (+ 3740us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 12:10:51.620277 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 12:10:51.620280 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0809 12:10:51.620315 (+ 35us) server_negotiation.cc:922] calling sasl_server_start()
0809 12:10:51.622671 (+ 2356us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 12:10:51.623356 (+ 685us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 12:10:51.623360 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 12:10:51.623362 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0809 12:10:51.623484 (+ 122us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 12:10:51.623878 (+ 394us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 12:10:51.623881 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 12:10:51.623883 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0809 12:10:51.624141 (+ 258us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 12:10:51.624335 (+ 194us) server_negotiation.cc:1035] Waiting for connection context
0809 12:10:51.624579 (+ 244us) server_negotiation.cc:299] Negotiation successful
0809 12:10:51.624766 (+ 187us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":80}
I20260809 12:10:51.628859 9085 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:53632:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260809 12:10:51.636054 9085 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260809 12:10:51.671792 9636 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 12:10:51.648724 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.222.3:33697 (local address 127.0.0.1:53446)
0809 12:10:51.661513 (+ 12789us) negotiation.cc:107] Waiting for socket to connect
0809 12:10:51.661539 (+ 26us) client_negotiation.cc:174] Beginning negotiation
0809 12:10:51.662043 (+ 504us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 12:10:51.663511 (+ 1468us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 12:10:51.663518 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 12:10:51.663548 (+ 30us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0809 12:10:51.663972 (+ 424us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 12:10:51.663984 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 12:10:51.665976 (+ 1992us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 12:10:51.665983 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 12:10:51.668590 (+ 2607us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 12:10:51.668610 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 12:10:51.670601 (+ 1991us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 12:10:51.670609 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 12:10:51.670829 (+ 220us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 12:10:51.670895 (+ 66us) client_negotiation.cc:769] Sending connection context
0809 12:10:51.671082 (+ 187us) client_negotiation.cc:240] Negotiation successful
0809 12:10:51.671340 (+ 258us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":12612,"mutex_wait_us":7,"thread_start_us":15635,"threads_started":1}
I20260809 12:10:51.673228 9635 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 12:10:51.649637 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53446 (local address 127.0.222.3:33697)
0809 12:10:51.656767 (+ 7130us) server_negotiation.cc:206] Beginning negotiation
0809 12:10:51.656781 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0809 12:10:51.661651 (+ 4870us) server_negotiation.cc:407] Connection header received
0809 12:10:51.662378 (+ 727us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 12:10:51.662393 (+ 15us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 12:10:51.662756 (+ 363us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 12:10:51.663105 (+ 349us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0809 12:10:51.664192 (+ 1087us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 12:10:51.665830 (+ 1638us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 12:10:51.668794 (+ 2964us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 12:10:51.670356 (+ 1562us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 12:10:51.670497 (+ 141us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 12:10:51.670735 (+ 238us) server_negotiation.cc:1035] Waiting for connection context
0809 12:10:51.672683 (+ 1948us) server_negotiation.cc:299] Negotiation successful
0809 12:10:51.672881 (+ 198us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":6891,"thread_start_us":968,"threads_started":1}
I20260809 12:10:51.673393 9637 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 12:10:51.650659 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49628 (local address 127.0.222.1:41951)
0809 12:10:51.663397 (+ 12738us) server_negotiation.cc:206] Beginning negotiation
0809 12:10:51.663406 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0809 12:10:51.664870 (+ 1464us) server_negotiation.cc:407] Connection header received
0809 12:10:51.664994 (+ 124us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 12:10:51.665001 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 12:10:51.665489 (+ 488us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 12:10:51.665723 (+ 234us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0809 12:10:51.666754 (+ 1031us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 12:10:51.668460 (+ 1706us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 12:10:51.671113 (+ 2653us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 12:10:51.672210 (+ 1097us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 12:10:51.672367 (+ 157us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 12:10:51.672577 (+ 210us) server_negotiation.cc:1035] Waiting for connection context
0809 12:10:51.672850 (+ 273us) server_negotiation.cc:299] Negotiation successful
0809 12:10:51.673049 (+ 199us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":12551,"thread_start_us":12741,"threads_started":1}
I20260809 12:10:51.673290 9633 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 12:10:51.650427 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.222.1:41951 (local address 127.0.0.1:49628)
0809 12:10:51.664657 (+ 14230us) negotiation.cc:107] Waiting for socket to connect
0809 12:10:51.664676 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0809 12:10:51.664815 (+ 139us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 12:10:51.665958 (+ 1143us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 12:10:51.665963 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 12:10:51.665991 (+ 28us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0809 12:10:51.666534 (+ 543us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 12:10:51.666551 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 12:10:51.668663 (+ 2112us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 12:10:51.668670 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 12:10:51.670925 (+ 2255us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 12:10:51.670937 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 12:10:51.672453 (+ 1516us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 12:10:51.672460 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 12:10:51.672632 (+ 172us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 12:10:51.672663 (+ 31us) client_negotiation.cc:769] Sending connection context
0809 12:10:51.672763 (+ 100us) client_negotiation.cc:240] Negotiation successful
0809 12:10:51.672895 (+ 132us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":14101,"mutex_wait_us":29,"thread_start_us":11293,"threads_started":1}
I20260809 12:10:51.674095 9632 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 12:10:51.648476 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.222.2:43017 (local address 127.0.0.1:40518)
0809 12:10:51.649431 (+ 955us) negotiation.cc:107] Waiting for socket to connect
0809 12:10:51.649484 (+ 53us) client_negotiation.cc:174] Beginning negotiation
0809 12:10:51.649671 (+ 187us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 12:10:51.663693 (+ 14022us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 12:10:51.663702 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 12:10:51.663749 (+ 47us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0809 12:10:51.664515 (+ 766us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 12:10:51.664533 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 12:10:51.667338 (+ 2805us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 12:10:51.667346 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 12:10:51.669666 (+ 2320us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 12:10:51.669684 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 12:10:51.672673 (+ 2989us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 12:10:51.672681 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 12:10:51.672905 (+ 224us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 12:10:51.672939 (+ 34us) client_negotiation.cc:769] Sending connection context
0809 12:10:51.673015 (+ 76us) client_negotiation.cc:240] Negotiation successful
0809 12:10:51.673152 (+ 137us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":630,"mutex_wait_us":1,"thread_start_us":816,"threads_started":1}
I20260809 12:10:51.674260 9634 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 12:10:51.649091 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40518 (local address 127.0.222.2:43017)
0809 12:10:51.662619 (+ 13528us) server_negotiation.cc:206] Beginning negotiation
0809 12:10:51.662633 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0809 12:10:51.662673 (+ 40us) server_negotiation.cc:407] Connection header received
0809 12:10:51.662906 (+ 233us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 12:10:51.662920 (+ 14us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 12:10:51.663366 (+ 446us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 12:10:51.664739 (+ 1373us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0809 12:10:51.665395 (+ 656us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 12:10:51.667197 (+ 1802us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 12:10:51.669845 (+ 2648us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 12:10:51.671162 (+ 1317us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 12:10:51.673400 (+ 2238us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 12:10:51.673622 (+ 222us) server_negotiation.cc:1035] Waiting for connection context
0809 12:10:51.673705 (+ 83us) server_negotiation.cc:299] Negotiation successful
0809 12:10:51.673916 (+ 211us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":13301,"thread_start_us":12882,"threads_started":1}
I20260809 12:10:51.677449 9235 tablet_service.cc:1467] Processing CreateTablet for tablet f360476a491744e092adc54e8e38a51e (DEFAULT_TABLE table=test-table [id=d1f6e99d50a44c74a2715f16cbb78778]), partition=RANGE (key) PARTITION UNBOUNDED
I20260809 12:10:51.677561 9537 tablet_service.cc:1467] Processing CreateTablet for tablet f360476a491744e092adc54e8e38a51e (DEFAULT_TABLE table=test-table [id=d1f6e99d50a44c74a2715f16cbb78778]), partition=RANGE (key) PARTITION UNBOUNDED
I20260809 12:10:51.677561 9386 tablet_service.cc:1467] Processing CreateTablet for tablet f360476a491744e092adc54e8e38a51e (DEFAULT_TABLE table=test-table [id=d1f6e99d50a44c74a2715f16cbb78778]), partition=RANGE (key) PARTITION UNBOUNDED
I20260809 12:10:51.678799 9235 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f360476a491744e092adc54e8e38a51e. 1 dirs total, 0 dirs full, 0 dirs failed
I20260809 12:10:51.678906 9537 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f360476a491744e092adc54e8e38a51e. 1 dirs total, 0 dirs full, 0 dirs failed
I20260809 12:10:51.678905 9386 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f360476a491744e092adc54e8e38a51e. 1 dirs total, 0 dirs full, 0 dirs failed
I20260809 12:10:51.692569 9638 tablet_bootstrap.cc:492] T f360476a491744e092adc54e8e38a51e P af75f3a57f6f48fba26736ad36aff045: Bootstrap starting.
I20260809 12:10:51.694619 9640 tablet_bootstrap.cc:492] T f360476a491744e092adc54e8e38a51e P 040356c8db9f4f8e93ae96f8fcaecf29: Bootstrap starting.
I20260809 12:10:51.696740 9638 tablet_bootstrap.cc:654] T f360476a491744e092adc54e8e38a51e P af75f3a57f6f48fba26736ad36aff045: Neither blocks nor log segments found. Creating new log.
I20260809 12:10:51.698153 9638 log.cc:824] T f360476a491744e092adc54e8e38a51e P af75f3a57f6f48fba26736ad36aff045: Log is configured to *not* fsync() on all Append() calls
I20260809 12:10:51.698319 9640 tablet_bootstrap.cc:654] T f360476a491744e092adc54e8e38a51e P 040356c8db9f4f8e93ae96f8fcaecf29: Neither blocks nor log segments found. Creating new log.
I20260809 12:10:51.699841 9640 log.cc:824] T f360476a491744e092adc54e8e38a51e P 040356c8db9f4f8e93ae96f8fcaecf29: Log is configured to *not* fsync() on all Append() calls
I20260809 12:10:51.699990 9639 tablet_bootstrap.cc:492] T f360476a491744e092adc54e8e38a51e P cd3a4ffa0f4544a1b74b74fe5b6e3711: Bootstrap starting.
I20260809 12:10:51.703603 9640 tablet_bootstrap.cc:492] T f360476a491744e092adc54e8e38a51e P 040356c8db9f4f8e93ae96f8fcaecf29: No bootstrap required, opened a new log
I20260809 12:10:51.702950 9639 tablet_bootstrap.cc:654] T f360476a491744e092adc54e8e38a51e P cd3a4ffa0f4544a1b74b74fe5b6e3711: Neither blocks nor log segments found. Creating new log.
I20260809 12:10:51.703929 9640 ts_tablet_manager.cc:1397] T f360476a491744e092adc54e8e38a51e P 040356c8db9f4f8e93ae96f8fcaecf29: Time spent bootstrapping tablet: real 0.010s user 0.005s sys 0.003s
I20260809 12:10:51.706250 9638 tablet_bootstrap.cc:492] T f360476a491744e092adc54e8e38a51e P af75f3a57f6f48fba26736ad36aff045: No bootstrap required, opened a new log
I20260809 12:10:51.706547 9638 ts_tablet_manager.cc:1397] T f360476a491744e092adc54e8e38a51e P af75f3a57f6f48fba26736ad36aff045: Time spent bootstrapping tablet: real 0.014s user 0.004s sys 0.009s
I20260809 12:10:51.707834 9639 log.cc:824] T f360476a491744e092adc54e8e38a51e P cd3a4ffa0f4544a1b74b74fe5b6e3711: Log is configured to *not* fsync() on all Append() calls
I20260809 12:10:51.711468 9639 tablet_bootstrap.cc:492] T f360476a491744e092adc54e8e38a51e P cd3a4ffa0f4544a1b74b74fe5b6e3711: No bootstrap required, opened a new log
I20260809 12:10:51.711758 9639 ts_tablet_manager.cc:1397] T f360476a491744e092adc54e8e38a51e P cd3a4ffa0f4544a1b74b74fe5b6e3711: Time spent bootstrapping tablet: real 0.012s user 0.007s sys 0.000s
I20260809 12:10:51.715214 9640 raft_consensus.cc:348] T f360476a491744e092adc54e8e38a51e P 040356c8db9f4f8e93ae96f8fcaecf29 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cd3a4ffa0f4544a1b74b74fe5b6e3711" member_type: VOTER last_known_addr { host: "127.0.222.2" port: 43017 } } peers { permanent_uuid: "040356c8db9f4f8e93ae96f8fcaecf29" member_type: VOTER last_known_addr { host: "127.0.222.3" port: 33697 } } peers { permanent_uuid: "af75f3a57f6f48fba26736ad36aff045" member_type: VOTER last_known_addr { host: "127.0.222.1" port: 41951 } }
I20260809 12:10:51.715703 9640 raft_consensus.cc:374] T f360476a491744e092adc54e8e38a51e P 040356c8db9f4f8e93ae96f8fcaecf29 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260809 12:10:51.715818 9640 raft_consensus.cc:729] T f360476a491744e092adc54e8e38a51e P 040356c8db9f4f8e93ae96f8fcaecf29 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 040356c8db9f4f8e93ae96f8fcaecf29, State: Initialized, Role: FOLLOWER
I20260809 12:10:51.716609 9640 consensus_queue.cc:260] T f360476a491744e092adc54e8e38a51e P 040356c8db9f4f8e93ae96f8fcaecf29 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cd3a4ffa0f4544a1b74b74fe5b6e3711" member_type: VOTER last_known_addr { host: "127.0.222.2" port: 43017 } } peers { permanent_uuid: "040356c8db9f4f8e93ae96f8fcaecf29" member_type: VOTER last_known_addr { host: "127.0.222.3" port: 33697 } } peers { permanent_uuid: "af75f3a57f6f48fba26736ad36aff045" member_type: VOTER last_known_addr { host: "127.0.222.1" port: 41951 } }
I20260809 12:10:51.717830 9638 raft_consensus.cc:348] T f360476a491744e092adc54e8e38a51e P af75f3a57f6f48fba26736ad36aff045 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cd3a4ffa0f4544a1b74b74fe5b6e3711" member_type: VOTER last_known_addr { host: "127.0.222.2" port: 43017 } } peers { permanent_uuid: "040356c8db9f4f8e93ae96f8fcaecf29" member_type: VOTER last_known_addr { host: "127.0.222.3" port: 33697 } } peers { permanent_uuid: "af75f3a57f6f48fba26736ad36aff045" member_type: VOTER last_known_addr { host: "127.0.222.1" port: 41951 } }
I20260809 12:10:51.718391 9638 raft_consensus.cc:374] T f360476a491744e092adc54e8e38a51e P af75f3a57f6f48fba26736ad36aff045 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260809 12:10:51.718513 9638 raft_consensus.cc:729] T f360476a491744e092adc54e8e38a51e P af75f3a57f6f48fba26736ad36aff045 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: af75f3a57f6f48fba26736ad36aff045, State: Initialized, Role: FOLLOWER
I20260809 12:10:51.719575 9638 consensus_queue.cc:260] T f360476a491744e092adc54e8e38a51e P af75f3a57f6f48fba26736ad36aff045 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cd3a4ffa0f4544a1b74b74fe5b6e3711" member_type: VOTER last_known_addr { host: "127.0.222.2" port: 43017 } } peers { permanent_uuid: "040356c8db9f4f8e93ae96f8fcaecf29" member_type: VOTER last_known_addr { host: "127.0.222.3" port: 33697 } } peers { permanent_uuid: "af75f3a57f6f48fba26736ad36aff045" member_type: VOTER last_known_addr { host: "127.0.222.1" port: 41951 } }
I20260809 12:10:51.722254 9639 raft_consensus.cc:348] T f360476a491744e092adc54e8e38a51e P cd3a4ffa0f4544a1b74b74fe5b6e3711 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cd3a4ffa0f4544a1b74b74fe5b6e3711" member_type: VOTER last_known_addr { host: "127.0.222.2" port: 43017 } } peers { permanent_uuid: "040356c8db9f4f8e93ae96f8fcaecf29" member_type: VOTER last_known_addr { host: "127.0.222.3" port: 33697 } } peers { permanent_uuid: "af75f3a57f6f48fba26736ad36aff045" member_type: VOTER last_known_addr { host: "127.0.222.1" port: 41951 } }
I20260809 12:10:51.722663 9639 raft_consensus.cc:374] T f360476a491744e092adc54e8e38a51e P cd3a4ffa0f4544a1b74b74fe5b6e3711 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260809 12:10:51.722774 9639 raft_consensus.cc:729] T f360476a491744e092adc54e8e38a51e P cd3a4ffa0f4544a1b74b74fe5b6e3711 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: cd3a4ffa0f4544a1b74b74fe5b6e3711, State: Initialized, Role: FOLLOWER
I20260809 12:10:51.723506 9639 consensus_queue.cc:260] T f360476a491744e092adc54e8e38a51e P cd3a4ffa0f4544a1b74b74fe5b6e3711 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cd3a4ffa0f4544a1b74b74fe5b6e3711" member_type: VOTER last_known_addr { host: "127.0.222.2" port: 43017 } } peers { permanent_uuid: "040356c8db9f4f8e93ae96f8fcaecf29" member_type: VOTER last_known_addr { host: "127.0.222.3" port: 33697 } } peers { permanent_uuid: "af75f3a57f6f48fba26736ad36aff045" member_type: VOTER last_known_addr { host: "127.0.222.1" port: 41951 } }
I20260809 12:10:51.728623 9638 ts_tablet_manager.cc:1428] T f360476a491744e092adc54e8e38a51e P af75f3a57f6f48fba26736ad36aff045: Time spent starting tablet: real 0.022s user 0.014s sys 0.005s
I20260809 12:10:51.730629 9463 heartbeater.cc:503] Master 127.0.222.62:36963 was elected leader, sending a full tablet report...
I20260809 12:10:51.731598 9639 ts_tablet_manager.cc:1428] T f360476a491744e092adc54e8e38a51e P cd3a4ffa0f4544a1b74b74fe5b6e3711: Time spent starting tablet: real 0.020s user 0.019s sys 0.000s
I20260809 12:10:51.737217 9640 ts_tablet_manager.cc:1428] T f360476a491744e092adc54e8e38a51e P 040356c8db9f4f8e93ae96f8fcaecf29: Time spent starting tablet: real 0.033s user 0.016s sys 0.014s
I20260809 12:10:51.737425 9611 heartbeater.cc:503] Master 127.0.222.62:36963 was elected leader, sending a full tablet report...
W20260809 12:10:51.787294 9612 tablet.cc:2369] T f360476a491744e092adc54e8e38a51e P 040356c8db9f4f8e93ae96f8fcaecf29: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260809 12:10:51.901742 9313 tablet.cc:2369] T f360476a491744e092adc54e8e38a51e P af75f3a57f6f48fba26736ad36aff045: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260809 12:10:51.977867 9464 tablet.cc:2369] T f360476a491744e092adc54e8e38a51e P cd3a4ffa0f4544a1b74b74fe5b6e3711: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260809 12:10:52.076736 9644 raft_consensus.cc:482] T f360476a491744e092adc54e8e38a51e P 040356c8db9f4f8e93ae96f8fcaecf29 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260809 12:10:52.077015 9644 raft_consensus.cc:504] T f360476a491744e092adc54e8e38a51e P 040356c8db9f4f8e93ae96f8fcaecf29 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cd3a4ffa0f4544a1b74b74fe5b6e3711" member_type: VOTER last_known_addr { host: "127.0.222.2" port: 43017 } } peers { permanent_uuid: "040356c8db9f4f8e93ae96f8fcaecf29" member_type: VOTER last_known_addr { host: "127.0.222.3" port: 33697 } } peers { permanent_uuid: "af75f3a57f6f48fba26736ad36aff045" member_type: VOTER last_known_addr { host: "127.0.222.1" port: 41951 } }
I20260809 12:10:52.078588 9644 leader_election.cc:288] T f360476a491744e092adc54e8e38a51e P 040356c8db9f4f8e93ae96f8fcaecf29 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers cd3a4ffa0f4544a1b74b74fe5b6e3711 (127.0.222.2:43017), af75f3a57f6f48fba26736ad36aff045 (127.0.222.1:41951)
I20260809 12:10:52.085706 9637 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 12:10:52.079195 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.222.3:55353 (local address 127.0.222.1:41951)
0809 12:10:52.079476 (+ 281us) server_negotiation.cc:206] Beginning negotiation
0809 12:10:52.079485 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0809 12:10:52.080104 (+ 619us) server_negotiation.cc:407] Connection header received
0809 12:10:52.080391 (+ 287us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 12:10:52.080400 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 12:10:52.080494 (+ 94us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 12:10:52.080567 (+ 73us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0809 12:10:52.081207 (+ 640us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 12:10:52.081967 (+ 760us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 12:10:52.083815 (+ 1848us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 12:10:52.084669 (+ 854us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 12:10:52.084777 (+ 108us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 12:10:52.084889 (+ 112us) server_negotiation.cc:1035] Waiting for connection context
0809 12:10:52.085302 (+ 413us) server_negotiation.cc:299] Negotiation successful
0809 12:10:52.085440 (+ 138us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":86}
I20260809 12:10:52.085736 9634 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 12:10:52.079257 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.222.3:60369 (local address 127.0.222.2:43017)
0809 12:10:52.079544 (+ 287us) server_negotiation.cc:206] Beginning negotiation
0809 12:10:52.079552 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0809 12:10:52.080279 (+ 727us) server_negotiation.cc:407] Connection header received
0809 12:10:52.080495 (+ 216us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 12:10:52.080499 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 12:10:52.080586 (+ 87us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 12:10:52.080681 (+ 95us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0809 12:10:52.081352 (+ 671us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 12:10:52.082249 (+ 897us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 12:10:52.084027 (+ 1778us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 12:10:52.084790 (+ 763us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 12:10:52.084941 (+ 151us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 12:10:52.085082 (+ 141us) server_negotiation.cc:1035] Waiting for connection context
0809 12:10:52.085368 (+ 286us) server_negotiation.cc:299] Negotiation successful
0809 12:10:52.085498 (+ 130us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":91}
I20260809 12:10:52.085826 9648 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 12:10:52.078875 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.222.1:41951 (local address 127.0.222.3:55353)
0809 12:10:52.079989 (+ 1114us) negotiation.cc:107] Waiting for socket to connect
0809 12:10:52.080005 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0809 12:10:52.080196 (+ 191us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 12:10:52.080607 (+ 411us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 12:10:52.080614 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 12:10:52.080644 (+ 30us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0809 12:10:52.080958 (+ 314us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 12:10:52.080971 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 12:10:52.082107 (+ 1136us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 12:10:52.082113 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 12:10:52.083630 (+ 1517us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 12:10:52.083642 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 12:10:52.084865 (+ 1223us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 12:10:52.084874 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 12:10:52.085121 (+ 247us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 12:10:52.085160 (+ 39us) client_negotiation.cc:769] Sending connection context
0809 12:10:52.085236 (+ 76us) client_negotiation.cc:240] Negotiation successful
0809 12:10:52.085374 (+ 138us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1007,"mutex_wait_us":37,"thread_start_us":834,"threads_started":1}
I20260809 12:10:52.086067 9649 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 12:10:52.078876 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.222.2:43017 (local address 127.0.222.3:60369)
0809 12:10:52.080165 (+ 1289us) negotiation.cc:107] Waiting for socket to connect
0809 12:10:52.080180 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0809 12:10:52.080323 (+ 143us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 12:10:52.080724 (+ 401us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 12:10:52.080731 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 12:10:52.080760 (+ 29us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0809 12:10:52.081150 (+ 390us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 12:10:52.081163 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 12:10:52.082378 (+ 1215us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 12:10:52.082382 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 12:10:52.083862 (+ 1480us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 12:10:52.083878 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 12:10:52.085054 (+ 1176us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 12:10:52.085059 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 12:10:52.085185 (+ 126us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 12:10:52.085208 (+ 23us) client_negotiation.cc:769] Sending connection context
0809 12:10:52.085335 (+ 127us) client_negotiation.cc:240] Negotiation successful
0809 12:10:52.085459 (+ 124us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1129,"mutex_wait_us":35,"thread_start_us":787,"threads_started":1}
I20260809 12:10:52.086905 9261 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f360476a491744e092adc54e8e38a51e" candidate_uuid: "040356c8db9f4f8e93ae96f8fcaecf29" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "af75f3a57f6f48fba26736ad36aff045" is_pre_election: true
I20260809 12:10:52.087236 9409 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f360476a491744e092adc54e8e38a51e" candidate_uuid: "040356c8db9f4f8e93ae96f8fcaecf29" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "cd3a4ffa0f4544a1b74b74fe5b6e3711" is_pre_election: true
I20260809 12:10:52.087453 9261 raft_consensus.cc:2454] T f360476a491744e092adc54e8e38a51e P af75f3a57f6f48fba26736ad36aff045 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 040356c8db9f4f8e93ae96f8fcaecf29 in term 0.
I20260809 12:10:52.087751 9409 raft_consensus.cc:2454] T f360476a491744e092adc54e8e38a51e P cd3a4ffa0f4544a1b74b74fe5b6e3711 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 040356c8db9f4f8e93ae96f8fcaecf29 in term 0.
I20260809 12:10:52.088164 9486 leader_election.cc:302] T f360476a491744e092adc54e8e38a51e P 040356c8db9f4f8e93ae96f8fcaecf29 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 040356c8db9f4f8e93ae96f8fcaecf29, af75f3a57f6f48fba26736ad36aff045; no voters:
I20260809 12:10:52.088588 9644 raft_consensus.cc:2781] T f360476a491744e092adc54e8e38a51e P 040356c8db9f4f8e93ae96f8fcaecf29 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260809 12:10:52.088766 9644 raft_consensus.cc:482] T f360476a491744e092adc54e8e38a51e P 040356c8db9f4f8e93ae96f8fcaecf29 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260809 12:10:52.088889 9644 raft_consensus.cc:3037] T f360476a491744e092adc54e8e38a51e P 040356c8db9f4f8e93ae96f8fcaecf29 [term 0 FOLLOWER]: Advancing to term 1
I20260809 12:10:52.091120 9644 raft_consensus.cc:504] T f360476a491744e092adc54e8e38a51e P 040356c8db9f4f8e93ae96f8fcaecf29 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cd3a4ffa0f4544a1b74b74fe5b6e3711" member_type: VOTER last_known_addr { host: "127.0.222.2" port: 43017 } } peers { permanent_uuid: "040356c8db9f4f8e93ae96f8fcaecf29" member_type: VOTER last_known_addr { host: "127.0.222.3" port: 33697 } } peers { permanent_uuid: "af75f3a57f6f48fba26736ad36aff045" member_type: VOTER last_known_addr { host: "127.0.222.1" port: 41951 } }
I20260809 12:10:52.091702 9644 leader_election.cc:288] T f360476a491744e092adc54e8e38a51e P 040356c8db9f4f8e93ae96f8fcaecf29 [CANDIDATE]: Term 1 election: Requested vote from peers cd3a4ffa0f4544a1b74b74fe5b6e3711 (127.0.222.2:43017), af75f3a57f6f48fba26736ad36aff045 (127.0.222.1:41951)
I20260809 12:10:52.092245 9409 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f360476a491744e092adc54e8e38a51e" candidate_uuid: "040356c8db9f4f8e93ae96f8fcaecf29" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "cd3a4ffa0f4544a1b74b74fe5b6e3711"
I20260809 12:10:52.092264 9261 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f360476a491744e092adc54e8e38a51e" candidate_uuid: "040356c8db9f4f8e93ae96f8fcaecf29" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "af75f3a57f6f48fba26736ad36aff045"
I20260809 12:10:52.092478 9409 raft_consensus.cc:3037] T f360476a491744e092adc54e8e38a51e P cd3a4ffa0f4544a1b74b74fe5b6e3711 [term 0 FOLLOWER]: Advancing to term 1
I20260809 12:10:52.092478 9261 raft_consensus.cc:3037] T f360476a491744e092adc54e8e38a51e P af75f3a57f6f48fba26736ad36aff045 [term 0 FOLLOWER]: Advancing to term 1
I20260809 12:10:52.095809 9409 raft_consensus.cc:2454] T f360476a491744e092adc54e8e38a51e P cd3a4ffa0f4544a1b74b74fe5b6e3711 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 040356c8db9f4f8e93ae96f8fcaecf29 in term 1.
I20260809 12:10:52.095819 9261 raft_consensus.cc:2454] T f360476a491744e092adc54e8e38a51e P af75f3a57f6f48fba26736ad36aff045 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 040356c8db9f4f8e93ae96f8fcaecf29 in term 1.
I20260809 12:10:52.096280 9486 leader_election.cc:302] T f360476a491744e092adc54e8e38a51e P 040356c8db9f4f8e93ae96f8fcaecf29 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 040356c8db9f4f8e93ae96f8fcaecf29, af75f3a57f6f48fba26736ad36aff045; no voters:
I20260809 12:10:52.096678 9644 raft_consensus.cc:2781] T f360476a491744e092adc54e8e38a51e P 040356c8db9f4f8e93ae96f8fcaecf29 [term 1 FOLLOWER]: Leader election won for term 1
I20260809 12:10:52.097105 9644 raft_consensus.cc:686] T f360476a491744e092adc54e8e38a51e P 040356c8db9f4f8e93ae96f8fcaecf29 [term 1 LEADER]: Becoming Leader. State: Replica: 040356c8db9f4f8e93ae96f8fcaecf29, State: Running, Role: LEADER
I20260809 12:10:52.097666 9644 consensus_queue.cc:237] T f360476a491744e092adc54e8e38a51e P 040356c8db9f4f8e93ae96f8fcaecf29 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cd3a4ffa0f4544a1b74b74fe5b6e3711" member_type: VOTER last_known_addr { host: "127.0.222.2" port: 43017 } } peers { permanent_uuid: "040356c8db9f4f8e93ae96f8fcaecf29" member_type: VOTER last_known_addr { host: "127.0.222.3" port: 33697 } } peers { permanent_uuid: "af75f3a57f6f48fba26736ad36aff045" member_type: VOTER last_known_addr { host: "127.0.222.1" port: 41951 } }
I20260809 12:10:52.102272 9085 catalog_manager.cc:5495] T f360476a491744e092adc54e8e38a51e P 040356c8db9f4f8e93ae96f8fcaecf29 reported cstate change: term changed from 0 to 1, leader changed from <none> to 040356c8db9f4f8e93ae96f8fcaecf29 (127.0.222.3). New cstate: current_term: 1 leader_uuid: "040356c8db9f4f8e93ae96f8fcaecf29" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cd3a4ffa0f4544a1b74b74fe5b6e3711" member_type: VOTER last_known_addr { host: "127.0.222.2" port: 43017 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "040356c8db9f4f8e93ae96f8fcaecf29" member_type: VOTER last_known_addr { host: "127.0.222.3" port: 33697 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "af75f3a57f6f48fba26736ad36aff045" member_type: VOTER last_known_addr { host: "127.0.222.1" port: 41951 } health_report { overall_health: UNKNOWN } } }
I20260809 12:10:52.135694 9635 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 12:10:52.130052 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53452 (local address 127.0.222.3:33697)
0809 12:10:52.130287 (+ 235us) server_negotiation.cc:206] Beginning negotiation
0809 12:10:52.130295 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0809 12:10:52.130513 (+ 218us) server_negotiation.cc:407] Connection header received
0809 12:10:52.130705 (+ 192us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 12:10:52.130714 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 12:10:52.130829 (+ 115us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 12:10:52.130981 (+ 152us) server_negotiation.cc:226] Negotiated authn=TOKEN
0809 12:10:52.131414 (+ 433us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 12:10:52.132151 (+ 737us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 12:10:52.133488 (+ 1337us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 12:10:52.133966 (+ 478us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 12:10:52.134072 (+ 106us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 12:10:52.134432 (+ 360us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0809 12:10:52.135099 (+ 667us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0809 12:10:52.135212 (+ 113us) server_negotiation.cc:1035] Waiting for connection context
0809 12:10:52.135309 (+ 97us) server_negotiation.cc:299] Negotiation successful
0809 12:10:52.135379 (+ 70us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":93}
I20260809 12:10:52.144160 9261 raft_consensus.cc:1261] T f360476a491744e092adc54e8e38a51e P af75f3a57f6f48fba26736ad36aff045 [term 1 FOLLOWER]: Refusing update from remote peer 040356c8db9f4f8e93ae96f8fcaecf29: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260809 12:10:52.144160 9409 raft_consensus.cc:1261] T f360476a491744e092adc54e8e38a51e P cd3a4ffa0f4544a1b74b74fe5b6e3711 [term 1 FOLLOWER]: Refusing update from remote peer 040356c8db9f4f8e93ae96f8fcaecf29: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260809 12:10:52.145136 9644 consensus_queue.cc:1035] T f360476a491744e092adc54e8e38a51e P 040356c8db9f4f8e93ae96f8fcaecf29 [LEADER]: Connected to new peer: Peer: permanent_uuid: "cd3a4ffa0f4544a1b74b74fe5b6e3711" member_type: VOTER last_known_addr { host: "127.0.222.2" port: 43017 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260809 12:10:52.145460 9650 consensus_queue.cc:1035] T f360476a491744e092adc54e8e38a51e P 040356c8db9f4f8e93ae96f8fcaecf29 [LEADER]: Connected to new peer: Peer: permanent_uuid: "af75f3a57f6f48fba26736ad36aff045" member_type: VOTER last_known_addr { host: "127.0.222.1" port: 41951 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260809 12:10:52.160435 9655 mvcc.cc:205] Tried to move back new op lower bound from 7316592443972743168 to 7316592443796123648. Current Snapshot: MvccSnapshot[applied={T|T < 7316592443972743168}]
I20260809 12:10:52.162359 9656 mvcc.cc:205] Tried to move back new op lower bound from 7316592443972743168 to 7316592443796123648. Current Snapshot: MvccSnapshot[applied={T|T < 7316592443972743168}]
I20260809 12:10:52.171834 9083 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:53632:
table { table_name: "test-table" } modify_external_catalogs: true
I20260809 12:10:52.172163 9083 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:53632:
table { table_name: "test-table" } modify_external_catalogs: true
I20260809 12:10:52.175715 9083 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 7873beb5ce214b9d8d238a51db1accb1: Sending DeleteTablet for 3 replicas of tablet f360476a491744e092adc54e8e38a51e
I20260809 12:10:52.177000 9386 tablet_service.cc:1514] Processing DeleteTablet for tablet f360476a491744e092adc54e8e38a51e with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-09 12:10:52 UTC) from {username='oryx', principal='oryx/127.0.222.62@KRBTEST.COM'} at 127.0.0.1:40518
I20260809 12:10:52.177306 9235 tablet_service.cc:1514] Processing DeleteTablet for tablet f360476a491744e092adc54e8e38a51e with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-09 12:10:52 UTC) from {username='oryx', principal='oryx/127.0.222.62@KRBTEST.COM'} at 127.0.0.1:49628
I20260809 12:10:52.177318 9537 tablet_service.cc:1514] Processing DeleteTablet for tablet f360476a491744e092adc54e8e38a51e with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-09 12:10:52 UTC) from {username='oryx', principal='oryx/127.0.222.62@KRBTEST.COM'} at 127.0.0.1:53446
I20260809 12:10:52.178926 9083 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:53632:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260809 12:10:52.179840 9083 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260809 12:10:52.191928 9234 tablet_service.cc:1467] Processing CreateTablet for tablet c105f50bcad744b0900363b66ec4cf33 (DEFAULT_TABLE table=test-table [id=fe2ecdf3ef1e468194e0999a3be19b18]), partition=RANGE (key) PARTITION UNBOUNDED
I20260809 12:10:52.192721 9234 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c105f50bcad744b0900363b66ec4cf33. 1 dirs total, 0 dirs full, 0 dirs failed
I20260809 12:10:52.195427 9664 tablet_replica.cc:330] T f360476a491744e092adc54e8e38a51e P af75f3a57f6f48fba26736ad36aff045: stopping tablet replica
I20260809 12:10:52.195770 9666 tablet_replica.cc:330] T f360476a491744e092adc54e8e38a51e P 040356c8db9f4f8e93ae96f8fcaecf29: stopping tablet replica
I20260809 12:10:52.195964 9664 raft_consensus.cc:2229] T f360476a491744e092adc54e8e38a51e P af75f3a57f6f48fba26736ad36aff045 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260809 12:10:52.196266 9666 raft_consensus.cc:2229] T f360476a491744e092adc54e8e38a51e P 040356c8db9f4f8e93ae96f8fcaecf29 [term 1 LEADER]: Raft consensus shutting down.
I20260809 12:10:52.197078 9663 tablet_replica.cc:330] T f360476a491744e092adc54e8e38a51e P cd3a4ffa0f4544a1b74b74fe5b6e3711: stopping tablet replica
I20260809 12:10:52.197103 9666 raft_consensus.cc:2258] T f360476a491744e092adc54e8e38a51e P 040356c8db9f4f8e93ae96f8fcaecf29 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260809 12:10:52.197728 9663 raft_consensus.cc:2229] T f360476a491744e092adc54e8e38a51e P cd3a4ffa0f4544a1b74b74fe5b6e3711 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260809 12:10:52.198231 9663 raft_consensus.cc:2258] T f360476a491744e092adc54e8e38a51e P cd3a4ffa0f4544a1b74b74fe5b6e3711 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260809 12:10:52.198833 9664 raft_consensus.cc:2258] T f360476a491744e092adc54e8e38a51e P af75f3a57f6f48fba26736ad36aff045 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260809 12:10:52.199692 9386 tablet_service.cc:1467] Processing CreateTablet for tablet c105f50bcad744b0900363b66ec4cf33 (DEFAULT_TABLE table=test-table [id=fe2ecdf3ef1e468194e0999a3be19b18]), partition=RANGE (key) PARTITION UNBOUNDED
I20260809 12:10:52.200184 9386 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c105f50bcad744b0900363b66ec4cf33. 1 dirs total, 0 dirs full, 0 dirs failed
I20260809 12:10:52.200790 9537 tablet_service.cc:1467] Processing CreateTablet for tablet c105f50bcad744b0900363b66ec4cf33 (DEFAULT_TABLE table=test-table [id=fe2ecdf3ef1e468194e0999a3be19b18]), partition=RANGE (key) PARTITION UNBOUNDED
I20260809 12:10:52.201004 9663 ts_tablet_manager.cc:1905] T f360476a491744e092adc54e8e38a51e P cd3a4ffa0f4544a1b74b74fe5b6e3711: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260809 12:10:52.201256 9537 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c105f50bcad744b0900363b66ec4cf33. 1 dirs total, 0 dirs full, 0 dirs failed
I20260809 12:10:52.208392 9663 ts_tablet_manager.cc:1918] T f360476a491744e092adc54e8e38a51e P cd3a4ffa0f4544a1b74b74fe5b6e3711: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260809 12:10:52.208922 9666 ts_tablet_manager.cc:1905] T f360476a491744e092adc54e8e38a51e P 040356c8db9f4f8e93ae96f8fcaecf29: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260809 12:10:52.209800 9640 tablet_bootstrap.cc:492] T c105f50bcad744b0900363b66ec4cf33 P 040356c8db9f4f8e93ae96f8fcaecf29: Bootstrap starting.
I20260809 12:10:52.210757 9638 tablet_bootstrap.cc:492] T c105f50bcad744b0900363b66ec4cf33 P af75f3a57f6f48fba26736ad36aff045: Bootstrap starting.
I20260809 12:10:52.210834 9664 ts_tablet_manager.cc:1905] T f360476a491744e092adc54e8e38a51e P af75f3a57f6f48fba26736ad36aff045: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260809 12:10:52.212641 9638 tablet_bootstrap.cc:654] T c105f50bcad744b0900363b66ec4cf33 P af75f3a57f6f48fba26736ad36aff045: Neither blocks nor log segments found. Creating new log.
I20260809 12:10:52.213658 9640 tablet_bootstrap.cc:654] T c105f50bcad744b0900363b66ec4cf33 P 040356c8db9f4f8e93ae96f8fcaecf29: Neither blocks nor log segments found. Creating new log.
I20260809 12:10:52.218089 9664 ts_tablet_manager.cc:1918] T f360476a491744e092adc54e8e38a51e P af75f3a57f6f48fba26736ad36aff045: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260809 12:10:52.218271 9664 log.cc:1196] T f360476a491744e092adc54e8e38a51e P af75f3a57f6f48fba26736ad36aff045: Deleting WAL directory at /tmp/dist-test-taskgH5Pjm/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786277382073938-888-0/minicluster-data/ts-0/wal/wals/f360476a491744e092adc54e8e38a51e
I20260809 12:10:52.218837 9664 ts_tablet_manager.cc:1939] T f360476a491744e092adc54e8e38a51e P af75f3a57f6f48fba26736ad36aff045: Deleting consensus metadata
I20260809 12:10:52.219126 9663 log.cc:1196] T f360476a491744e092adc54e8e38a51e P cd3a4ffa0f4544a1b74b74fe5b6e3711: Deleting WAL directory at /tmp/dist-test-taskgH5Pjm/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786277382073938-888-0/minicluster-data/ts-1/wal/wals/f360476a491744e092adc54e8e38a51e
I20260809 12:10:52.219749 9639 tablet_bootstrap.cc:492] T c105f50bcad744b0900363b66ec4cf33 P cd3a4ffa0f4544a1b74b74fe5b6e3711: Bootstrap starting.
I20260809 12:10:52.221454 9639 tablet_bootstrap.cc:654] T c105f50bcad744b0900363b66ec4cf33 P cd3a4ffa0f4544a1b74b74fe5b6e3711: Neither blocks nor log segments found. Creating new log.
I20260809 12:10:52.219750 9663 ts_tablet_manager.cc:1939] T f360476a491744e092adc54e8e38a51e P cd3a4ffa0f4544a1b74b74fe5b6e3711: Deleting consensus metadata
I20260809 12:10:52.224191 9639 tablet_bootstrap.cc:492] T c105f50bcad744b0900363b66ec4cf33 P cd3a4ffa0f4544a1b74b74fe5b6e3711: No bootstrap required, opened a new log
I20260809 12:10:52.224354 9639 ts_tablet_manager.cc:1397] T c105f50bcad744b0900363b66ec4cf33 P cd3a4ffa0f4544a1b74b74fe5b6e3711: Time spent bootstrapping tablet: real 0.005s user 0.003s sys 0.000s
I20260809 12:10:52.224735 9068 catalog_manager.cc:4841] TS cd3a4ffa0f4544a1b74b74fe5b6e3711 (127.0.222.2:43017): tablet f360476a491744e092adc54e8e38a51e (table test-table [id=d1f6e99d50a44c74a2715f16cbb78778]) successfully deleted
I20260809 12:10:52.225221 9639 raft_consensus.cc:348] T c105f50bcad744b0900363b66ec4cf33 P cd3a4ffa0f4544a1b74b74fe5b6e3711 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "af75f3a57f6f48fba26736ad36aff045" member_type: VOTER last_known_addr { host: "127.0.222.1" port: 41951 } } peers { permanent_uuid: "cd3a4ffa0f4544a1b74b74fe5b6e3711" member_type: VOTER last_known_addr { host: "127.0.222.2" port: 43017 } } peers { permanent_uuid: "040356c8db9f4f8e93ae96f8fcaecf29" member_type: VOTER last_known_addr { host: "127.0.222.3" port: 33697 } }
I20260809 12:10:52.225459 9639 raft_consensus.cc:374] T c105f50bcad744b0900363b66ec4cf33 P cd3a4ffa0f4544a1b74b74fe5b6e3711 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260809 12:10:52.225561 9639 raft_consensus.cc:729] T c105f50bcad744b0900363b66ec4cf33 P cd3a4ffa0f4544a1b74b74fe5b6e3711 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: cd3a4ffa0f4544a1b74b74fe5b6e3711, State: Initialized, Role: FOLLOWER
I20260809 12:10:52.225883 9639 consensus_queue.cc:260] T c105f50bcad744b0900363b66ec4cf33 P cd3a4ffa0f4544a1b74b74fe5b6e3711 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "af75f3a57f6f48fba26736ad36aff045" member_type: VOTER last_known_addr { host: "127.0.222.1" port: 41951 } } peers { permanent_uuid: "cd3a4ffa0f4544a1b74b74fe5b6e3711" member_type: VOTER last_known_addr { host: "127.0.222.2" port: 43017 } } peers { permanent_uuid: "040356c8db9f4f8e93ae96f8fcaecf29" member_type: VOTER last_known_addr { host: "127.0.222.3" port: 33697 } }
I20260809 12:10:52.226747 9069 catalog_manager.cc:4841] TS af75f3a57f6f48fba26736ad36aff045 (127.0.222.1:41951): tablet f360476a491744e092adc54e8e38a51e (table test-table [id=d1f6e99d50a44c74a2715f16cbb78778]) successfully deleted
I20260809 12:10:52.227989 9666 ts_tablet_manager.cc:1918] T f360476a491744e092adc54e8e38a51e P 040356c8db9f4f8e93ae96f8fcaecf29: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260809 12:10:52.228235 9666 log.cc:1196] T f360476a491744e092adc54e8e38a51e P 040356c8db9f4f8e93ae96f8fcaecf29: Deleting WAL directory at /tmp/dist-test-taskgH5Pjm/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786277382073938-888-0/minicluster-data/ts-2/wal/wals/f360476a491744e092adc54e8e38a51e
I20260809 12:10:52.228242 9640 tablet_bootstrap.cc:492] T c105f50bcad744b0900363b66ec4cf33 P 040356c8db9f4f8e93ae96f8fcaecf29: No bootstrap required, opened a new log
I20260809 12:10:52.228438 9640 ts_tablet_manager.cc:1397] T c105f50bcad744b0900363b66ec4cf33 P 040356c8db9f4f8e93ae96f8fcaecf29: Time spent bootstrapping tablet: real 0.019s user 0.002s sys 0.006s
I20260809 12:10:52.229002 9666 ts_tablet_manager.cc:1939] T f360476a491744e092adc54e8e38a51e P 040356c8db9f4f8e93ae96f8fcaecf29: Deleting consensus metadata
I20260809 12:10:52.229344 9640 raft_consensus.cc:348] T c105f50bcad744b0900363b66ec4cf33 P 040356c8db9f4f8e93ae96f8fcaecf29 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "af75f3a57f6f48fba26736ad36aff045" member_type: VOTER last_known_addr { host: "127.0.222.1" port: 41951 } } peers { permanent_uuid: "cd3a4ffa0f4544a1b74b74fe5b6e3711" member_type: VOTER last_known_addr { host: "127.0.222.2" port: 43017 } } peers { permanent_uuid: "040356c8db9f4f8e93ae96f8fcaecf29" member_type: VOTER last_known_addr { host: "127.0.222.3" port: 33697 } }
I20260809 12:10:52.229605 9640 raft_consensus.cc:374] T c105f50bcad744b0900363b66ec4cf33 P 040356c8db9f4f8e93ae96f8fcaecf29 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260809 12:10:52.229722 9640 raft_consensus.cc:729] T c105f50bcad744b0900363b66ec4cf33 P 040356c8db9f4f8e93ae96f8fcaecf29 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 040356c8db9f4f8e93ae96f8fcaecf29, State: Initialized, Role: FOLLOWER
I20260809 12:10:52.230010 9640 consensus_queue.cc:260] T c105f50bcad744b0900363b66ec4cf33 P 040356c8db9f4f8e93ae96f8fcaecf29 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "af75f3a57f6f48fba26736ad36aff045" member_type: VOTER last_known_addr { host: "127.0.222.1" port: 41951 } } peers { permanent_uuid: "cd3a4ffa0f4544a1b74b74fe5b6e3711" member_type: VOTER last_known_addr { host: "127.0.222.2" port: 43017 } } peers { permanent_uuid: "040356c8db9f4f8e93ae96f8fcaecf29" member_type: VOTER last_known_addr { host: "127.0.222.3" port: 33697 } }
I20260809 12:10:52.230370 9639 ts_tablet_manager.cc:1428] T c105f50bcad744b0900363b66ec4cf33 P cd3a4ffa0f4544a1b74b74fe5b6e3711: Time spent starting tablet: real 0.006s user 0.003s sys 0.000s
I20260809 12:10:52.231835 9071 catalog_manager.cc:4841] TS 040356c8db9f4f8e93ae96f8fcaecf29 (127.0.222.3:33697): tablet f360476a491744e092adc54e8e38a51e (table test-table [id=d1f6e99d50a44c74a2715f16cbb78778]) successfully deleted
I20260809 12:10:52.230623 9640 ts_tablet_manager.cc:1428] T c105f50bcad744b0900363b66ec4cf33 P 040356c8db9f4f8e93ae96f8fcaecf29: Time spent starting tablet: real 0.002s user 0.001s sys 0.001s
I20260809 12:10:52.233798 9638 tablet_bootstrap.cc:492] T c105f50bcad744b0900363b66ec4cf33 P af75f3a57f6f48fba26736ad36aff045: No bootstrap required, opened a new log
I20260809 12:10:52.234586 9638 ts_tablet_manager.cc:1397] T c105f50bcad744b0900363b66ec4cf33 P af75f3a57f6f48fba26736ad36aff045: Time spent bootstrapping tablet: real 0.024s user 0.001s sys 0.012s
I20260809 12:10:52.235585 9638 raft_consensus.cc:348] T c105f50bcad744b0900363b66ec4cf33 P af75f3a57f6f48fba26736ad36aff045 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "af75f3a57f6f48fba26736ad36aff045" member_type: VOTER last_known_addr { host: "127.0.222.1" port: 41951 } } peers { permanent_uuid: "cd3a4ffa0f4544a1b74b74fe5b6e3711" member_type: VOTER last_known_addr { host: "127.0.222.2" port: 43017 } } peers { permanent_uuid: "040356c8db9f4f8e93ae96f8fcaecf29" member_type: VOTER last_known_addr { host: "127.0.222.3" port: 33697 } }
I20260809 12:10:52.235826 9638 raft_consensus.cc:374] T c105f50bcad744b0900363b66ec4cf33 P af75f3a57f6f48fba26736ad36aff045 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260809 12:10:52.235926 9638 raft_consensus.cc:729] T c105f50bcad744b0900363b66ec4cf33 P af75f3a57f6f48fba26736ad36aff045 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: af75f3a57f6f48fba26736ad36aff045, State: Initialized, Role: FOLLOWER
I20260809 12:10:52.236238 9638 consensus_queue.cc:260] T c105f50bcad744b0900363b66ec4cf33 P af75f3a57f6f48fba26736ad36aff045 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "af75f3a57f6f48fba26736ad36aff045" member_type: VOTER last_known_addr { host: "127.0.222.1" port: 41951 } } peers { permanent_uuid: "cd3a4ffa0f4544a1b74b74fe5b6e3711" member_type: VOTER last_known_addr { host: "127.0.222.2" port: 43017 } } peers { permanent_uuid: "040356c8db9f4f8e93ae96f8fcaecf29" member_type: VOTER last_known_addr { host: "127.0.222.3" port: 33697 } }
I20260809 12:10:52.237327 9638 ts_tablet_manager.cc:1428] T c105f50bcad744b0900363b66ec4cf33 P af75f3a57f6f48fba26736ad36aff045: Time spent starting tablet: real 0.003s user 0.002s sys 0.000s
I20260809 12:10:52.244154 9646 raft_consensus.cc:482] T c105f50bcad744b0900363b66ec4cf33 P cd3a4ffa0f4544a1b74b74fe5b6e3711 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260809 12:10:52.244343 9646 raft_consensus.cc:504] T c105f50bcad744b0900363b66ec4cf33 P cd3a4ffa0f4544a1b74b74fe5b6e3711 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "af75f3a57f6f48fba26736ad36aff045" member_type: VOTER last_known_addr { host: "127.0.222.1" port: 41951 } } peers { permanent_uuid: "cd3a4ffa0f4544a1b74b74fe5b6e3711" member_type: VOTER last_known_addr { host: "127.0.222.2" port: 43017 } } peers { permanent_uuid: "040356c8db9f4f8e93ae96f8fcaecf29" member_type: VOTER last_known_addr { host: "127.0.222.3" port: 33697 } }
I20260809 12:10:52.246246 9646 leader_election.cc:288] T c105f50bcad744b0900363b66ec4cf33 P cd3a4ffa0f4544a1b74b74fe5b6e3711 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers af75f3a57f6f48fba26736ad36aff045 (127.0.222.1:41951), 040356c8db9f4f8e93ae96f8fcaecf29 (127.0.222.3:33697)
I20260809 12:10:52.254774 9635 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 12:10:52.247370 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.222.2:45047 (local address 127.0.222.3:33697)
0809 12:10:52.247568 (+ 198us) server_negotiation.cc:206] Beginning negotiation
0809 12:10:52.247575 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0809 12:10:52.248132 (+ 557us) server_negotiation.cc:407] Connection header received
0809 12:10:52.248336 (+ 204us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 12:10:52.248341 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 12:10:52.248420 (+ 79us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 12:10:52.248538 (+ 118us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0809 12:10:52.249193 (+ 655us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 12:10:52.250385 (+ 1192us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 12:10:52.252801 (+ 2416us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 12:10:52.253644 (+ 843us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 12:10:52.253816 (+ 172us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 12:10:52.253918 (+ 102us) server_negotiation.cc:1035] Waiting for connection context
0809 12:10:52.254314 (+ 396us) server_negotiation.cc:299] Negotiation successful
0809 12:10:52.254449 (+ 135us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":94}
I20260809 12:10:52.254849 9671 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 12:10:52.246650 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.222.3:33697 (local address 127.0.222.2:45047)
0809 12:10:52.248038 (+ 1388us) negotiation.cc:107] Waiting for socket to connect
0809 12:10:52.248055 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0809 12:10:52.248188 (+ 133us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 12:10:52.248576 (+ 388us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 12:10:52.248581 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 12:10:52.248615 (+ 34us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0809 12:10:52.249009 (+ 394us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 12:10:52.249020 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 12:10:52.250534 (+ 1514us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 12:10:52.250540 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 12:10:52.252642 (+ 2102us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 12:10:52.252661 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 12:10:52.253868 (+ 1207us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 12:10:52.253874 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 12:10:52.254123 (+ 249us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 12:10:52.254165 (+ 42us) client_negotiation.cc:769] Sending connection context
0809 12:10:52.254270 (+ 105us) client_negotiation.cc:240] Negotiation successful
0809 12:10:52.254414 (+ 144us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1249,"mutex_wait_us":4,"thread_start_us":1094,"threads_started":1}
I20260809 12:10:52.255290 9637 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 12:10:52.246717 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.222.2:44869 (local address 127.0.222.1:41951)
0809 12:10:52.247126 (+ 409us) server_negotiation.cc:206] Beginning negotiation
0809 12:10:52.247132 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0809 12:10:52.247443 (+ 311us) server_negotiation.cc:407] Connection header received
0809 12:10:52.247646 (+ 203us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 12:10:52.247654 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 12:10:52.247744 (+ 90us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 12:10:52.247838 (+ 94us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0809 12:10:52.248653 (+ 815us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 12:10:52.249882 (+ 1229us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 12:10:52.252608 (+ 2726us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 12:10:52.253890 (+ 1282us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 12:10:52.254054 (+ 164us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 12:10:52.254154 (+ 100us) server_negotiation.cc:1035] Waiting for connection context
0809 12:10:52.254482 (+ 328us) server_negotiation.cc:299] Negotiation successful
0809 12:10:52.254629 (+ 147us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":276}
I20260809 12:10:52.255724 9558 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c105f50bcad744b0900363b66ec4cf33" candidate_uuid: "cd3a4ffa0f4544a1b74b74fe5b6e3711" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "040356c8db9f4f8e93ae96f8fcaecf29" is_pre_election: true
I20260809 12:10:52.256175 9558 raft_consensus.cc:2454] T c105f50bcad744b0900363b66ec4cf33 P 040356c8db9f4f8e93ae96f8fcaecf29 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate cd3a4ffa0f4544a1b74b74fe5b6e3711 in term 0.
I20260809 12:10:52.256268 9670 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 12:10:52.246369 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.222.1:41951 (local address 127.0.222.2:44869)
0809 12:10:52.247336 (+ 967us) negotiation.cc:107] Waiting for socket to connect
0809 12:10:52.247351 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0809 12:10:52.247502 (+ 151us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 12:10:52.248021 (+ 519us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 12:10:52.248030 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 12:10:52.248059 (+ 29us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0809 12:10:52.248456 (+ 397us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 12:10:52.248466 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 12:10:52.250021 (+ 1555us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 12:10:52.250026 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 12:10:52.252439 (+ 2413us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 12:10:52.252454 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 12:10:52.254117 (+ 1663us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 12:10:52.254123 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 12:10:52.254306 (+ 183us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 12:10:52.254336 (+ 30us) client_negotiation.cc:769] Sending connection context
0809 12:10:52.255681 (+ 1345us) client_negotiation.cc:240] Negotiation successful
0809 12:10:52.255819 (+ 138us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":840,"mutex_wait_us":63,"thread_start_us":688,"threads_started":1}
I20260809 12:10:52.256824 9338 leader_election.cc:302] T c105f50bcad744b0900363b66ec4cf33 P cd3a4ffa0f4544a1b74b74fe5b6e3711 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 040356c8db9f4f8e93ae96f8fcaecf29, cd3a4ffa0f4544a1b74b74fe5b6e3711; no voters:
I20260809 12:10:52.257243 9646 raft_consensus.cc:2781] T c105f50bcad744b0900363b66ec4cf33 P cd3a4ffa0f4544a1b74b74fe5b6e3711 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260809 12:10:52.257184 9261 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c105f50bcad744b0900363b66ec4cf33" candidate_uuid: "cd3a4ffa0f4544a1b74b74fe5b6e3711" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "af75f3a57f6f48fba26736ad36aff045" is_pre_election: true
I20260809 12:10:52.257364 9646 raft_consensus.cc:482] T c105f50bcad744b0900363b66ec4cf33 P cd3a4ffa0f4544a1b74b74fe5b6e3711 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260809 12:10:52.257433 9261 raft_consensus.cc:2454] T c105f50bcad744b0900363b66ec4cf33 P af75f3a57f6f48fba26736ad36aff045 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate cd3a4ffa0f4544a1b74b74fe5b6e3711 in term 0.
I20260809 12:10:52.257490 9646 raft_consensus.cc:3037] T c105f50bcad744b0900363b66ec4cf33 P cd3a4ffa0f4544a1b74b74fe5b6e3711 [term 0 FOLLOWER]: Advancing to term 1
I20260809 12:10:52.259950 9646 raft_consensus.cc:504] T c105f50bcad744b0900363b66ec4cf33 P cd3a4ffa0f4544a1b74b74fe5b6e3711 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "af75f3a57f6f48fba26736ad36aff045" member_type: VOTER last_known_addr { host: "127.0.222.1" port: 41951 } } peers { permanent_uuid: "cd3a4ffa0f4544a1b74b74fe5b6e3711" member_type: VOTER last_known_addr { host: "127.0.222.2" port: 43017 } } peers { permanent_uuid: "040356c8db9f4f8e93ae96f8fcaecf29" member_type: VOTER last_known_addr { host: "127.0.222.3" port: 33697 } }
I20260809 12:10:52.260627 9646 leader_election.cc:288] T c105f50bcad744b0900363b66ec4cf33 P cd3a4ffa0f4544a1b74b74fe5b6e3711 [CANDIDATE]: Term 1 election: Requested vote from peers af75f3a57f6f48fba26736ad36aff045 (127.0.222.1:41951), 040356c8db9f4f8e93ae96f8fcaecf29 (127.0.222.3:33697)
I20260809 12:10:52.261039 9261 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c105f50bcad744b0900363b66ec4cf33" candidate_uuid: "cd3a4ffa0f4544a1b74b74fe5b6e3711" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "af75f3a57f6f48fba26736ad36aff045"
I20260809 12:10:52.261251 9261 raft_consensus.cc:3037] T c105f50bcad744b0900363b66ec4cf33 P af75f3a57f6f48fba26736ad36aff045 [term 0 FOLLOWER]: Advancing to term 1
I20260809 12:10:52.261220 9558 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c105f50bcad744b0900363b66ec4cf33" candidate_uuid: "cd3a4ffa0f4544a1b74b74fe5b6e3711" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "040356c8db9f4f8e93ae96f8fcaecf29"
I20260809 12:10:52.261472 9558 raft_consensus.cc:3037] T c105f50bcad744b0900363b66ec4cf33 P 040356c8db9f4f8e93ae96f8fcaecf29 [term 0 FOLLOWER]: Advancing to term 1
I20260809 12:10:52.264389 9261 raft_consensus.cc:2454] T c105f50bcad744b0900363b66ec4cf33 P af75f3a57f6f48fba26736ad36aff045 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate cd3a4ffa0f4544a1b74b74fe5b6e3711 in term 1.
I20260809 12:10:52.264803 9336 leader_election.cc:302] T c105f50bcad744b0900363b66ec4cf33 P cd3a4ffa0f4544a1b74b74fe5b6e3711 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: af75f3a57f6f48fba26736ad36aff045, cd3a4ffa0f4544a1b74b74fe5b6e3711; no voters:
I20260809 12:10:52.265100 9558 raft_consensus.cc:2454] T c105f50bcad744b0900363b66ec4cf33 P 040356c8db9f4f8e93ae96f8fcaecf29 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate cd3a4ffa0f4544a1b74b74fe5b6e3711 in term 1.
I20260809 12:10:52.265163 9646 raft_consensus.cc:2781] T c105f50bcad744b0900363b66ec4cf33 P cd3a4ffa0f4544a1b74b74fe5b6e3711 [term 1 FOLLOWER]: Leader election won for term 1
I20260809 12:10:52.267298 9646 raft_consensus.cc:686] T c105f50bcad744b0900363b66ec4cf33 P cd3a4ffa0f4544a1b74b74fe5b6e3711 [term 1 LEADER]: Becoming Leader. State: Replica: cd3a4ffa0f4544a1b74b74fe5b6e3711, State: Running, Role: LEADER
I20260809 12:10:52.267740 9646 consensus_queue.cc:237] T c105f50bcad744b0900363b66ec4cf33 P cd3a4ffa0f4544a1b74b74fe5b6e3711 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "af75f3a57f6f48fba26736ad36aff045" member_type: VOTER last_known_addr { host: "127.0.222.1" port: 41951 } } peers { permanent_uuid: "cd3a4ffa0f4544a1b74b74fe5b6e3711" member_type: VOTER last_known_addr { host: "127.0.222.2" port: 43017 } } peers { permanent_uuid: "040356c8db9f4f8e93ae96f8fcaecf29" member_type: VOTER last_known_addr { host: "127.0.222.3" port: 33697 } }
I20260809 12:10:52.271135 9085 catalog_manager.cc:5495] T c105f50bcad744b0900363b66ec4cf33 P cd3a4ffa0f4544a1b74b74fe5b6e3711 reported cstate change: term changed from 0 to 1, leader changed from <none> to cd3a4ffa0f4544a1b74b74fe5b6e3711 (127.0.222.2). New cstate: current_term: 1 leader_uuid: "cd3a4ffa0f4544a1b74b74fe5b6e3711" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "af75f3a57f6f48fba26736ad36aff045" member_type: VOTER last_known_addr { host: "127.0.222.1" port: 41951 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "cd3a4ffa0f4544a1b74b74fe5b6e3711" member_type: VOTER last_known_addr { host: "127.0.222.2" port: 43017 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "040356c8db9f4f8e93ae96f8fcaecf29" member_type: VOTER last_known_addr { host: "127.0.222.3" port: 33697 } health_report { overall_health: UNKNOWN } } }
Aug 09 12:10:52 dist-test-slave-2w3w krb5kdc[9036](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786277449, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.222.62@KRBTEST.COM for oryx/127.0.222.62@KRBTEST.COM
I20260809 12:10:52.328984 9681 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 12:10:52.313487 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.222.62:36963 (local address 127.0.0.1:53634)
0809 12:10:52.314444 (+ 957us) negotiation.cc:107] Waiting for socket to connect
0809 12:10:52.314461 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0809 12:10:52.314604 (+ 143us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 12:10:52.315046 (+ 442us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 12:10:52.315049 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 12:10:52.315770 (+ 721us) client_negotiation.cc:189] Negotiated authn=SASL
0809 12:10:52.315993 (+ 223us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 12:10:52.316003 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 12:10:52.317522 (+ 1519us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 12:10:52.317525 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 12:10:52.318528 (+ 1003us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 12:10:52.318539 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 12:10:52.319581 (+ 1042us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 12:10:52.319585 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 12:10:52.319732 (+ 147us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 12:10:52.320433 (+ 701us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0809 12:10:52.320450 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0809 12:10:52.322694 (+ 2244us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0809 12:10:52.326527 (+ 3833us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0809 12:10:52.326538 (+ 11us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0809 12:10:52.326544 (+ 6us) client_negotiation.cc:760] calling sasl_client_step()
0809 12:10:52.327010 (+ 466us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0809 12:10:52.327403 (+ 393us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0809 12:10:52.327406 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0809 12:10:52.327408 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0809 12:10:52.327564 (+ 156us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0809 12:10:52.328170 (+ 606us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0809 12:10:52.328177 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0809 12:10:52.328330 (+ 153us) client_negotiation.cc:769] Sending connection context
0809 12:10:52.328476 (+ 146us) client_negotiation.cc:240] Negotiation successful
0809 12:10:52.328682 (+ 206us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":801,"thread_start_us":643,"threads_started":1}
I20260809 12:10:52.329329 9680 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 12:10:52.313500 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53634 (local address 127.0.222.62:36963)
0809 12:10:52.314222 (+ 722us) server_negotiation.cc:206] Beginning negotiation
0809 12:10:52.314229 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0809 12:10:52.314566 (+ 337us) server_negotiation.cc:407] Connection header received
0809 12:10:52.314730 (+ 164us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 12:10:52.314736 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 12:10:52.314831 (+ 95us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 12:10:52.315000 (+ 169us) server_negotiation.cc:226] Negotiated authn=SASL
0809 12:10:52.316169 (+ 1169us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 12:10:52.317345 (+ 1176us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 12:10:52.318678 (+ 1333us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 12:10:52.319417 (+ 739us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 12:10:52.319545 (+ 128us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 12:10:52.322912 (+ 3367us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 12:10:52.322919 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 12:10:52.322924 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0809 12:10:52.323002 (+ 78us) server_negotiation.cc:922] calling sasl_server_start()
0809 12:10:52.326392 (+ 3390us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 12:10:52.327133 (+ 741us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 12:10:52.327138 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 12:10:52.327142 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0809 12:10:52.327291 (+ 149us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 12:10:52.327669 (+ 378us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 12:10:52.327674 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 12:10:52.327677 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0809 12:10:52.328043 (+ 366us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 12:10:52.328192 (+ 149us) server_negotiation.cc:1035] Waiting for connection context
0809 12:10:52.328602 (+ 410us) server_negotiation.cc:299] Negotiation successful
0809 12:10:52.328831 (+ 229us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":548,"thread_start_us":325,"threads_started":1}
I20260809 12:10:52.334445 9085 catalog_manager.cc:1908] Servicing CreateTable request from {username='oryx', principal='oryx/127.0.222.62@KRBTEST.COM'} at 127.0.0.1:53634:
name: "kudu_system.kudu_transactions"
schema {
columns {
name: "txn_id"
type: INT64
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "entry_type"
type: INT8
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "identifier"
type: STRING
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "metadata"
type: STRING
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
rows: "<redacted>""\006\001\000\000\000\000\000\000\000\000\007\001@B\017\000\000\000\000\000"
indirect_data: "<redacted>"""
}
partition_schema {
range_schema {
columns {
name: "txn_id"
}
}
}
table_type: TXN_STATUS_TABLE
W20260809 12:10:52.335507 9085 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table kudu_system.kudu_transactions in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260809 12:10:52.342751 9234 tablet_service.cc:1467] Processing CreateTablet for tablet 19d7e57f295343b892ba515f7de56686 (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=e83d10c8460a4d0ab17cc2076a8ac867]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260809 12:10:52.343307 9234 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 19d7e57f295343b892ba515f7de56686. 1 dirs total, 0 dirs full, 0 dirs failed
I20260809 12:10:52.347426 9386 tablet_service.cc:1467] Processing CreateTablet for tablet 19d7e57f295343b892ba515f7de56686 (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=e83d10c8460a4d0ab17cc2076a8ac867]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260809 12:10:52.347566 9537 tablet_service.cc:1467] Processing CreateTablet for tablet 19d7e57f295343b892ba515f7de56686 (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=e83d10c8460a4d0ab17cc2076a8ac867]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260809 12:10:52.348055 9386 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 19d7e57f295343b892ba515f7de56686. 1 dirs total, 0 dirs full, 0 dirs failed
I20260809 12:10:52.348198 9537 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 19d7e57f295343b892ba515f7de56686. 1 dirs total, 0 dirs full, 0 dirs failed
I20260809 12:10:52.348888 9638 tablet_bootstrap.cc:492] T 19d7e57f295343b892ba515f7de56686 P af75f3a57f6f48fba26736ad36aff045: Bootstrap starting.
I20260809 12:10:52.350659 9638 tablet_bootstrap.cc:654] T 19d7e57f295343b892ba515f7de56686 P af75f3a57f6f48fba26736ad36aff045: Neither blocks nor log segments found. Creating new log.
I20260809 12:10:52.355717 9639 tablet_bootstrap.cc:492] T 19d7e57f295343b892ba515f7de56686 P cd3a4ffa0f4544a1b74b74fe5b6e3711: Bootstrap starting.
I20260809 12:10:52.357383 9639 tablet_bootstrap.cc:654] T 19d7e57f295343b892ba515f7de56686 P cd3a4ffa0f4544a1b74b74fe5b6e3711: Neither blocks nor log segments found. Creating new log.
I20260809 12:10:52.358502 9640 tablet_bootstrap.cc:492] T 19d7e57f295343b892ba515f7de56686 P 040356c8db9f4f8e93ae96f8fcaecf29: Bootstrap starting.
I20260809 12:10:52.358757 9638 tablet_bootstrap.cc:492] T 19d7e57f295343b892ba515f7de56686 P af75f3a57f6f48fba26736ad36aff045: No bootstrap required, opened a new log
I20260809 12:10:52.358924 9638 ts_tablet_manager.cc:1397] T 19d7e57f295343b892ba515f7de56686 P af75f3a57f6f48fba26736ad36aff045: Time spent bootstrapping tablet: real 0.010s user 0.004s sys 0.000s
I20260809 12:10:52.359586 9638 raft_consensus.cc:348] T 19d7e57f295343b892ba515f7de56686 P af75f3a57f6f48fba26736ad36aff045 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "af75f3a57f6f48fba26736ad36aff045" member_type: VOTER last_known_addr { host: "127.0.222.1" port: 41951 } } peers { permanent_uuid: "cd3a4ffa0f4544a1b74b74fe5b6e3711" member_type: VOTER last_known_addr { host: "127.0.222.2" port: 43017 } } peers { permanent_uuid: "040356c8db9f4f8e93ae96f8fcaecf29" member_type: VOTER last_known_addr { host: "127.0.222.3" port: 33697 } }
I20260809 12:10:52.359763 9638 raft_consensus.cc:374] T 19d7e57f295343b892ba515f7de56686 P af75f3a57f6f48fba26736ad36aff045 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260809 12:10:52.359812 9638 raft_consensus.cc:729] T 19d7e57f295343b892ba515f7de56686 P af75f3a57f6f48fba26736ad36aff045 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: af75f3a57f6f48fba26736ad36aff045, State: Initialized, Role: FOLLOWER
I20260809 12:10:52.360042 9638 consensus_queue.cc:260] T 19d7e57f295343b892ba515f7de56686 P af75f3a57f6f48fba26736ad36aff045 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "af75f3a57f6f48fba26736ad36aff045" member_type: VOTER last_known_addr { host: "127.0.222.1" port: 41951 } } peers { permanent_uuid: "cd3a4ffa0f4544a1b74b74fe5b6e3711" member_type: VOTER last_known_addr { host: "127.0.222.2" port: 43017 } } peers { permanent_uuid: "040356c8db9f4f8e93ae96f8fcaecf29" member_type: VOTER last_known_addr { host: "127.0.222.3" port: 33697 } }
I20260809 12:10:52.360236 9640 tablet_bootstrap.cc:654] T 19d7e57f295343b892ba515f7de56686 P 040356c8db9f4f8e93ae96f8fcaecf29: Neither blocks nor log segments found. Creating new log.
I20260809 12:10:52.360667 9638 ts_tablet_manager.cc:1428] T 19d7e57f295343b892ba515f7de56686 P af75f3a57f6f48fba26736ad36aff045: Time spent starting tablet: real 0.002s user 0.000s sys 0.000s
I20260809 12:10:52.360469 9645 tablet_replica.cc:439] T 19d7e57f295343b892ba515f7de56686 P af75f3a57f6f48fba26736ad36aff045: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "af75f3a57f6f48fba26736ad36aff045" member_type: VOTER last_known_addr { host: "127.0.222.1" port: 41951 } } peers { permanent_uuid: "cd3a4ffa0f4544a1b74b74fe5b6e3711" member_type: VOTER last_known_addr { host: "127.0.222.2" port: 43017 } } peers { permanent_uuid: "040356c8db9f4f8e93ae96f8fcaecf29" member_type: VOTER last_known_addr { host: "127.0.222.3" port: 33697 } } }
I20260809 12:10:52.360874 9645 tablet_replica.cc:442] T 19d7e57f295343b892ba515f7de56686 P af75f3a57f6f48fba26736ad36aff045: This TxnStatusTablet replica's current role is: FOLLOWER
I20260809 12:10:52.363997 9640 tablet_bootstrap.cc:492] T 19d7e57f295343b892ba515f7de56686 P 040356c8db9f4f8e93ae96f8fcaecf29: No bootstrap required, opened a new log
I20260809 12:10:52.364221 9640 ts_tablet_manager.cc:1397] T 19d7e57f295343b892ba515f7de56686 P 040356c8db9f4f8e93ae96f8fcaecf29: Time spent bootstrapping tablet: real 0.006s user 0.004s sys 0.000s
I20260809 12:10:52.365362 9639 tablet_bootstrap.cc:492] T 19d7e57f295343b892ba515f7de56686 P cd3a4ffa0f4544a1b74b74fe5b6e3711: No bootstrap required, opened a new log
I20260809 12:10:52.365408 9640 raft_consensus.cc:348] T 19d7e57f295343b892ba515f7de56686 P 040356c8db9f4f8e93ae96f8fcaecf29 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "af75f3a57f6f48fba26736ad36aff045" member_type: VOTER last_known_addr { host: "127.0.222.1" port: 41951 } } peers { permanent_uuid: "cd3a4ffa0f4544a1b74b74fe5b6e3711" member_type: VOTER last_known_addr { host: "127.0.222.2" port: 43017 } } peers { permanent_uuid: "040356c8db9f4f8e93ae96f8fcaecf29" member_type: VOTER last_known_addr { host: "127.0.222.3" port: 33697 } }
I20260809 12:10:52.365658 9639 ts_tablet_manager.cc:1397] T 19d7e57f295343b892ba515f7de56686 P cd3a4ffa0f4544a1b74b74fe5b6e3711: Time spent bootstrapping tablet: real 0.010s user 0.001s sys 0.004s
I20260809 12:10:52.365748 9640 raft_consensus.cc:374] T 19d7e57f295343b892ba515f7de56686 P 040356c8db9f4f8e93ae96f8fcaecf29 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260809 12:10:52.365846 9640 raft_consensus.cc:729] T 19d7e57f295343b892ba515f7de56686 P 040356c8db9f4f8e93ae96f8fcaecf29 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 040356c8db9f4f8e93ae96f8fcaecf29, State: Initialized, Role: FOLLOWER
I20260809 12:10:52.366127 9640 consensus_queue.cc:260] T 19d7e57f295343b892ba515f7de56686 P 040356c8db9f4f8e93ae96f8fcaecf29 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "af75f3a57f6f48fba26736ad36aff045" member_type: VOTER last_known_addr { host: "127.0.222.1" port: 41951 } } peers { permanent_uuid: "cd3a4ffa0f4544a1b74b74fe5b6e3711" member_type: VOTER last_known_addr { host: "127.0.222.2" port: 43017 } } peers { permanent_uuid: "040356c8db9f4f8e93ae96f8fcaecf29" member_type: VOTER last_known_addr { host: "127.0.222.3" port: 33697 } }
I20260809 12:10:52.366425 9639 raft_consensus.cc:348] T 19d7e57f295343b892ba515f7de56686 P cd3a4ffa0f4544a1b74b74fe5b6e3711 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "af75f3a57f6f48fba26736ad36aff045" member_type: VOTER last_known_addr { host: "127.0.222.1" port: 41951 } } peers { permanent_uuid: "cd3a4ffa0f4544a1b74b74fe5b6e3711" member_type: VOTER last_known_addr { host: "127.0.222.2" port: 43017 } } peers { permanent_uuid: "040356c8db9f4f8e93ae96f8fcaecf29" member_type: VOTER last_known_addr { host: "127.0.222.3" port: 33697 } }
I20260809 12:10:52.366671 9639 raft_consensus.cc:374] T 19d7e57f295343b892ba515f7de56686 P cd3a4ffa0f4544a1b74b74fe5b6e3711 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260809 12:10:52.366739 9639 raft_consensus.cc:729] T 19d7e57f295343b892ba515f7de56686 P cd3a4ffa0f4544a1b74b74fe5b6e3711 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: cd3a4ffa0f4544a1b74b74fe5b6e3711, State: Initialized, Role: FOLLOWER
I20260809 12:10:52.366596 9650 tablet_replica.cc:439] T 19d7e57f295343b892ba515f7de56686 P 040356c8db9f4f8e93ae96f8fcaecf29: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "af75f3a57f6f48fba26736ad36aff045" member_type: VOTER last_known_addr { host: "127.0.222.1" port: 41951 } } peers { permanent_uuid: "cd3a4ffa0f4544a1b74b74fe5b6e3711" member_type: VOTER last_known_addr { host: "127.0.222.2" port: 43017 } } peers { permanent_uuid: "040356c8db9f4f8e93ae96f8fcaecf29" member_type: VOTER last_known_addr { host: "127.0.222.3" port: 33697 } } }
I20260809 12:10:52.366926 9650 tablet_replica.cc:442] T 19d7e57f295343b892ba515f7de56686 P 040356c8db9f4f8e93ae96f8fcaecf29: This TxnStatusTablet replica's current role is: FOLLOWER
I20260809 12:10:52.367002 9639 consensus_queue.cc:260] T 19d7e57f295343b892ba515f7de56686 P cd3a4ffa0f4544a1b74b74fe5b6e3711 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "af75f3a57f6f48fba26736ad36aff045" member_type: VOTER last_known_addr { host: "127.0.222.1" port: 41951 } } peers { permanent_uuid: "cd3a4ffa0f4544a1b74b74fe5b6e3711" member_type: VOTER last_known_addr { host: "127.0.222.2" port: 43017 } } peers { permanent_uuid: "040356c8db9f4f8e93ae96f8fcaecf29" member_type: VOTER last_known_addr { host: "127.0.222.3" port: 33697 } }
I20260809 12:10:52.367563 9640 ts_tablet_manager.cc:1428] T 19d7e57f295343b892ba515f7de56686 P 040356c8db9f4f8e93ae96f8fcaecf29: Time spent starting tablet: real 0.003s user 0.002s sys 0.001s
I20260809 12:10:52.367671 9646 tablet_replica.cc:439] T 19d7e57f295343b892ba515f7de56686 P cd3a4ffa0f4544a1b74b74fe5b6e3711: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "af75f3a57f6f48fba26736ad36aff045" member_type: VOTER last_known_addr { host: "127.0.222.1" port: 41951 } } peers { permanent_uuid: "cd3a4ffa0f4544a1b74b74fe5b6e3711" member_type: VOTER last_known_addr { host: "127.0.222.2" port: 43017 } } peers { permanent_uuid: "040356c8db9f4f8e93ae96f8fcaecf29" member_type: VOTER last_known_addr { host: "127.0.222.3" port: 33697 } } }
I20260809 12:10:52.367939 9646 tablet_replica.cc:442] T 19d7e57f295343b892ba515f7de56686 P cd3a4ffa0f4544a1b74b74fe5b6e3711: This TxnStatusTablet replica's current role is: FOLLOWER
I20260809 12:10:52.369406 9639 ts_tablet_manager.cc:1428] T 19d7e57f295343b892ba515f7de56686 P cd3a4ffa0f4544a1b74b74fe5b6e3711: Time spent starting tablet: real 0.004s user 0.000s sys 0.002s
I20260809 12:10:52.754784 9645 raft_consensus.cc:482] T 19d7e57f295343b892ba515f7de56686 P af75f3a57f6f48fba26736ad36aff045 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260809 12:10:52.755129 9645 raft_consensus.cc:504] T 19d7e57f295343b892ba515f7de56686 P af75f3a57f6f48fba26736ad36aff045 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "af75f3a57f6f48fba26736ad36aff045" member_type: VOTER last_known_addr { host: "127.0.222.1" port: 41951 } } peers { permanent_uuid: "cd3a4ffa0f4544a1b74b74fe5b6e3711" member_type: VOTER last_known_addr { host: "127.0.222.2" port: 43017 } } peers { permanent_uuid: "040356c8db9f4f8e93ae96f8fcaecf29" member_type: VOTER last_known_addr { host: "127.0.222.3" port: 33697 } }
I20260809 12:10:52.756810 9645 leader_election.cc:288] T 19d7e57f295343b892ba515f7de56686 P af75f3a57f6f48fba26736ad36aff045 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers cd3a4ffa0f4544a1b74b74fe5b6e3711 (127.0.222.2:43017), 040356c8db9f4f8e93ae96f8fcaecf29 (127.0.222.3:33697)
I20260809 12:10:52.763720 9686 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 12:10:52.757317 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.222.1:45377 (local address 127.0.222.2:43017)
0809 12:10:52.758292 (+ 975us) server_negotiation.cc:206] Beginning negotiation
0809 12:10:52.758299 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0809 12:10:52.758320 (+ 21us) server_negotiation.cc:407] Connection header received
0809 12:10:52.758399 (+ 79us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 12:10:52.758410 (+ 11us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 12:10:52.758466 (+ 56us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 12:10:52.758578 (+ 112us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0809 12:10:52.759182 (+ 604us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 12:10:52.759915 (+ 733us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 12:10:52.761582 (+ 1667us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 12:10:52.762592 (+ 1010us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 12:10:52.762734 (+ 142us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 12:10:52.762847 (+ 113us) server_negotiation.cc:1035] Waiting for connection context
0809 12:10:52.763236 (+ 389us) server_negotiation.cc:299] Negotiation successful
0809 12:10:52.763372 (+ 136us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":839,"thread_start_us":724,"threads_started":1}
I20260809 12:10:52.763774 9685 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 12:10:52.757088 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.222.2:43017 (local address 127.0.222.1:45377)
0809 12:10:52.757609 (+ 521us) negotiation.cc:107] Waiting for socket to connect
0809 12:10:52.757639 (+ 30us) client_negotiation.cc:174] Beginning negotiation
0809 12:10:52.757825 (+ 186us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 12:10:52.758599 (+ 774us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 12:10:52.758603 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 12:10:52.758630 (+ 27us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0809 12:10:52.759031 (+ 401us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 12:10:52.759044 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 12:10:52.760069 (+ 1025us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 12:10:52.760074 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 12:10:52.761418 (+ 1344us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 12:10:52.761432 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 12:10:52.762756 (+ 1324us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 12:10:52.762761 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 12:10:52.762937 (+ 176us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 12:10:52.763041 (+ 104us) client_negotiation.cc:769] Sending connection context
0809 12:10:52.763188 (+ 147us) client_negotiation.cc:240] Negotiation successful
0809 12:10:52.763352 (+ 164us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":360,"mutex_wait_us":21,"thread_start_us":249,"threads_started":1}
I20260809 12:10:52.764323 9688 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 12:10:52.757088 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.222.3:33697 (local address 127.0.222.1:40625)
0809 12:10:52.758126 (+ 1038us) negotiation.cc:107] Waiting for socket to connect
0809 12:10:52.758144 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0809 12:10:52.758275 (+ 131us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 12:10:52.758652 (+ 377us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 12:10:52.758656 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 12:10:52.758704 (+ 48us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0809 12:10:52.759160 (+ 456us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 12:10:52.759170 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 12:10:52.760226 (+ 1056us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 12:10:52.760232 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 12:10:52.761773 (+ 1541us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 12:10:52.761786 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 12:10:52.763320 (+ 1534us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 12:10:52.763389 (+ 69us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 12:10:52.763588 (+ 199us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 12:10:52.763613 (+ 25us) client_negotiation.cc:769] Sending connection context
0809 12:10:52.763733 (+ 120us) client_negotiation.cc:240] Negotiation successful
0809 12:10:52.763878 (+ 145us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":909,"mutex_wait_us":417,"thread_start_us":368,"threads_started":1}
I20260809 12:10:52.764429 9687 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 12:10:52.757332 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.222.1:40625 (local address 127.0.222.3:33697)
0809 12:10:52.757948 (+ 616us) server_negotiation.cc:206] Beginning negotiation
0809 12:10:52.757955 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0809 12:10:52.758260 (+ 305us) server_negotiation.cc:407] Connection header received
0809 12:10:52.758408 (+ 148us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 12:10:52.758412 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 12:10:52.758488 (+ 76us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 12:10:52.758629 (+ 141us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0809 12:10:52.759345 (+ 716us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 12:10:52.760073 (+ 728us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 12:10:52.761943 (+ 1870us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 12:10:52.763022 (+ 1079us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 12:10:52.763197 (+ 175us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 12:10:52.763323 (+ 126us) server_negotiation.cc:1035] Waiting for connection context
0809 12:10:52.763773 (+ 450us) server_negotiation.cc:299] Negotiation successful
0809 12:10:52.764113 (+ 340us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":483,"thread_start_us":313,"threads_started":1}
I20260809 12:10:52.764613 9409 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "19d7e57f295343b892ba515f7de56686" candidate_uuid: "af75f3a57f6f48fba26736ad36aff045" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "cd3a4ffa0f4544a1b74b74fe5b6e3711" is_pre_election: true
I20260809 12:10:52.764895 9409 raft_consensus.cc:2454] T 19d7e57f295343b892ba515f7de56686 P cd3a4ffa0f4544a1b74b74fe5b6e3711 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate af75f3a57f6f48fba26736ad36aff045 in term 0.
I20260809 12:10:52.765216 9558 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "19d7e57f295343b892ba515f7de56686" candidate_uuid: "af75f3a57f6f48fba26736ad36aff045" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "040356c8db9f4f8e93ae96f8fcaecf29" is_pre_election: true
I20260809 12:10:52.765503 9558 raft_consensus.cc:2454] T 19d7e57f295343b892ba515f7de56686 P 040356c8db9f4f8e93ae96f8fcaecf29 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate af75f3a57f6f48fba26736ad36aff045 in term 0.
I20260809 12:10:52.765491 9180 leader_election.cc:302] T 19d7e57f295343b892ba515f7de56686 P af75f3a57f6f48fba26736ad36aff045 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: af75f3a57f6f48fba26736ad36aff045, cd3a4ffa0f4544a1b74b74fe5b6e3711; no voters:
I20260809 12:10:52.765944 9645 raft_consensus.cc:2781] T 19d7e57f295343b892ba515f7de56686 P af75f3a57f6f48fba26736ad36aff045 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260809 12:10:52.766060 9645 raft_consensus.cc:482] T 19d7e57f295343b892ba515f7de56686 P af75f3a57f6f48fba26736ad36aff045 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260809 12:10:52.766189 9645 raft_consensus.cc:3037] T 19d7e57f295343b892ba515f7de56686 P af75f3a57f6f48fba26736ad36aff045 [term 0 FOLLOWER]: Advancing to term 1
I20260809 12:10:52.768707 9645 raft_consensus.cc:504] T 19d7e57f295343b892ba515f7de56686 P af75f3a57f6f48fba26736ad36aff045 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "af75f3a57f6f48fba26736ad36aff045" member_type: VOTER last_known_addr { host: "127.0.222.1" port: 41951 } } peers { permanent_uuid: "cd3a4ffa0f4544a1b74b74fe5b6e3711" member_type: VOTER last_known_addr { host: "127.0.222.2" port: 43017 } } peers { permanent_uuid: "040356c8db9f4f8e93ae96f8fcaecf29" member_type: VOTER last_known_addr { host: "127.0.222.3" port: 33697 } }
I20260809 12:10:52.769359 9645 leader_election.cc:288] T 19d7e57f295343b892ba515f7de56686 P af75f3a57f6f48fba26736ad36aff045 [CANDIDATE]: Term 1 election: Requested vote from peers cd3a4ffa0f4544a1b74b74fe5b6e3711 (127.0.222.2:43017), 040356c8db9f4f8e93ae96f8fcaecf29 (127.0.222.3:33697)
I20260809 12:10:52.769798 9409 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "19d7e57f295343b892ba515f7de56686" candidate_uuid: "af75f3a57f6f48fba26736ad36aff045" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "cd3a4ffa0f4544a1b74b74fe5b6e3711"
I20260809 12:10:52.770011 9409 raft_consensus.cc:3037] T 19d7e57f295343b892ba515f7de56686 P cd3a4ffa0f4544a1b74b74fe5b6e3711 [term 0 FOLLOWER]: Advancing to term 1
I20260809 12:10:52.769977 9558 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "19d7e57f295343b892ba515f7de56686" candidate_uuid: "af75f3a57f6f48fba26736ad36aff045" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "040356c8db9f4f8e93ae96f8fcaecf29"
I20260809 12:10:52.770253 9558 raft_consensus.cc:3037] T 19d7e57f295343b892ba515f7de56686 P 040356c8db9f4f8e93ae96f8fcaecf29 [term 0 FOLLOWER]: Advancing to term 1
I20260809 12:10:52.772284 9409 raft_consensus.cc:2454] T 19d7e57f295343b892ba515f7de56686 P cd3a4ffa0f4544a1b74b74fe5b6e3711 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate af75f3a57f6f48fba26736ad36aff045 in term 1.
I20260809 12:10:52.772574 9558 raft_consensus.cc:2454] T 19d7e57f295343b892ba515f7de56686 P 040356c8db9f4f8e93ae96f8fcaecf29 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate af75f3a57f6f48fba26736ad36aff045 in term 1.
I20260809 12:10:52.772676 9180 leader_election.cc:302] T 19d7e57f295343b892ba515f7de56686 P af75f3a57f6f48fba26736ad36aff045 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: af75f3a57f6f48fba26736ad36aff045, cd3a4ffa0f4544a1b74b74fe5b6e3711; no voters:
I20260809 12:10:52.773025 9645 raft_consensus.cc:2781] T 19d7e57f295343b892ba515f7de56686 P af75f3a57f6f48fba26736ad36aff045 [term 1 FOLLOWER]: Leader election won for term 1
I20260809 12:10:52.773479 9645 raft_consensus.cc:686] T 19d7e57f295343b892ba515f7de56686 P af75f3a57f6f48fba26736ad36aff045 [term 1 LEADER]: Becoming Leader. State: Replica: af75f3a57f6f48fba26736ad36aff045, State: Running, Role: LEADER
I20260809 12:10:52.773910 9645 consensus_queue.cc:237] T 19d7e57f295343b892ba515f7de56686 P af75f3a57f6f48fba26736ad36aff045 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "af75f3a57f6f48fba26736ad36aff045" member_type: VOTER last_known_addr { host: "127.0.222.1" port: 41951 } } peers { permanent_uuid: "cd3a4ffa0f4544a1b74b74fe5b6e3711" member_type: VOTER last_known_addr { host: "127.0.222.2" port: 43017 } } peers { permanent_uuid: "040356c8db9f4f8e93ae96f8fcaecf29" member_type: VOTER last_known_addr { host: "127.0.222.3" port: 33697 } }
I20260809 12:10:52.775978 9689 tablet_replica.cc:439] T 19d7e57f295343b892ba515f7de56686 P af75f3a57f6f48fba26736ad36aff045: TxnStatusTablet state changed. Reason: New leader af75f3a57f6f48fba26736ad36aff045. Latest consensus state: current_term: 1 leader_uuid: "af75f3a57f6f48fba26736ad36aff045" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "af75f3a57f6f48fba26736ad36aff045" member_type: VOTER last_known_addr { host: "127.0.222.1" port: 41951 } } peers { permanent_uuid: "cd3a4ffa0f4544a1b74b74fe5b6e3711" member_type: VOTER last_known_addr { host: "127.0.222.2" port: 43017 } } peers { permanent_uuid: "040356c8db9f4f8e93ae96f8fcaecf29" member_type: VOTER last_known_addr { host: "127.0.222.3" port: 33697 } } }
I20260809 12:10:52.776440 9689 tablet_replica.cc:442] T 19d7e57f295343b892ba515f7de56686 P af75f3a57f6f48fba26736ad36aff045: This TxnStatusTablet replica's current role is: LEADER
I20260809 12:10:52.777191 9691 txn_status_manager.cc:873] Waiting until node catch up with all replicated operations in previous term...
I20260809 12:10:52.777454 9691 txn_status_manager.cc:929] Loading transaction status metadata into memory...
I20260809 12:10:52.777868 9084 catalog_manager.cc:5495] T 19d7e57f295343b892ba515f7de56686 P af75f3a57f6f48fba26736ad36aff045 reported cstate change: term changed from 0 to 1, leader changed from <none> to af75f3a57f6f48fba26736ad36aff045 (127.0.222.1). New cstate: current_term: 1 leader_uuid: "af75f3a57f6f48fba26736ad36aff045" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "af75f3a57f6f48fba26736ad36aff045" member_type: VOTER last_known_addr { host: "127.0.222.1" port: 41951 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "cd3a4ffa0f4544a1b74b74fe5b6e3711" member_type: VOTER last_known_addr { host: "127.0.222.2" port: 43017 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "040356c8db9f4f8e93ae96f8fcaecf29" member_type: VOTER last_known_addr { host: "127.0.222.3" port: 33697 } health_report { overall_health: UNKNOWN } } }
I20260809 12:10:52.829728 9681 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 12:10:52.821139 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.222.1:41951 (local address 127.0.0.1:49634)
0809 12:10:52.821345 (+ 206us) negotiation.cc:107] Waiting for socket to connect
0809 12:10:52.821364 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0809 12:10:52.821493 (+ 129us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 12:10:52.823990 (+ 2497us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 12:10:52.823995 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 12:10:52.824010 (+ 15us) client_negotiation.cc:189] Negotiated authn=TOKEN
0809 12:10:52.824249 (+ 239us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 12:10:52.824261 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 12:10:52.825889 (+ 1628us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 12:10:52.825894 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 12:10:52.826873 (+ 979us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 12:10:52.826885 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 12:10:52.827903 (+ 1018us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 12:10:52.827907 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 12:10:52.828081 (+ 174us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 12:10:52.828136 (+ 55us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0809 12:10:52.829306 (+ 1170us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0809 12:10:52.829321 (+ 15us) client_negotiation.cc:769] Sending connection context
0809 12:10:52.829375 (+ 54us) client_negotiation.cc:240] Negotiation successful
0809 12:10:52.829497 (+ 122us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":41}
I20260809 12:10:52.830168 9694 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 12:10:52.821156 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49634 (local address 127.0.222.1:41951)
0809 12:10:52.823483 (+ 2327us) server_negotiation.cc:206] Beginning negotiation
0809 12:10:52.823490 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0809 12:10:52.823516 (+ 26us) server_negotiation.cc:407] Connection header received
0809 12:10:52.823607 (+ 91us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 12:10:52.823611 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 12:10:52.823740 (+ 129us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 12:10:52.823946 (+ 206us) server_negotiation.cc:226] Negotiated authn=TOKEN
0809 12:10:52.824474 (+ 528us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 12:10:52.825733 (+ 1259us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 12:10:52.827055 (+ 1322us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 12:10:52.827742 (+ 687us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 12:10:52.827896 (+ 154us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 12:10:52.828366 (+ 470us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0809 12:10:52.829196 (+ 830us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0809 12:10:52.829354 (+ 158us) server_negotiation.cc:1035] Waiting for connection context
0809 12:10:52.829404 (+ 50us) server_negotiation.cc:299] Negotiation successful
0809 12:10:52.829496 (+ 92us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":2192,"thread_start_us":2018,"threads_started":1}
I20260809 12:10:52.835186 9409 raft_consensus.cc:1261] T 19d7e57f295343b892ba515f7de56686 P cd3a4ffa0f4544a1b74b74fe5b6e3711 [term 1 FOLLOWER]: Refusing update from remote peer af75f3a57f6f48fba26736ad36aff045: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260809 12:10:52.835306 9558 raft_consensus.cc:1261] T 19d7e57f295343b892ba515f7de56686 P 040356c8db9f4f8e93ae96f8fcaecf29 [term 1 FOLLOWER]: Refusing update from remote peer af75f3a57f6f48fba26736ad36aff045: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260809 12:10:52.836097 9689 consensus_queue.cc:1035] T 19d7e57f295343b892ba515f7de56686 P af75f3a57f6f48fba26736ad36aff045 [LEADER]: Connected to new peer: Peer: permanent_uuid: "cd3a4ffa0f4544a1b74b74fe5b6e3711" member_type: VOTER last_known_addr { host: "127.0.222.2" port: 43017 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260809 12:10:52.836404 9645 consensus_queue.cc:1035] T 19d7e57f295343b892ba515f7de56686 P af75f3a57f6f48fba26736ad36aff045 [LEADER]: Connected to new peer: Peer: permanent_uuid: "040356c8db9f4f8e93ae96f8fcaecf29" member_type: VOTER last_known_addr { host: "127.0.222.3" port: 33697 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260809 12:10:52.840248 9646 tablet_replica.cc:439] T 19d7e57f295343b892ba515f7de56686 P cd3a4ffa0f4544a1b74b74fe5b6e3711: TxnStatusTablet state changed. Reason: New leader af75f3a57f6f48fba26736ad36aff045. Latest consensus state: current_term: 1 leader_uuid: "af75f3a57f6f48fba26736ad36aff045" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "af75f3a57f6f48fba26736ad36aff045" member_type: VOTER last_known_addr { host: "127.0.222.1" port: 41951 } } peers { permanent_uuid: "cd3a4ffa0f4544a1b74b74fe5b6e3711" member_type: VOTER last_known_addr { host: "127.0.222.2" port: 43017 } } peers { permanent_uuid: "040356c8db9f4f8e93ae96f8fcaecf29" member_type: VOTER last_known_addr { host: "127.0.222.3" port: 33697 } } }
I20260809 12:10:52.840533 9646 tablet_replica.cc:442] T 19d7e57f295343b892ba515f7de56686 P cd3a4ffa0f4544a1b74b74fe5b6e3711: This TxnStatusTablet replica's current role is: FOLLOWER
I20260809 12:10:52.840741 9646 consensus_queue.cc:1035] T c105f50bcad744b0900363b66ec4cf33 P cd3a4ffa0f4544a1b74b74fe5b6e3711 [LEADER]: Connected to new peer: Peer: permanent_uuid: "af75f3a57f6f48fba26736ad36aff045" member_type: VOTER last_known_addr { host: "127.0.222.1" port: 41951 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260809 12:10:52.847556 9650 tablet_replica.cc:439] T 19d7e57f295343b892ba515f7de56686 P 040356c8db9f4f8e93ae96f8fcaecf29: TxnStatusTablet state changed. Reason: New leader af75f3a57f6f48fba26736ad36aff045. Latest consensus state: current_term: 1 leader_uuid: "af75f3a57f6f48fba26736ad36aff045" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "af75f3a57f6f48fba26736ad36aff045" member_type: VOTER last_known_addr { host: "127.0.222.1" port: 41951 } } peers { permanent_uuid: "cd3a4ffa0f4544a1b74b74fe5b6e3711" member_type: VOTER last_known_addr { host: "127.0.222.2" port: 43017 } } peers { permanent_uuid: "040356c8db9f4f8e93ae96f8fcaecf29" member_type: VOTER last_known_addr { host: "127.0.222.3" port: 33697 } } }
I20260809 12:10:52.847923 9650 tablet_replica.cc:442] T 19d7e57f295343b892ba515f7de56686 P 040356c8db9f4f8e93ae96f8fcaecf29: This TxnStatusTablet replica's current role is: FOLLOWER
I20260809 12:10:52.862694 9689 tablet_replica.cc:439] T 19d7e57f295343b892ba515f7de56686 P af75f3a57f6f48fba26736ad36aff045: TxnStatusTablet state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "af75f3a57f6f48fba26736ad36aff045" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "af75f3a57f6f48fba26736ad36aff045" member_type: VOTER last_known_addr { host: "127.0.222.1" port: 41951 } } peers { permanent_uuid: "cd3a4ffa0f4544a1b74b74fe5b6e3711" member_type: VOTER last_known_addr { host: "127.0.222.2" port: 43017 } } peers { permanent_uuid: "040356c8db9f4f8e93ae96f8fcaecf29" member_type: VOTER last_known_addr { host: "127.0.222.3" port: 33697 } } }
I20260809 12:10:52.862996 9689 tablet_replica.cc:442] T 19d7e57f295343b892ba515f7de56686 P af75f3a57f6f48fba26736ad36aff045: This TxnStatusTablet replica's current role is: LEADER
I20260809 12:10:52.862320 9698 mvcc.cc:205] Tried to move back new op lower bound from 7316592446804807680 to 7316592446565883904. Current Snapshot: MvccSnapshot[applied={T|T < 7316592446804807680}]
I20260809 12:10:52.862522 9645 tablet_replica.cc:439] T 19d7e57f295343b892ba515f7de56686 P af75f3a57f6f48fba26736ad36aff045: TxnStatusTablet state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "af75f3a57f6f48fba26736ad36aff045" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "af75f3a57f6f48fba26736ad36aff045" member_type: VOTER last_known_addr { host: "127.0.222.1" port: 41951 } } peers { permanent_uuid: "cd3a4ffa0f4544a1b74b74fe5b6e3711" member_type: VOTER last_known_addr { host: "127.0.222.2" port: 43017 } } peers { permanent_uuid: "040356c8db9f4f8e93ae96f8fcaecf29" member_type: VOTER last_known_addr { host: "127.0.222.3" port: 33697 } } }
I20260809 12:10:52.866950 9645 tablet_replica.cc:442] T 19d7e57f295343b892ba515f7de56686 P af75f3a57f6f48fba26736ad36aff045: This TxnStatusTablet replica's current role is: LEADER
I20260809 12:10:52.870455 9646 tablet_replica.cc:439] T 19d7e57f295343b892ba515f7de56686 P cd3a4ffa0f4544a1b74b74fe5b6e3711: TxnStatusTablet state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "af75f3a57f6f48fba26736ad36aff045" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "af75f3a57f6f48fba26736ad36aff045" member_type: VOTER last_known_addr { host: "127.0.222.1" port: 41951 } } peers { permanent_uuid: "cd3a4ffa0f4544a1b74b74fe5b6e3711" member_type: VOTER last_known_addr { host: "127.0.222.2" port: 43017 } } peers { permanent_uuid: "040356c8db9f4f8e93ae96f8fcaecf29" member_type: VOTER last_known_addr { host: "127.0.222.3" port: 33697 } } }
I20260809 12:10:52.870730 9646 tablet_replica.cc:442] T 19d7e57f295343b892ba515f7de56686 P cd3a4ffa0f4544a1b74b74fe5b6e3711: This TxnStatusTablet replica's current role is: FOLLOWER
I20260809 12:10:52.871794 9650 tablet_replica.cc:439] T 19d7e57f295343b892ba515f7de56686 P 040356c8db9f4f8e93ae96f8fcaecf29: TxnStatusTablet state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "af75f3a57f6f48fba26736ad36aff045" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "af75f3a57f6f48fba26736ad36aff045" member_type: VOTER last_known_addr { host: "127.0.222.1" port: 41951 } } peers { permanent_uuid: "cd3a4ffa0f4544a1b74b74fe5b6e3711" member_type: VOTER last_known_addr { host: "127.0.222.2" port: 43017 } } peers { permanent_uuid: "040356c8db9f4f8e93ae96f8fcaecf29" member_type: VOTER last_known_addr { host: "127.0.222.3" port: 33697 } } }
I20260809 12:10:52.872107 9650 tablet_replica.cc:442] T 19d7e57f295343b892ba515f7de56686 P 040356c8db9f4f8e93ae96f8fcaecf29: This TxnStatusTablet replica's current role is: FOLLOWER
I20260809 12:10:52.873310 9646 consensus_queue.cc:1035] T c105f50bcad744b0900363b66ec4cf33 P cd3a4ffa0f4544a1b74b74fe5b6e3711 [LEADER]: Connected to new peer: Peer: permanent_uuid: "040356c8db9f4f8e93ae96f8fcaecf29" member_type: VOTER last_known_addr { host: "127.0.222.3" port: 33697 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260809 12:10:52.894992 9686 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 12:10:52.882733 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40530 (local address 127.0.222.2:43017)
0809 12:10:52.882949 (+ 216us) server_negotiation.cc:206] Beginning negotiation
0809 12:10:52.882956 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0809 12:10:52.889187 (+ 6231us) server_negotiation.cc:407] Connection header received
0809 12:10:52.889459 (+ 272us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 12:10:52.889471 (+ 12us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 12:10:52.889562 (+ 91us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 12:10:52.889723 (+ 161us) server_negotiation.cc:226] Negotiated authn=TOKEN
0809 12:10:52.890324 (+ 601us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 12:10:52.891378 (+ 1054us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 12:10:52.892759 (+ 1381us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 12:10:52.893199 (+ 440us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 12:10:52.893333 (+ 134us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 12:10:52.893774 (+ 441us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0809 12:10:52.894338 (+ 564us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0809 12:10:52.894495 (+ 157us) server_negotiation.cc:1035] Waiting for connection context
0809 12:10:52.894593 (+ 98us) server_negotiation.cc:299] Negotiation successful
0809 12:10:52.894675 (+ 82us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":71}
I20260809 12:10:52.912158 9694 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 12:10:52.905673 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.222.2:60841 (local address 127.0.222.1:41951)
0809 12:10:52.905907 (+ 234us) server_negotiation.cc:206] Beginning negotiation
0809 12:10:52.905913 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0809 12:10:52.906448 (+ 535us) server_negotiation.cc:407] Connection header received
0809 12:10:52.906652 (+ 204us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 12:10:52.906656 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 12:10:52.906723 (+ 67us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 12:10:52.906798 (+ 75us) server_negotiation.cc:226] Negotiated authn=TOKEN
0809 12:10:52.907316 (+ 518us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 12:10:52.908113 (+ 797us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 12:10:52.910429 (+ 2316us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 12:10:52.910853 (+ 424us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 12:10:52.911003 (+ 150us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 12:10:52.911518 (+ 515us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0809 12:10:52.911615 (+ 97us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0809 12:10:52.911690 (+ 75us) server_negotiation.cc:1035] Waiting for connection context
0809 12:10:52.911872 (+ 182us) server_negotiation.cc:299] Negotiation successful
0809 12:10:52.911959 (+ 87us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":79}
I20260809 12:10:52.912639 9715 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 12:10:52.905572 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.222.1:41951 (local address 127.0.222.2:60841)
0809 12:10:52.906235 (+ 663us) negotiation.cc:107] Waiting for socket to connect
0809 12:10:52.906259 (+ 24us) client_negotiation.cc:174] Beginning negotiation
0809 12:10:52.906520 (+ 261us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 12:10:52.906847 (+ 327us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 12:10:52.906852 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 12:10:52.906867 (+ 15us) client_negotiation.cc:189] Negotiated authn=TOKEN
0809 12:10:52.907175 (+ 308us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 12:10:52.907188 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 12:10:52.908294 (+ 1106us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 12:10:52.908302 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 12:10:52.910277 (+ 1975us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 12:10:52.910293 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 12:10:52.911051 (+ 758us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 12:10:52.911059 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 12:10:52.911269 (+ 210us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 12:10:52.911344 (+ 75us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0809 12:10:52.911725 (+ 381us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0809 12:10:52.911738 (+ 13us) client_negotiation.cc:769] Sending connection context
0809 12:10:52.911844 (+ 106us) client_negotiation.cc:240] Negotiation successful
0809 12:10:52.912012 (+ 168us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":469,"thread_start_us":310,"threads_started":1}
I20260809 12:10:52.958195 9686 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 12:10:52.944956 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.222.1:35989 (local address 127.0.222.2:43017)
0809 12:10:52.945138 (+ 182us) server_negotiation.cc:206] Beginning negotiation
0809 12:10:52.945147 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0809 12:10:52.951426 (+ 6279us) server_negotiation.cc:407] Connection header received
0809 12:10:52.951682 (+ 256us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 12:10:52.951686 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 12:10:52.951766 (+ 80us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 12:10:52.951841 (+ 75us) server_negotiation.cc:226] Negotiated authn=TOKEN
0809 12:10:52.952610 (+ 769us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 12:10:52.953365 (+ 755us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 12:10:52.955847 (+ 2482us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 12:10:52.956540 (+ 693us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 12:10:52.956650 (+ 110us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 12:10:52.957397 (+ 747us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0809 12:10:52.957509 (+ 112us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0809 12:10:52.957583 (+ 74us) server_negotiation.cc:1035] Waiting for connection context
0809 12:10:52.957763 (+ 180us) server_negotiation.cc:299] Negotiation successful
0809 12:10:52.957888 (+ 125us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":64}
I20260809 12:10:52.958384 9717 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 12:10:52.944926 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.222.2:43017 (local address 127.0.222.1:35989)
0809 12:10:52.951313 (+ 6387us) negotiation.cc:107] Waiting for socket to connect
0809 12:10:52.951342 (+ 29us) client_negotiation.cc:174] Beginning negotiation
0809 12:10:52.951511 (+ 169us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 12:10:52.951893 (+ 382us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 12:10:52.951899 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 12:10:52.951913 (+ 14us) client_negotiation.cc:189] Negotiated authn=TOKEN
0809 12:10:52.952310 (+ 397us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 12:10:52.952323 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 12:10:52.953502 (+ 1179us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 12:10:52.953510 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 12:10:52.955629 (+ 2119us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 12:10:52.955646 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 12:10:52.956909 (+ 1263us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 12:10:52.956916 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 12:10:52.957124 (+ 208us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 12:10:52.957206 (+ 82us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0809 12:10:52.957619 (+ 413us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0809 12:10:52.957639 (+ 20us) client_negotiation.cc:769] Sending connection context
0809 12:10:52.957737 (+ 98us) client_negotiation.cc:240] Negotiation successful
0809 12:10:52.957910 (+ 173us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":6131,"thread_start_us":5952,"threads_started":1}
I20260809 12:10:53.012247 9084 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:53632:
table { table_name: "test-table" } modify_external_catalogs: true
I20260809 12:10:53.012550 9084 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:53632:
table { table_name: "test-table" } modify_external_catalogs: true
I20260809 12:10:53.016543 9084 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 7873beb5ce214b9d8d238a51db1accb1: Sending DeleteTablet for 3 replicas of tablet c105f50bcad744b0900363b66ec4cf33
I20260809 12:10:53.017513 9234 tablet_service.cc:1514] Processing DeleteTablet for tablet c105f50bcad744b0900363b66ec4cf33 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-09 12:10:53 UTC) from {username='oryx', principal='oryx/127.0.222.62@KRBTEST.COM'} at 127.0.0.1:49628
I20260809 12:10:53.017709 9386 tablet_service.cc:1514] Processing DeleteTablet for tablet c105f50bcad744b0900363b66ec4cf33 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-09 12:10:53 UTC) from {username='oryx', principal='oryx/127.0.222.62@KRBTEST.COM'} at 127.0.0.1:40518
I20260809 12:10:53.018739 9537 tablet_service.cc:1514] Processing DeleteTablet for tablet c105f50bcad744b0900363b66ec4cf33 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-09 12:10:53 UTC) from {username='oryx', principal='oryx/127.0.222.62@KRBTEST.COM'} at 127.0.0.1:53446
I20260809 12:10:53.028693 9723 tablet_replica.cc:330] T c105f50bcad744b0900363b66ec4cf33 P 040356c8db9f4f8e93ae96f8fcaecf29: stopping tablet replica
I20260809 12:10:53.029738 9723 raft_consensus.cc:2229] T c105f50bcad744b0900363b66ec4cf33 P 040356c8db9f4f8e93ae96f8fcaecf29 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260809 12:10:53.030488 9723 raft_consensus.cc:2258] T c105f50bcad744b0900363b66ec4cf33 P 040356c8db9f4f8e93ae96f8fcaecf29 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260809 12:10:53.032145 9723 ts_tablet_manager.cc:1905] T c105f50bcad744b0900363b66ec4cf33 P 040356c8db9f4f8e93ae96f8fcaecf29: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260809 12:10:53.032745 9721 tablet_replica.cc:330] T c105f50bcad744b0900363b66ec4cf33 P af75f3a57f6f48fba26736ad36aff045: stopping tablet replica
I20260809 12:10:53.033233 9721 raft_consensus.cc:2229] T c105f50bcad744b0900363b66ec4cf33 P af75f3a57f6f48fba26736ad36aff045 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260809 12:10:53.033593 9721 raft_consensus.cc:2258] T c105f50bcad744b0900363b66ec4cf33 P af75f3a57f6f48fba26736ad36aff045 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260809 12:10:53.037114 9722 tablet_replica.cc:330] T c105f50bcad744b0900363b66ec4cf33 P cd3a4ffa0f4544a1b74b74fe5b6e3711: stopping tablet replica
I20260809 12:10:53.037384 9722 raft_consensus.cc:2229] T c105f50bcad744b0900363b66ec4cf33 P cd3a4ffa0f4544a1b74b74fe5b6e3711 [term 1 LEADER]: Raft consensus shutting down.
I20260809 12:10:53.038137 9722 raft_consensus.cc:2258] T c105f50bcad744b0900363b66ec4cf33 P cd3a4ffa0f4544a1b74b74fe5b6e3711 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260809 12:10:53.039090 9721 ts_tablet_manager.cc:1905] T c105f50bcad744b0900363b66ec4cf33 P af75f3a57f6f48fba26736ad36aff045: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260809 12:10:53.040251 9722 ts_tablet_manager.cc:1905] T c105f50bcad744b0900363b66ec4cf33 P cd3a4ffa0f4544a1b74b74fe5b6e3711: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260809 12:10:53.062426 9722 ts_tablet_manager.cc:1918] T c105f50bcad744b0900363b66ec4cf33 P cd3a4ffa0f4544a1b74b74fe5b6e3711: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260809 12:10:53.062711 9722 log.cc:1196] T c105f50bcad744b0900363b66ec4cf33 P cd3a4ffa0f4544a1b74b74fe5b6e3711: Deleting WAL directory at /tmp/dist-test-taskgH5Pjm/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786277382073938-888-0/minicluster-data/ts-1/wal/wals/c105f50bcad744b0900363b66ec4cf33
I20260809 12:10:53.063450 9722 ts_tablet_manager.cc:1939] T c105f50bcad744b0900363b66ec4cf33 P cd3a4ffa0f4544a1b74b74fe5b6e3711: Deleting consensus metadata
I20260809 12:10:53.064793 9068 catalog_manager.cc:4841] TS cd3a4ffa0f4544a1b74b74fe5b6e3711 (127.0.222.2:43017): tablet c105f50bcad744b0900363b66ec4cf33 (table test-table [id=fe2ecdf3ef1e468194e0999a3be19b18]) successfully deleted
I20260809 12:10:53.070384 9721 ts_tablet_manager.cc:1918] T c105f50bcad744b0900363b66ec4cf33 P af75f3a57f6f48fba26736ad36aff045: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260809 12:10:53.070572 9721 log.cc:1196] T c105f50bcad744b0900363b66ec4cf33 P af75f3a57f6f48fba26736ad36aff045: Deleting WAL directory at /tmp/dist-test-taskgH5Pjm/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786277382073938-888-0/minicluster-data/ts-0/wal/wals/c105f50bcad744b0900363b66ec4cf33
I20260809 12:10:53.071171 9721 ts_tablet_manager.cc:1939] T c105f50bcad744b0900363b66ec4cf33 P af75f3a57f6f48fba26736ad36aff045: Deleting consensus metadata
I20260809 12:10:53.072573 9069 catalog_manager.cc:4841] TS af75f3a57f6f48fba26736ad36aff045 (127.0.222.1:41951): tablet c105f50bcad744b0900363b66ec4cf33 (table test-table [id=fe2ecdf3ef1e468194e0999a3be19b18]) successfully deleted
I20260809 12:10:53.074460 9723 ts_tablet_manager.cc:1918] T c105f50bcad744b0900363b66ec4cf33 P 040356c8db9f4f8e93ae96f8fcaecf29: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260809 12:10:53.074644 9723 log.cc:1196] T c105f50bcad744b0900363b66ec4cf33 P 040356c8db9f4f8e93ae96f8fcaecf29: Deleting WAL directory at /tmp/dist-test-taskgH5Pjm/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786277382073938-888-0/minicluster-data/ts-2/wal/wals/c105f50bcad744b0900363b66ec4cf33
I20260809 12:10:53.075225 9723 ts_tablet_manager.cc:1939] T c105f50bcad744b0900363b66ec4cf33 P 040356c8db9f4f8e93ae96f8fcaecf29: Deleting consensus metadata
I20260809 12:10:53.076417 9071 catalog_manager.cc:4841] TS 040356c8db9f4f8e93ae96f8fcaecf29 (127.0.222.3:33697): tablet c105f50bcad744b0900363b66ec4cf33 (table test-table [id=fe2ecdf3ef1e468194e0999a3be19b18]) successfully deleted
Aug 09 12:10:53 dist-test-slave-2w3w krb5kdc[9036](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786277453, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
Aug 09 12:10:53 dist-test-slave-2w3w krb5kdc[9036](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, test-admin@KRBTEST.COM for kudu/127.0.222.62@KRBTEST.COM, Server not found in Kerberos database
Aug 09 12:10:53 dist-test-slave-2w3w krb5kdc[9036](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, test-admin@KRBTEST.COM for kudu/127.0.222.62@KRBTEST.COM, Server not found in Kerberos database
W20260809 12:10:53.133325 9732 negotiation.cc:337] Failed RPC negotiation. Trace:
0809 12:10:53.104286 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53638 (local address 127.0.222.62:36963)
0809 12:10:53.119887 (+ 15601us) server_negotiation.cc:206] Beginning negotiation
0809 12:10:53.119894 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0809 12:10:53.119921 (+ 27us) server_negotiation.cc:407] Connection header received
0809 12:10:53.120134 (+ 213us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 12:10:53.120139 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 12:10:53.120241 (+ 102us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 12:10:53.120582 (+ 341us) server_negotiation.cc:226] Negotiated authn=SASL
0809 12:10:53.122006 (+ 1424us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 12:10:53.122855 (+ 849us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 12:10:53.124168 (+ 1313us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 12:10:53.124612 (+ 444us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 12:10:53.124759 (+ 147us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 12:10:53.132980 (+ 8221us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:53638: BlockingRecv error: recv got EOF from 127.0.0.1:53638 (error 108)
Metrics: {"server-negotiator.queue_time_us":15353,"thread_start_us":15647,"threads_started":1}
Aug 09 12:10:53 dist-test-slave-2w3w krb5kdc[9036](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786277453, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.0.222.62@KRBTEST.COM
I20260809 12:10:53.148144 9732 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 12:10:53.133434 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53650 (local address 127.0.222.62:36963)
0809 12:10:53.133919 (+ 485us) server_negotiation.cc:206] Beginning negotiation
0809 12:10:53.133927 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0809 12:10:53.134723 (+ 796us) server_negotiation.cc:407] Connection header received
0809 12:10:53.135012 (+ 289us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 12:10:53.135021 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 12:10:53.135085 (+ 64us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 12:10:53.135189 (+ 104us) server_negotiation.cc:226] Negotiated authn=SASL
0809 12:10:53.136331 (+ 1142us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 12:10:53.137034 (+ 703us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 12:10:53.138255 (+ 1221us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 12:10:53.138709 (+ 454us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 12:10:53.138853 (+ 144us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 12:10:53.142582 (+ 3729us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 12:10:53.142586 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 12:10:53.142590 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0809 12:10:53.142622 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0809 12:10:53.145386 (+ 2764us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 12:10:53.146036 (+ 650us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 12:10:53.146043 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 12:10:53.146047 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0809 12:10:53.146189 (+ 142us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 12:10:53.146562 (+ 373us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 12:10:53.146567 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 12:10:53.146570 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0809 12:10:53.146848 (+ 278us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 12:10:53.147152 (+ 304us) server_negotiation.cc:1035] Waiting for connection context
0809 12:10:53.147489 (+ 337us) server_negotiation.cc:299] Negotiation successful
0809 12:10:53.147725 (+ 236us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":345,"thread_start_us":661,"threads_started":1}
I20260809 12:10:53.151983 9084 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:53650:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260809 12:10:53.152611 9084 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260809 12:10:53.161116 9234 tablet_service.cc:1467] Processing CreateTablet for tablet e90bd93abb9a4c8ab235669ce8a8ee10 (DEFAULT_TABLE table=test-table [id=edfc6c2230404ea8baaa632b7ad65605]), partition=RANGE (key) PARTITION UNBOUNDED
I20260809 12:10:53.161664 9234 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e90bd93abb9a4c8ab235669ce8a8ee10. 1 dirs total, 0 dirs full, 0 dirs failed
I20260809 12:10:53.161597 9386 tablet_service.cc:1467] Processing CreateTablet for tablet e90bd93abb9a4c8ab235669ce8a8ee10 (DEFAULT_TABLE table=test-table [id=edfc6c2230404ea8baaa632b7ad65605]), partition=RANGE (key) PARTITION UNBOUNDED
I20260809 12:10:53.162077 9386 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e90bd93abb9a4c8ab235669ce8a8ee10. 1 dirs total, 0 dirs full, 0 dirs failed
I20260809 12:10:53.166290 9537 tablet_service.cc:1467] Processing CreateTablet for tablet e90bd93abb9a4c8ab235669ce8a8ee10 (DEFAULT_TABLE table=test-table [id=edfc6c2230404ea8baaa632b7ad65605]), partition=RANGE (key) PARTITION UNBOUNDED
I20260809 12:10:53.166769 9537 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e90bd93abb9a4c8ab235669ce8a8ee10. 1 dirs total, 0 dirs full, 0 dirs failed
I20260809 12:10:53.177593 9741 tablet_bootstrap.cc:492] T e90bd93abb9a4c8ab235669ce8a8ee10 P af75f3a57f6f48fba26736ad36aff045: Bootstrap starting.
I20260809 12:10:53.180708 9740 tablet_bootstrap.cc:492] T e90bd93abb9a4c8ab235669ce8a8ee10 P cd3a4ffa0f4544a1b74b74fe5b6e3711: Bootstrap starting.
I20260809 12:10:53.182662 9740 tablet_bootstrap.cc:654] T e90bd93abb9a4c8ab235669ce8a8ee10 P cd3a4ffa0f4544a1b74b74fe5b6e3711: Neither blocks nor log segments found. Creating new log.
I20260809 12:10:53.184526 9741 tablet_bootstrap.cc:654] T e90bd93abb9a4c8ab235669ce8a8ee10 P af75f3a57f6f48fba26736ad36aff045: Neither blocks nor log segments found. Creating new log.
I20260809 12:10:53.188326 9742 tablet_bootstrap.cc:492] T e90bd93abb9a4c8ab235669ce8a8ee10 P 040356c8db9f4f8e93ae96f8fcaecf29: Bootstrap starting.
I20260809 12:10:53.190397 9742 tablet_bootstrap.cc:654] T e90bd93abb9a4c8ab235669ce8a8ee10 P 040356c8db9f4f8e93ae96f8fcaecf29: Neither blocks nor log segments found. Creating new log.
I20260809 12:10:53.194170 9741 tablet_bootstrap.cc:492] T e90bd93abb9a4c8ab235669ce8a8ee10 P af75f3a57f6f48fba26736ad36aff045: No bootstrap required, opened a new log
I20260809 12:10:53.194466 9741 ts_tablet_manager.cc:1397] T e90bd93abb9a4c8ab235669ce8a8ee10 P af75f3a57f6f48fba26736ad36aff045: Time spent bootstrapping tablet: real 0.017s user 0.003s sys 0.008s
I20260809 12:10:53.195736 9741 raft_consensus.cc:348] T e90bd93abb9a4c8ab235669ce8a8ee10 P af75f3a57f6f48fba26736ad36aff045 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "af75f3a57f6f48fba26736ad36aff045" member_type: VOTER last_known_addr { host: "127.0.222.1" port: 41951 } } peers { permanent_uuid: "cd3a4ffa0f4544a1b74b74fe5b6e3711" member_type: VOTER last_known_addr { host: "127.0.222.2" port: 43017 } } peers { permanent_uuid: "040356c8db9f4f8e93ae96f8fcaecf29" member_type: VOTER last_known_addr { host: "127.0.222.3" port: 33697 } }
I20260809 12:10:53.196285 9741 raft_consensus.cc:374] T e90bd93abb9a4c8ab235669ce8a8ee10 P af75f3a57f6f48fba26736ad36aff045 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260809 12:10:53.196478 9741 raft_consensus.cc:729] T e90bd93abb9a4c8ab235669ce8a8ee10 P af75f3a57f6f48fba26736ad36aff045 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: af75f3a57f6f48fba26736ad36aff045, State: Initialized, Role: FOLLOWER
I20260809 12:10:53.196919 9741 consensus_queue.cc:260] T e90bd93abb9a4c8ab235669ce8a8ee10 P af75f3a57f6f48fba26736ad36aff045 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "af75f3a57f6f48fba26736ad36aff045" member_type: VOTER last_known_addr { host: "127.0.222.1" port: 41951 } } peers { permanent_uuid: "cd3a4ffa0f4544a1b74b74fe5b6e3711" member_type: VOTER last_known_addr { host: "127.0.222.2" port: 43017 } } peers { permanent_uuid: "040356c8db9f4f8e93ae96f8fcaecf29" member_type: VOTER last_known_addr { host: "127.0.222.3" port: 33697 } }
I20260809 12:10:53.198782 9741 ts_tablet_manager.cc:1428] T e90bd93abb9a4c8ab235669ce8a8ee10 P af75f3a57f6f48fba26736ad36aff045: Time spent starting tablet: real 0.004s user 0.001s sys 0.001s
I20260809 12:10:53.200488 9742 tablet_bootstrap.cc:492] T e90bd93abb9a4c8ab235669ce8a8ee10 P 040356c8db9f4f8e93ae96f8fcaecf29: No bootstrap required, opened a new log
I20260809 12:10:53.200659 9742 ts_tablet_manager.cc:1397] T e90bd93abb9a4c8ab235669ce8a8ee10 P 040356c8db9f4f8e93ae96f8fcaecf29: Time spent bootstrapping tablet: real 0.012s user 0.004s sys 0.000s
I20260809 12:10:53.200803 9740 tablet_bootstrap.cc:492] T e90bd93abb9a4c8ab235669ce8a8ee10 P cd3a4ffa0f4544a1b74b74fe5b6e3711: No bootstrap required, opened a new log
I20260809 12:10:53.200961 9740 ts_tablet_manager.cc:1397] T e90bd93abb9a4c8ab235669ce8a8ee10 P cd3a4ffa0f4544a1b74b74fe5b6e3711: Time spent bootstrapping tablet: real 0.020s user 0.009s sys 0.005s
I20260809 12:10:53.201505 9742 raft_consensus.cc:348] T e90bd93abb9a4c8ab235669ce8a8ee10 P 040356c8db9f4f8e93ae96f8fcaecf29 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "af75f3a57f6f48fba26736ad36aff045" member_type: VOTER last_known_addr { host: "127.0.222.1" port: 41951 } } peers { permanent_uuid: "cd3a4ffa0f4544a1b74b74fe5b6e3711" member_type: VOTER last_known_addr { host: "127.0.222.2" port: 43017 } } peers { permanent_uuid: "040356c8db9f4f8e93ae96f8fcaecf29" member_type: VOTER last_known_addr { host: "127.0.222.3" port: 33697 } }
I20260809 12:10:53.201821 9742 raft_consensus.cc:374] T e90bd93abb9a4c8ab235669ce8a8ee10 P 040356c8db9f4f8e93ae96f8fcaecf29 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260809 12:10:53.201795 9740 raft_consensus.cc:348] T e90bd93abb9a4c8ab235669ce8a8ee10 P cd3a4ffa0f4544a1b74b74fe5b6e3711 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "af75f3a57f6f48fba26736ad36aff045" member_type: VOTER last_known_addr { host: "127.0.222.1" port: 41951 } } peers { permanent_uuid: "cd3a4ffa0f4544a1b74b74fe5b6e3711" member_type: VOTER last_known_addr { host: "127.0.222.2" port: 43017 } } peers { permanent_uuid: "040356c8db9f4f8e93ae96f8fcaecf29" member_type: VOTER last_known_addr { host: "127.0.222.3" port: 33697 } }
I20260809 12:10:53.201951 9742 raft_consensus.cc:729] T e90bd93abb9a4c8ab235669ce8a8ee10 P 040356c8db9f4f8e93ae96f8fcaecf29 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 040356c8db9f4f8e93ae96f8fcaecf29, State: Initialized, Role: FOLLOWER
I20260809 12:10:53.202036 9740 raft_consensus.cc:374] T e90bd93abb9a4c8ab235669ce8a8ee10 P cd3a4ffa0f4544a1b74b74fe5b6e3711 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260809 12:10:53.202132 9740 raft_consensus.cc:729] T e90bd93abb9a4c8ab235669ce8a8ee10 P cd3a4ffa0f4544a1b74b74fe5b6e3711 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: cd3a4ffa0f4544a1b74b74fe5b6e3711, State: Initialized, Role: FOLLOWER
I20260809 12:10:53.202261 9742 consensus_queue.cc:260] T e90bd93abb9a4c8ab235669ce8a8ee10 P 040356c8db9f4f8e93ae96f8fcaecf29 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "af75f3a57f6f48fba26736ad36aff045" member_type: VOTER last_known_addr { host: "127.0.222.1" port: 41951 } } peers { permanent_uuid: "cd3a4ffa0f4544a1b74b74fe5b6e3711" member_type: VOTER last_known_addr { host: "127.0.222.2" port: 43017 } } peers { permanent_uuid: "040356c8db9f4f8e93ae96f8fcaecf29" member_type: VOTER last_known_addr { host: "127.0.222.3" port: 33697 } }
I20260809 12:10:53.202399 9740 consensus_queue.cc:260] T e90bd93abb9a4c8ab235669ce8a8ee10 P cd3a4ffa0f4544a1b74b74fe5b6e3711 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "af75f3a57f6f48fba26736ad36aff045" member_type: VOTER last_known_addr { host: "127.0.222.1" port: 41951 } } peers { permanent_uuid: "cd3a4ffa0f4544a1b74b74fe5b6e3711" member_type: VOTER last_known_addr { host: "127.0.222.2" port: 43017 } } peers { permanent_uuid: "040356c8db9f4f8e93ae96f8fcaecf29" member_type: VOTER last_known_addr { host: "127.0.222.3" port: 33697 } }
I20260809 12:10:53.202996 9742 ts_tablet_manager.cc:1428] T e90bd93abb9a4c8ab235669ce8a8ee10 P 040356c8db9f4f8e93ae96f8fcaecf29: Time spent starting tablet: real 0.002s user 0.003s sys 0.000s
I20260809 12:10:53.204372 9740 ts_tablet_manager.cc:1428] T e90bd93abb9a4c8ab235669ce8a8ee10 P cd3a4ffa0f4544a1b74b74fe5b6e3711: Time spent starting tablet: real 0.003s user 0.002s sys 0.001s
I20260809 12:10:53.313812 9645 raft_consensus.cc:482] T e90bd93abb9a4c8ab235669ce8a8ee10 P af75f3a57f6f48fba26736ad36aff045 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260809 12:10:53.314071 9645 raft_consensus.cc:504] T e90bd93abb9a4c8ab235669ce8a8ee10 P af75f3a57f6f48fba26736ad36aff045 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "af75f3a57f6f48fba26736ad36aff045" member_type: VOTER last_known_addr { host: "127.0.222.1" port: 41951 } } peers { permanent_uuid: "cd3a4ffa0f4544a1b74b74fe5b6e3711" member_type: VOTER last_known_addr { host: "127.0.222.2" port: 43017 } } peers { permanent_uuid: "040356c8db9f4f8e93ae96f8fcaecf29" member_type: VOTER last_known_addr { host: "127.0.222.3" port: 33697 } }
I20260809 12:10:53.315173 9645 leader_election.cc:288] T e90bd93abb9a4c8ab235669ce8a8ee10 P af75f3a57f6f48fba26736ad36aff045 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers cd3a4ffa0f4544a1b74b74fe5b6e3711 (127.0.222.2:43017), 040356c8db9f4f8e93ae96f8fcaecf29 (127.0.222.3:33697)
I20260809 12:10:53.315874 9558 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e90bd93abb9a4c8ab235669ce8a8ee10" candidate_uuid: "af75f3a57f6f48fba26736ad36aff045" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "040356c8db9f4f8e93ae96f8fcaecf29" is_pre_election: true
I20260809 12:10:53.315954 9409 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e90bd93abb9a4c8ab235669ce8a8ee10" candidate_uuid: "af75f3a57f6f48fba26736ad36aff045" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "cd3a4ffa0f4544a1b74b74fe5b6e3711" is_pre_election: true
I20260809 12:10:53.316186 9558 raft_consensus.cc:2454] T e90bd93abb9a4c8ab235669ce8a8ee10 P 040356c8db9f4f8e93ae96f8fcaecf29 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate af75f3a57f6f48fba26736ad36aff045 in term 0.
I20260809 12:10:53.316267 9409 raft_consensus.cc:2454] T e90bd93abb9a4c8ab235669ce8a8ee10 P cd3a4ffa0f4544a1b74b74fe5b6e3711 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate af75f3a57f6f48fba26736ad36aff045 in term 0.
I20260809 12:10:53.316948 9180 leader_election.cc:302] T e90bd93abb9a4c8ab235669ce8a8ee10 P af75f3a57f6f48fba26736ad36aff045 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 3 yes votes; 0 no votes. yes voters: 040356c8db9f4f8e93ae96f8fcaecf29, af75f3a57f6f48fba26736ad36aff045, cd3a4ffa0f4544a1b74b74fe5b6e3711; no voters:
I20260809 12:10:53.317615 9645 raft_consensus.cc:2781] T e90bd93abb9a4c8ab235669ce8a8ee10 P af75f3a57f6f48fba26736ad36aff045 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260809 12:10:53.317817 9645 raft_consensus.cc:482] T e90bd93abb9a4c8ab235669ce8a8ee10 P af75f3a57f6f48fba26736ad36aff045 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260809 12:10:53.317932 9645 raft_consensus.cc:3037] T e90bd93abb9a4c8ab235669ce8a8ee10 P af75f3a57f6f48fba26736ad36aff045 [term 0 FOLLOWER]: Advancing to term 1
I20260809 12:10:53.320219 9645 raft_consensus.cc:504] T e90bd93abb9a4c8ab235669ce8a8ee10 P af75f3a57f6f48fba26736ad36aff045 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "af75f3a57f6f48fba26736ad36aff045" member_type: VOTER last_known_addr { host: "127.0.222.1" port: 41951 } } peers { permanent_uuid: "cd3a4ffa0f4544a1b74b74fe5b6e3711" member_type: VOTER last_known_addr { host: "127.0.222.2" port: 43017 } } peers { permanent_uuid: "040356c8db9f4f8e93ae96f8fcaecf29" member_type: VOTER last_known_addr { host: "127.0.222.3" port: 33697 } }
I20260809 12:10:53.321125 9645 leader_election.cc:288] T e90bd93abb9a4c8ab235669ce8a8ee10 P af75f3a57f6f48fba26736ad36aff045 [CANDIDATE]: Term 1 election: Requested vote from peers cd3a4ffa0f4544a1b74b74fe5b6e3711 (127.0.222.2:43017), 040356c8db9f4f8e93ae96f8fcaecf29 (127.0.222.3:33697)
I20260809 12:10:53.321417 9409 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e90bd93abb9a4c8ab235669ce8a8ee10" candidate_uuid: "af75f3a57f6f48fba26736ad36aff045" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "cd3a4ffa0f4544a1b74b74fe5b6e3711"
I20260809 12:10:53.321647 9409 raft_consensus.cc:3037] T e90bd93abb9a4c8ab235669ce8a8ee10 P cd3a4ffa0f4544a1b74b74fe5b6e3711 [term 0 FOLLOWER]: Advancing to term 1
I20260809 12:10:53.321902 9558 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e90bd93abb9a4c8ab235669ce8a8ee10" candidate_uuid: "af75f3a57f6f48fba26736ad36aff045" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "040356c8db9f4f8e93ae96f8fcaecf29"
I20260809 12:10:53.322144 9558 raft_consensus.cc:3037] T e90bd93abb9a4c8ab235669ce8a8ee10 P 040356c8db9f4f8e93ae96f8fcaecf29 [term 0 FOLLOWER]: Advancing to term 1
I20260809 12:10:53.325076 9409 raft_consensus.cc:2454] T e90bd93abb9a4c8ab235669ce8a8ee10 P cd3a4ffa0f4544a1b74b74fe5b6e3711 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate af75f3a57f6f48fba26736ad36aff045 in term 1.
I20260809 12:10:53.325574 9180 leader_election.cc:302] T e90bd93abb9a4c8ab235669ce8a8ee10 P af75f3a57f6f48fba26736ad36aff045 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: af75f3a57f6f48fba26736ad36aff045, cd3a4ffa0f4544a1b74b74fe5b6e3711; no voters:
I20260809 12:10:53.326179 9558 raft_consensus.cc:2454] T e90bd93abb9a4c8ab235669ce8a8ee10 P 040356c8db9f4f8e93ae96f8fcaecf29 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate af75f3a57f6f48fba26736ad36aff045 in term 1.
I20260809 12:10:53.326330 9645 raft_consensus.cc:2781] T e90bd93abb9a4c8ab235669ce8a8ee10 P af75f3a57f6f48fba26736ad36aff045 [term 1 FOLLOWER]: Leader election won for term 1
I20260809 12:10:53.327165 9645 raft_consensus.cc:686] T e90bd93abb9a4c8ab235669ce8a8ee10 P af75f3a57f6f48fba26736ad36aff045 [term 1 LEADER]: Becoming Leader. State: Replica: af75f3a57f6f48fba26736ad36aff045, State: Running, Role: LEADER
I20260809 12:10:53.327575 9645 consensus_queue.cc:237] T e90bd93abb9a4c8ab235669ce8a8ee10 P af75f3a57f6f48fba26736ad36aff045 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "af75f3a57f6f48fba26736ad36aff045" member_type: VOTER last_known_addr { host: "127.0.222.1" port: 41951 } } peers { permanent_uuid: "cd3a4ffa0f4544a1b74b74fe5b6e3711" member_type: VOTER last_known_addr { host: "127.0.222.2" port: 43017 } } peers { permanent_uuid: "040356c8db9f4f8e93ae96f8fcaecf29" member_type: VOTER last_known_addr { host: "127.0.222.3" port: 33697 } }
I20260809 12:10:53.330868 9084 catalog_manager.cc:5495] T e90bd93abb9a4c8ab235669ce8a8ee10 P af75f3a57f6f48fba26736ad36aff045 reported cstate change: term changed from 0 to 1, leader changed from <none> to af75f3a57f6f48fba26736ad36aff045 (127.0.222.1). New cstate: current_term: 1 leader_uuid: "af75f3a57f6f48fba26736ad36aff045" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "af75f3a57f6f48fba26736ad36aff045" member_type: VOTER last_known_addr { host: "127.0.222.1" port: 41951 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "cd3a4ffa0f4544a1b74b74fe5b6e3711" member_type: VOTER last_known_addr { host: "127.0.222.2" port: 43017 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "040356c8db9f4f8e93ae96f8fcaecf29" member_type: VOTER last_known_addr { host: "127.0.222.3" port: 33697 } health_report { overall_health: UNKNOWN } } }
I20260809 12:10:53.376972 9694 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 12:10:53.371116 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49646 (local address 127.0.222.1:41951)
0809 12:10:53.371349 (+ 233us) server_negotiation.cc:206] Beginning negotiation
0809 12:10:53.371356 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0809 12:10:53.371380 (+ 24us) server_negotiation.cc:407] Connection header received
0809 12:10:53.371518 (+ 138us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 12:10:53.371524 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 12:10:53.371638 (+ 114us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 12:10:53.371778 (+ 140us) server_negotiation.cc:226] Negotiated authn=TOKEN
0809 12:10:53.372191 (+ 413us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 12:10:53.373277 (+ 1086us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 12:10:53.374854 (+ 1577us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 12:10:53.375430 (+ 576us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 12:10:53.375592 (+ 162us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 12:10:53.376077 (+ 485us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0809 12:10:53.376190 (+ 113us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0809 12:10:53.376301 (+ 111us) server_negotiation.cc:1035] Waiting for connection context
0809 12:10:53.376477 (+ 176us) server_negotiation.cc:299] Negotiation successful
0809 12:10:53.376563 (+ 86us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":74}
I20260809 12:10:53.380672 9558 raft_consensus.cc:1261] T e90bd93abb9a4c8ab235669ce8a8ee10 P 040356c8db9f4f8e93ae96f8fcaecf29 [term 1 FOLLOWER]: Refusing update from remote peer af75f3a57f6f48fba26736ad36aff045: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260809 12:10:53.380662 9409 raft_consensus.cc:1261] T e90bd93abb9a4c8ab235669ce8a8ee10 P cd3a4ffa0f4544a1b74b74fe5b6e3711 [term 1 FOLLOWER]: Refusing update from remote peer af75f3a57f6f48fba26736ad36aff045: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260809 12:10:53.381639 9702 consensus_queue.cc:1035] T e90bd93abb9a4c8ab235669ce8a8ee10 P af75f3a57f6f48fba26736ad36aff045 [LEADER]: Connected to new peer: Peer: permanent_uuid: "040356c8db9f4f8e93ae96f8fcaecf29" member_type: VOTER last_known_addr { host: "127.0.222.3" port: 33697 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260809 12:10:53.381980 9645 consensus_queue.cc:1035] T e90bd93abb9a4c8ab235669ce8a8ee10 P af75f3a57f6f48fba26736ad36aff045 [LEADER]: Connected to new peer: Peer: permanent_uuid: "cd3a4ffa0f4544a1b74b74fe5b6e3711" member_type: VOTER last_known_addr { host: "127.0.222.2" port: 43017 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260809 12:10:53.424764 9084 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:53650:
table { table_name: "test-table" } modify_external_catalogs: true
I20260809 12:10:53.425285 9084 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:53650:
table { table_name: "test-table" } modify_external_catalogs: true
I20260809 12:10:53.432461 9084 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 7873beb5ce214b9d8d238a51db1accb1: Sending DeleteTablet for 3 replicas of tablet e90bd93abb9a4c8ab235669ce8a8ee10
I20260809 12:10:53.433331 9234 tablet_service.cc:1514] Processing DeleteTablet for tablet e90bd93abb9a4c8ab235669ce8a8ee10 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-09 12:10:53 UTC) from {username='oryx', principal='oryx/127.0.222.62@KRBTEST.COM'} at 127.0.0.1:49628
I20260809 12:10:53.433923 9721 tablet_replica.cc:330] T e90bd93abb9a4c8ab235669ce8a8ee10 P af75f3a57f6f48fba26736ad36aff045: stopping tablet replica
I20260809 12:10:53.434003 9537 tablet_service.cc:1514] Processing DeleteTablet for tablet e90bd93abb9a4c8ab235669ce8a8ee10 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-09 12:10:53 UTC) from {username='oryx', principal='oryx/127.0.222.62@KRBTEST.COM'} at 127.0.0.1:53446
I20260809 12:10:53.434180 9721 raft_consensus.cc:2229] T e90bd93abb9a4c8ab235669ce8a8ee10 P af75f3a57f6f48fba26736ad36aff045 [term 1 LEADER]: Raft consensus shutting down.
I20260809 12:10:53.434792 9721 raft_consensus.cc:2258] T e90bd93abb9a4c8ab235669ce8a8ee10 P af75f3a57f6f48fba26736ad36aff045 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260809 12:10:53.435525 9723 tablet_replica.cc:330] T e90bd93abb9a4c8ab235669ce8a8ee10 P 040356c8db9f4f8e93ae96f8fcaecf29: stopping tablet replica
I20260809 12:10:53.435724 9723 raft_consensus.cc:2229] T e90bd93abb9a4c8ab235669ce8a8ee10 P 040356c8db9f4f8e93ae96f8fcaecf29 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260809 12:10:53.435853 9723 pending_rounds.cc:62] T e90bd93abb9a4c8ab235669ce8a8ee10 P 040356c8db9f4f8e93ae96f8fcaecf29: Trying to abort 2 pending ops.
I20260809 12:10:53.436054 9723 pending_rounds.cc:68] T e90bd93abb9a4c8ab235669ce8a8ee10 P 040356c8db9f4f8e93ae96f8fcaecf29: Aborting op as it isn't in flight: id { term: 1 index: 2 } timestamp: 7316592449039966208 op_type: WRITE_OP write_request { tablet_id: "e90bd93abb9a4c8ab235669ce8a8ee10" schema { columns { name: "key" type: INT32 is_key: true is_nullable: false immutable: false } columns { name: "val" type: INT32 is_key: false is_nullable: true immutable: false } } row_operations { rows: "<redacted>""\001\003\00090\000\0001\324\000\000" indirect_data: "<redacted>""" } external_consistency_mode: CLIENT_PROPAGATED authz_token { token_data: "\010\371\336\341\323\006\"8\n\ntest-admin\022*\n edfc6c2230404ea8baaa632b7ad65605\020\001\030\001 \001(\001" signature: "<redacted>""\022}F\275`\201*\227G\242\311Z+\255\343\n(\021(iz\026F*\377\370qE\003\236\265H\360\265\241\373\355\312\034\332\340o[8S\316\361\201[\013Yx\361\330\352c\275N\032Mr\337\311\370" signing_key_seq_num: 0 } } request_id { client_id: "92d5648b2b9f43b7a163dba730a5cce6" seq_no: 0 first_incomplete_seq_no: 0 attempt_no: 0 }
I20260809 12:10:53.436795 9723 pending_rounds.cc:68] T e90bd93abb9a4c8ab235669ce8a8ee10 P 040356c8db9f4f8e93ae96f8fcaecf29: Aborting op as it isn't in flight: id { term: 1 index: 1 } timestamp: 7316592448832626688 op_type: NO_OP noop_request { }
I20260809 12:10:53.436947 9723 raft_consensus.cc:2866] T e90bd93abb9a4c8ab235669ce8a8ee10 P 040356c8db9f4f8e93ae96f8fcaecf29 [term 1 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted
I20260809 12:10:53.437073 9723 raft_consensus.cc:2258] T e90bd93abb9a4c8ab235669ce8a8ee10 P 040356c8db9f4f8e93ae96f8fcaecf29 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260809 12:10:53.437844 9386 tablet_service.cc:1514] Processing DeleteTablet for tablet e90bd93abb9a4c8ab235669ce8a8ee10 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-09 12:10:53 UTC) from {username='oryx', principal='oryx/127.0.222.62@KRBTEST.COM'} at 127.0.0.1:40518
I20260809 12:10:53.438172 9723 ts_tablet_manager.cc:1905] T e90bd93abb9a4c8ab235669ce8a8ee10 P 040356c8db9f4f8e93ae96f8fcaecf29: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260809 12:10:53.438678 9722 tablet_replica.cc:330] T e90bd93abb9a4c8ab235669ce8a8ee10 P cd3a4ffa0f4544a1b74b74fe5b6e3711: stopping tablet replica
I20260809 12:10:53.438905 9722 raft_consensus.cc:2229] T e90bd93abb9a4c8ab235669ce8a8ee10 P cd3a4ffa0f4544a1b74b74fe5b6e3711 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260809 12:10:53.439196 9722 raft_consensus.cc:2258] T e90bd93abb9a4c8ab235669ce8a8ee10 P cd3a4ffa0f4544a1b74b74fe5b6e3711 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260809 12:10:53.441713 9085 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:53650:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260809 12:10:53.442531 9085 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260809 12:10:53.445655 9723 ts_tablet_manager.cc:1918] T e90bd93abb9a4c8ab235669ce8a8ee10 P 040356c8db9f4f8e93ae96f8fcaecf29: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260809 12:10:53.445819 9723 log.cc:1196] T e90bd93abb9a4c8ab235669ce8a8ee10 P 040356c8db9f4f8e93ae96f8fcaecf29: Deleting WAL directory at /tmp/dist-test-taskgH5Pjm/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786277382073938-888-0/minicluster-data/ts-2/wal/wals/e90bd93abb9a4c8ab235669ce8a8ee10
I20260809 12:10:53.446303 9723 ts_tablet_manager.cc:1939] T e90bd93abb9a4c8ab235669ce8a8ee10 P 040356c8db9f4f8e93ae96f8fcaecf29: Deleting consensus metadata
I20260809 12:10:53.448171 9722 ts_tablet_manager.cc:1905] T e90bd93abb9a4c8ab235669ce8a8ee10 P cd3a4ffa0f4544a1b74b74fe5b6e3711: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260809 12:10:53.453539 9537 tablet_service.cc:1467] Processing CreateTablet for tablet 5014bce7403a4d5f81f8700e1da90ddb (DEFAULT_TABLE table=test-table [id=0653f9f546f742b78cf0f7588bfb70c6]), partition=RANGE (key) PARTITION UNBOUNDED
I20260809 12:10:53.454013 9537 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 5014bce7403a4d5f81f8700e1da90ddb. 1 dirs total, 0 dirs full, 0 dirs failed
I20260809 12:10:53.453927 9386 tablet_service.cc:1467] Processing CreateTablet for tablet 5014bce7403a4d5f81f8700e1da90ddb (DEFAULT_TABLE table=test-table [id=0653f9f546f742b78cf0f7588bfb70c6]), partition=RANGE (key) PARTITION UNBOUNDED
I20260809 12:10:53.454370 9386 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 5014bce7403a4d5f81f8700e1da90ddb. 1 dirs total, 0 dirs full, 0 dirs failed
I20260809 12:10:53.456785 9234 tablet_service.cc:1467] Processing CreateTablet for tablet 5014bce7403a4d5f81f8700e1da90ddb (DEFAULT_TABLE table=test-table [id=0653f9f546f742b78cf0f7588bfb70c6]), partition=RANGE (key) PARTITION UNBOUNDED
I20260809 12:10:53.457234 9234 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 5014bce7403a4d5f81f8700e1da90ddb. 1 dirs total, 0 dirs full, 0 dirs failed
I20260809 12:10:53.461755 9721 ts_tablet_manager.cc:1905] T e90bd93abb9a4c8ab235669ce8a8ee10 P af75f3a57f6f48fba26736ad36aff045: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260809 12:10:53.471073 9742 tablet_bootstrap.cc:492] T 5014bce7403a4d5f81f8700e1da90ddb P 040356c8db9f4f8e93ae96f8fcaecf29: Bootstrap starting.
I20260809 12:10:53.473013 9742 tablet_bootstrap.cc:654] T 5014bce7403a4d5f81f8700e1da90ddb P 040356c8db9f4f8e93ae96f8fcaecf29: Neither blocks nor log segments found. Creating new log.
I20260809 12:10:53.473230 9740 tablet_bootstrap.cc:492] T 5014bce7403a4d5f81f8700e1da90ddb P cd3a4ffa0f4544a1b74b74fe5b6e3711: Bootstrap starting.
I20260809 12:10:53.474095 9722 ts_tablet_manager.cc:1918] T e90bd93abb9a4c8ab235669ce8a8ee10 P cd3a4ffa0f4544a1b74b74fe5b6e3711: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260809 12:10:53.474258 9722 log.cc:1196] T e90bd93abb9a4c8ab235669ce8a8ee10 P cd3a4ffa0f4544a1b74b74fe5b6e3711: Deleting WAL directory at /tmp/dist-test-taskgH5Pjm/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786277382073938-888-0/minicluster-data/ts-1/wal/wals/e90bd93abb9a4c8ab235669ce8a8ee10
I20260809 12:10:53.474784 9722 ts_tablet_manager.cc:1939] T e90bd93abb9a4c8ab235669ce8a8ee10 P cd3a4ffa0f4544a1b74b74fe5b6e3711: Deleting consensus metadata
I20260809 12:10:53.475299 9740 tablet_bootstrap.cc:654] T 5014bce7403a4d5f81f8700e1da90ddb P cd3a4ffa0f4544a1b74b74fe5b6e3711: Neither blocks nor log segments found. Creating new log.
I20260809 12:10:53.476176 9068 catalog_manager.cc:4841] TS cd3a4ffa0f4544a1b74b74fe5b6e3711 (127.0.222.2:43017): tablet e90bd93abb9a4c8ab235669ce8a8ee10 (table test-table [id=edfc6c2230404ea8baaa632b7ad65605]) successfully deleted
I20260809 12:10:53.476639 9741 tablet_bootstrap.cc:492] T 5014bce7403a4d5f81f8700e1da90ddb P af75f3a57f6f48fba26736ad36aff045: Bootstrap starting.
I20260809 12:10:53.478631 9741 tablet_bootstrap.cc:654] T 5014bce7403a4d5f81f8700e1da90ddb P af75f3a57f6f48fba26736ad36aff045: Neither blocks nor log segments found. Creating new log.
I20260809 12:10:53.483565 9071 catalog_manager.cc:4841] TS 040356c8db9f4f8e93ae96f8fcaecf29 (127.0.222.3:33697): tablet e90bd93abb9a4c8ab235669ce8a8ee10 (table test-table [id=edfc6c2230404ea8baaa632b7ad65605]) successfully deleted
I20260809 12:10:53.484778 9721 ts_tablet_manager.cc:1918] T e90bd93abb9a4c8ab235669ce8a8ee10 P af75f3a57f6f48fba26736ad36aff045: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260809 12:10:53.486431 9537 tablet_service.cc:1514] Processing DeleteTablet for tablet e90bd93abb9a4c8ab235669ce8a8ee10 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-09 12:10:53 UTC) from {username='oryx', principal='oryx/127.0.222.62@KRBTEST.COM'} at 127.0.0.1:53446
W20260809 12:10:53.487416 9071 catalog_manager.cc:4808] TS 040356c8db9f4f8e93ae96f8fcaecf29 (127.0.222.3:33697): delete failed for tablet e90bd93abb9a4c8ab235669ce8a8ee10 because the tablet was not found. No further retry: Not found: Tablet not found: e90bd93abb9a4c8ab235669ce8a8ee10
I20260809 12:10:53.488538 9740 tablet_bootstrap.cc:492] T 5014bce7403a4d5f81f8700e1da90ddb P cd3a4ffa0f4544a1b74b74fe5b6e3711: No bootstrap required, opened a new log
I20260809 12:10:53.488773 9740 ts_tablet_manager.cc:1397] T 5014bce7403a4d5f81f8700e1da90ddb P cd3a4ffa0f4544a1b74b74fe5b6e3711: Time spent bootstrapping tablet: real 0.016s user 0.002s sys 0.003s
I20260809 12:10:53.488826 9721 log.cc:1196] T e90bd93abb9a4c8ab235669ce8a8ee10 P af75f3a57f6f48fba26736ad36aff045: Deleting WAL directory at /tmp/dist-test-taskgH5Pjm/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786277382073938-888-0/minicluster-data/ts-0/wal/wals/e90bd93abb9a4c8ab235669ce8a8ee10
I20260809 12:10:53.489657 9721 ts_tablet_manager.cc:1939] T e90bd93abb9a4c8ab235669ce8a8ee10 P af75f3a57f6f48fba26736ad36aff045: Deleting consensus metadata
I20260809 12:10:53.489727 9740 raft_consensus.cc:348] T 5014bce7403a4d5f81f8700e1da90ddb P cd3a4ffa0f4544a1b74b74fe5b6e3711 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "af75f3a57f6f48fba26736ad36aff045" member_type: VOTER last_known_addr { host: "127.0.222.1" port: 41951 } } peers { permanent_uuid: "cd3a4ffa0f4544a1b74b74fe5b6e3711" member_type: VOTER last_known_addr { host: "127.0.222.2" port: 43017 } } peers { permanent_uuid: "040356c8db9f4f8e93ae96f8fcaecf29" member_type: VOTER last_known_addr { host: "127.0.222.3" port: 33697 } }
I20260809 12:10:53.490281 9740 raft_consensus.cc:374] T 5014bce7403a4d5f81f8700e1da90ddb P cd3a4ffa0f4544a1b74b74fe5b6e3711 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260809 12:10:53.490398 9740 raft_consensus.cc:729] T 5014bce7403a4d5f81f8700e1da90ddb P cd3a4ffa0f4544a1b74b74fe5b6e3711 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: cd3a4ffa0f4544a1b74b74fe5b6e3711, State: Initialized, Role: FOLLOWER
I20260809 12:10:53.491061 9741 tablet_bootstrap.cc:492] T 5014bce7403a4d5f81f8700e1da90ddb P af75f3a57f6f48fba26736ad36aff045: No bootstrap required, opened a new log
I20260809 12:10:53.491035 9740 consensus_queue.cc:260] T 5014bce7403a4d5f81f8700e1da90ddb P cd3a4ffa0f4544a1b74b74fe5b6e3711 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "af75f3a57f6f48fba26736ad36aff045" member_type: VOTER last_known_addr { host: "127.0.222.1" port: 41951 } } peers { permanent_uuid: "cd3a4ffa0f4544a1b74b74fe5b6e3711" member_type: VOTER last_known_addr { host: "127.0.222.2" port: 43017 } } peers { permanent_uuid: "040356c8db9f4f8e93ae96f8fcaecf29" member_type: VOTER last_known_addr { host: "127.0.222.3" port: 33697 } }
I20260809 12:10:53.491228 9741 ts_tablet_manager.cc:1397] T 5014bce7403a4d5f81f8700e1da90ddb P af75f3a57f6f48fba26736ad36aff045: Time spent bootstrapping tablet: real 0.015s user 0.010s sys 0.000s
I20260809 12:10:53.491848 9740 ts_tablet_manager.cc:1428] T 5014bce7403a4d5f81f8700e1da90ddb P cd3a4ffa0f4544a1b74b74fe5b6e3711: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20260809 12:10:53.491981 9741 raft_consensus.cc:348] T 5014bce7403a4d5f81f8700e1da90ddb P af75f3a57f6f48fba26736ad36aff045 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "af75f3a57f6f48fba26736ad36aff045" member_type: VOTER last_known_addr { host: "127.0.222.1" port: 41951 } } peers { permanent_uuid: "cd3a4ffa0f4544a1b74b74fe5b6e3711" member_type: VOTER last_known_addr { host: "127.0.222.2" port: 43017 } } peers { permanent_uuid: "040356c8db9f4f8e93ae96f8fcaecf29" member_type: VOTER last_known_addr { host: "127.0.222.3" port: 33697 } }
I20260809 12:10:53.492228 9741 raft_consensus.cc:374] T 5014bce7403a4d5f81f8700e1da90ddb P af75f3a57f6f48fba26736ad36aff045 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260809 12:10:53.492313 9741 raft_consensus.cc:729] T 5014bce7403a4d5f81f8700e1da90ddb P af75f3a57f6f48fba26736ad36aff045 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: af75f3a57f6f48fba26736ad36aff045, State: Initialized, Role: FOLLOWER
I20260809 12:10:53.492588 9741 consensus_queue.cc:260] T 5014bce7403a4d5f81f8700e1da90ddb P af75f3a57f6f48fba26736ad36aff045 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "af75f3a57f6f48fba26736ad36aff045" member_type: VOTER last_known_addr { host: "127.0.222.1" port: 41951 } } peers { permanent_uuid: "cd3a4ffa0f4544a1b74b74fe5b6e3711" member_type: VOTER last_known_addr { host: "127.0.222.2" port: 43017 } } peers { permanent_uuid: "040356c8db9f4f8e93ae96f8fcaecf29" member_type: VOTER last_known_addr { host: "127.0.222.3" port: 33697 } }
I20260809 12:10:53.493299 9741 ts_tablet_manager.cc:1428] T 5014bce7403a4d5f81f8700e1da90ddb P af75f3a57f6f48fba26736ad36aff045: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20260809 12:10:53.498817 9069 catalog_manager.cc:4841] TS af75f3a57f6f48fba26736ad36aff045 (127.0.222.1:41951): tablet e90bd93abb9a4c8ab235669ce8a8ee10 (table test-table [id=edfc6c2230404ea8baaa632b7ad65605]) successfully deleted
I20260809 12:10:53.509789 9742 tablet_bootstrap.cc:492] T 5014bce7403a4d5f81f8700e1da90ddb P 040356c8db9f4f8e93ae96f8fcaecf29: No bootstrap required, opened a new log
I20260809 12:10:53.510041 9742 ts_tablet_manager.cc:1397] T 5014bce7403a4d5f81f8700e1da90ddb P 040356c8db9f4f8e93ae96f8fcaecf29: Time spent bootstrapping tablet: real 0.039s user 0.000s sys 0.021s
I20260809 12:10:53.510771 9742 raft_consensus.cc:348] T 5014bce7403a4d5f81f8700e1da90ddb P 040356c8db9f4f8e93ae96f8fcaecf29 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "af75f3a57f6f48fba26736ad36aff045" member_type: VOTER last_known_addr { host: "127.0.222.1" port: 41951 } } peers { permanent_uuid: "cd3a4ffa0f4544a1b74b74fe5b6e3711" member_type: VOTER last_known_addr { host: "127.0.222.2" port: 43017 } } peers { permanent_uuid: "040356c8db9f4f8e93ae96f8fcaecf29" member_type: VOTER last_known_addr { host: "127.0.222.3" port: 33697 } }
I20260809 12:10:53.511093 9742 raft_consensus.cc:374] T 5014bce7403a4d5f81f8700e1da90ddb P 040356c8db9f4f8e93ae96f8fcaecf29 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260809 12:10:53.511183 9742 raft_consensus.cc:729] T 5014bce7403a4d5f81f8700e1da90ddb P 040356c8db9f4f8e93ae96f8fcaecf29 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 040356c8db9f4f8e93ae96f8fcaecf29, State: Initialized, Role: FOLLOWER
I20260809 12:10:53.511438 9742 consensus_queue.cc:260] T 5014bce7403a4d5f81f8700e1da90ddb P 040356c8db9f4f8e93ae96f8fcaecf29 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "af75f3a57f6f48fba26736ad36aff045" member_type: VOTER last_known_addr { host: "127.0.222.1" port: 41951 } } peers { permanent_uuid: "cd3a4ffa0f4544a1b74b74fe5b6e3711" member_type: VOTER last_known_addr { host: "127.0.222.2" port: 43017 } } peers { permanent_uuid: "040356c8db9f4f8e93ae96f8fcaecf29" member_type: VOTER last_known_addr { host: "127.0.222.3" port: 33697 } }
I20260809 12:10:53.512580 9742 ts_tablet_manager.cc:1428] T 5014bce7403a4d5f81f8700e1da90ddb P 040356c8db9f4f8e93ae96f8fcaecf29: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20260809 12:10:53.549715 9645 raft_consensus.cc:482] T 5014bce7403a4d5f81f8700e1da90ddb P af75f3a57f6f48fba26736ad36aff045 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260809 12:10:53.549955 9645 raft_consensus.cc:504] T 5014bce7403a4d5f81f8700e1da90ddb P af75f3a57f6f48fba26736ad36aff045 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "af75f3a57f6f48fba26736ad36aff045" member_type: VOTER last_known_addr { host: "127.0.222.1" port: 41951 } } peers { permanent_uuid: "cd3a4ffa0f4544a1b74b74fe5b6e3711" member_type: VOTER last_known_addr { host: "127.0.222.2" port: 43017 } } peers { permanent_uuid: "040356c8db9f4f8e93ae96f8fcaecf29" member_type: VOTER last_known_addr { host: "127.0.222.3" port: 33697 } }
I20260809 12:10:53.550747 9645 leader_election.cc:288] T 5014bce7403a4d5f81f8700e1da90ddb P af75f3a57f6f48fba26736ad36aff045 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers cd3a4ffa0f4544a1b74b74fe5b6e3711 (127.0.222.2:43017), 040356c8db9f4f8e93ae96f8fcaecf29 (127.0.222.3:33697)
I20260809 12:10:53.551783 9409 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5014bce7403a4d5f81f8700e1da90ddb" candidate_uuid: "af75f3a57f6f48fba26736ad36aff045" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "cd3a4ffa0f4544a1b74b74fe5b6e3711" is_pre_election: true
I20260809 12:10:53.552052 9409 raft_consensus.cc:2454] T 5014bce7403a4d5f81f8700e1da90ddb P cd3a4ffa0f4544a1b74b74fe5b6e3711 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate af75f3a57f6f48fba26736ad36aff045 in term 0.
I20260809 12:10:53.552640 9558 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5014bce7403a4d5f81f8700e1da90ddb" candidate_uuid: "af75f3a57f6f48fba26736ad36aff045" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "040356c8db9f4f8e93ae96f8fcaecf29" is_pre_election: true
I20260809 12:10:53.552929 9558 raft_consensus.cc:2454] T 5014bce7403a4d5f81f8700e1da90ddb P 040356c8db9f4f8e93ae96f8fcaecf29 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate af75f3a57f6f48fba26736ad36aff045 in term 0.
I20260809 12:10:53.553566 9183 leader_election.cc:302] T 5014bce7403a4d5f81f8700e1da90ddb P af75f3a57f6f48fba26736ad36aff045 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 040356c8db9f4f8e93ae96f8fcaecf29, af75f3a57f6f48fba26736ad36aff045; no voters:
I20260809 12:10:53.554353 9645 raft_consensus.cc:2781] T 5014bce7403a4d5f81f8700e1da90ddb P af75f3a57f6f48fba26736ad36aff045 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260809 12:10:53.554478 9645 raft_consensus.cc:482] T 5014bce7403a4d5f81f8700e1da90ddb P af75f3a57f6f48fba26736ad36aff045 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260809 12:10:53.554561 9645 raft_consensus.cc:3037] T 5014bce7403a4d5f81f8700e1da90ddb P af75f3a57f6f48fba26736ad36aff045 [term 0 FOLLOWER]: Advancing to term 1
I20260809 12:10:53.558883 9645 raft_consensus.cc:504] T 5014bce7403a4d5f81f8700e1da90ddb P af75f3a57f6f48fba26736ad36aff045 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "af75f3a57f6f48fba26736ad36aff045" member_type: VOTER last_known_addr { host: "127.0.222.1" port: 41951 } } peers { permanent_uuid: "cd3a4ffa0f4544a1b74b74fe5b6e3711" member_type: VOTER last_known_addr { host: "127.0.222.2" port: 43017 } } peers { permanent_uuid: "040356c8db9f4f8e93ae96f8fcaecf29" member_type: VOTER last_known_addr { host: "127.0.222.3" port: 33697 } }
I20260809 12:10:53.560606 9409 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5014bce7403a4d5f81f8700e1da90ddb" candidate_uuid: "af75f3a57f6f48fba26736ad36aff045" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "cd3a4ffa0f4544a1b74b74fe5b6e3711"
I20260809 12:10:53.560868 9409 raft_consensus.cc:3037] T 5014bce7403a4d5f81f8700e1da90ddb P cd3a4ffa0f4544a1b74b74fe5b6e3711 [term 0 FOLLOWER]: Advancing to term 1
I20260809 12:10:53.561146 9558 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5014bce7403a4d5f81f8700e1da90ddb" candidate_uuid: "af75f3a57f6f48fba26736ad36aff045" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "040356c8db9f4f8e93ae96f8fcaecf29"
I20260809 12:10:53.561406 9558 raft_consensus.cc:3037] T 5014bce7403a4d5f81f8700e1da90ddb P 040356c8db9f4f8e93ae96f8fcaecf29 [term 0 FOLLOWER]: Advancing to term 1
I20260809 12:10:53.564397 9558 raft_consensus.cc:2454] T 5014bce7403a4d5f81f8700e1da90ddb P 040356c8db9f4f8e93ae96f8fcaecf29 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate af75f3a57f6f48fba26736ad36aff045 in term 1.
I20260809 12:10:53.564428 9409 raft_consensus.cc:2454] T 5014bce7403a4d5f81f8700e1da90ddb P cd3a4ffa0f4544a1b74b74fe5b6e3711 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate af75f3a57f6f48fba26736ad36aff045 in term 1.
I20260809 12:10:53.564807 9183 leader_election.cc:302] T 5014bce7403a4d5f81f8700e1da90ddb P af75f3a57f6f48fba26736ad36aff045 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 040356c8db9f4f8e93ae96f8fcaecf29, af75f3a57f6f48fba26736ad36aff045; no voters:
I20260809 12:10:53.565128 9702 raft_consensus.cc:2781] T 5014bce7403a4d5f81f8700e1da90ddb P af75f3a57f6f48fba26736ad36aff045 [term 1 FOLLOWER]: Leader election won for term 1
I20260809 12:10:53.565287 9645 leader_election.cc:288] T 5014bce7403a4d5f81f8700e1da90ddb P af75f3a57f6f48fba26736ad36aff045 [CANDIDATE]: Term 1 election: Requested vote from peers cd3a4ffa0f4544a1b74b74fe5b6e3711 (127.0.222.2:43017), 040356c8db9f4f8e93ae96f8fcaecf29 (127.0.222.3:33697)
I20260809 12:10:53.566021 9702 raft_consensus.cc:686] T 5014bce7403a4d5f81f8700e1da90ddb P af75f3a57f6f48fba26736ad36aff045 [term 1 LEADER]: Becoming Leader. State: Replica: af75f3a57f6f48fba26736ad36aff045, State: Running, Role: LEADER
I20260809 12:10:53.566337 9702 consensus_queue.cc:237] T 5014bce7403a4d5f81f8700e1da90ddb P af75f3a57f6f48fba26736ad36aff045 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "af75f3a57f6f48fba26736ad36aff045" member_type: VOTER last_known_addr { host: "127.0.222.1" port: 41951 } } peers { permanent_uuid: "cd3a4ffa0f4544a1b74b74fe5b6e3711" member_type: VOTER last_known_addr { host: "127.0.222.2" port: 43017 } } peers { permanent_uuid: "040356c8db9f4f8e93ae96f8fcaecf29" member_type: VOTER last_known_addr { host: "127.0.222.3" port: 33697 } }
I20260809 12:10:53.569308 9083 catalog_manager.cc:5495] T 5014bce7403a4d5f81f8700e1da90ddb P af75f3a57f6f48fba26736ad36aff045 reported cstate change: term changed from 0 to 1, leader changed from <none> to af75f3a57f6f48fba26736ad36aff045 (127.0.222.1). New cstate: current_term: 1 leader_uuid: "af75f3a57f6f48fba26736ad36aff045" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "af75f3a57f6f48fba26736ad36aff045" member_type: VOTER last_known_addr { host: "127.0.222.1" port: 41951 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "cd3a4ffa0f4544a1b74b74fe5b6e3711" member_type: VOTER last_known_addr { host: "127.0.222.2" port: 43017 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "040356c8db9f4f8e93ae96f8fcaecf29" member_type: VOTER last_known_addr { host: "127.0.222.3" port: 33697 } health_report { overall_health: UNKNOWN } } }
I20260809 12:10:53.608224 9694 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 12:10:53.602429 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.222.1:44133 (local address 127.0.222.1:41951)
0809 12:10:53.602669 (+ 240us) server_negotiation.cc:206] Beginning negotiation
0809 12:10:53.602677 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0809 12:10:53.602832 (+ 155us) server_negotiation.cc:407] Connection header received
0809 12:10:53.603118 (+ 286us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 12:10:53.603126 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 12:10:53.603195 (+ 69us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 12:10:53.603299 (+ 104us) server_negotiation.cc:226] Negotiated authn=TOKEN
0809 12:10:53.603795 (+ 496us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 12:10:53.604579 (+ 784us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 12:10:53.606145 (+ 1566us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 12:10:53.606673 (+ 528us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 12:10:53.606851 (+ 178us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 12:10:53.607365 (+ 514us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0809 12:10:53.607489 (+ 124us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0809 12:10:53.607611 (+ 122us) server_negotiation.cc:1035] Waiting for connection context
0809 12:10:53.607779 (+ 168us) server_negotiation.cc:299] Negotiation successful
0809 12:10:53.607869 (+ 90us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":76}
I20260809 12:10:53.608908 9761 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 12:10:53.602029 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.222.1:41951 (local address 127.0.222.1:44133)
0809 12:10:53.602739 (+ 710us) negotiation.cc:107] Waiting for socket to connect
0809 12:10:53.602760 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0809 12:10:53.602945 (+ 185us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 12:10:53.603318 (+ 373us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 12:10:53.603322 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 12:10:53.603333 (+ 11us) client_negotiation.cc:189] Negotiated authn=TOKEN
0809 12:10:53.603633 (+ 300us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 12:10:53.603643 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 12:10:53.604733 (+ 1090us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 12:10:53.604742 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 12:10:53.605973 (+ 1231us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 12:10:53.605988 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 12:10:53.606907 (+ 919us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 12:10:53.606916 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 12:10:53.607132 (+ 216us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 12:10:53.607174 (+ 42us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0809 12:10:53.607610 (+ 436us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0809 12:10:53.607622 (+ 12us) client_negotiation.cc:769] Sending connection context
0809 12:10:53.607748 (+ 126us) client_negotiation.cc:240] Negotiation successful
0809 12:10:53.607890 (+ 142us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":548,"thread_start_us":674,"threads_started":1}
I20260809 12:10:53.620106 9409 raft_consensus.cc:1261] T 5014bce7403a4d5f81f8700e1da90ddb P cd3a4ffa0f4544a1b74b74fe5b6e3711 [term 1 FOLLOWER]: Refusing update from remote peer af75f3a57f6f48fba26736ad36aff045: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260809 12:10:53.620546 9557 raft_consensus.cc:1261] T 5014bce7403a4d5f81f8700e1da90ddb P 040356c8db9f4f8e93ae96f8fcaecf29 [term 1 FOLLOWER]: Refusing update from remote peer af75f3a57f6f48fba26736ad36aff045: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260809 12:10:53.620784 9702 consensus_queue.cc:1035] T 5014bce7403a4d5f81f8700e1da90ddb P af75f3a57f6f48fba26736ad36aff045 [LEADER]: Connected to new peer: Peer: permanent_uuid: "cd3a4ffa0f4544a1b74b74fe5b6e3711" member_type: VOTER last_known_addr { host: "127.0.222.2" port: 43017 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260809 12:10:53.621860 9702 consensus_queue.cc:1035] T 5014bce7403a4d5f81f8700e1da90ddb P af75f3a57f6f48fba26736ad36aff045 [LEADER]: Connected to new peer: Peer: permanent_uuid: "040356c8db9f4f8e93ae96f8fcaecf29" member_type: VOTER last_known_addr { host: "127.0.222.3" port: 33697 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260809 12:10:53.709647 9084 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:53650:
table { table_name: "test-table" } modify_external_catalogs: true
I20260809 12:10:53.709923 9084 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:53650:
table { table_name: "test-table" } modify_external_catalogs: true
I20260809 12:10:53.716454 9084 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 7873beb5ce214b9d8d238a51db1accb1: Sending DeleteTablet for 3 replicas of tablet 5014bce7403a4d5f81f8700e1da90ddb
I20260809 12:10:53.717787 9386 tablet_service.cc:1514] Processing DeleteTablet for tablet 5014bce7403a4d5f81f8700e1da90ddb with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-09 12:10:53 UTC) from {username='oryx', principal='oryx/127.0.222.62@KRBTEST.COM'} at 127.0.0.1:40518
I20260809 12:10:53.718156 9722 tablet_replica.cc:330] T 5014bce7403a4d5f81f8700e1da90ddb P cd3a4ffa0f4544a1b74b74fe5b6e3711: stopping tablet replica
I20260809 12:10:53.718087 9537 tablet_service.cc:1514] Processing DeleteTablet for tablet 5014bce7403a4d5f81f8700e1da90ddb with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-09 12:10:53 UTC) from {username='oryx', principal='oryx/127.0.222.62@KRBTEST.COM'} at 127.0.0.1:53446
I20260809 12:10:53.718348 9722 raft_consensus.cc:2229] T 5014bce7403a4d5f81f8700e1da90ddb P cd3a4ffa0f4544a1b74b74fe5b6e3711 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260809 12:10:53.718583 9722 raft_consensus.cc:2258] T 5014bce7403a4d5f81f8700e1da90ddb P cd3a4ffa0f4544a1b74b74fe5b6e3711 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260809 12:10:53.718909 9723 tablet_replica.cc:330] T 5014bce7403a4d5f81f8700e1da90ddb P 040356c8db9f4f8e93ae96f8fcaecf29: stopping tablet replica
I20260809 12:10:53.719129 9723 raft_consensus.cc:2229] T 5014bce7403a4d5f81f8700e1da90ddb P 040356c8db9f4f8e93ae96f8fcaecf29 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260809 12:10:53.719347 9723 raft_consensus.cc:2258] T 5014bce7403a4d5f81f8700e1da90ddb P 040356c8db9f4f8e93ae96f8fcaecf29 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260809 12:10:53.719686 888 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskgH5Pjm/build/asan/bin/kudu pid 9164
I20260809 12:10:53.720275 9234 tablet_service.cc:1514] Processing DeleteTablet for tablet 5014bce7403a4d5f81f8700e1da90ddb with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-09 12:10:53 UTC) from {username='oryx', principal='oryx/127.0.222.62@KRBTEST.COM'} at 127.0.0.1:49628
I20260809 12:10:53.720589 9721 tablet_replica.cc:330] T 5014bce7403a4d5f81f8700e1da90ddb P af75f3a57f6f48fba26736ad36aff045: stopping tablet replica
I20260809 12:10:53.720793 9721 raft_consensus.cc:2229] T 5014bce7403a4d5f81f8700e1da90ddb P af75f3a57f6f48fba26736ad36aff045 [term 1 LEADER]: Raft consensus shutting down.
I20260809 12:10:53.721231 9721 raft_consensus.cc:2258] T 5014bce7403a4d5f81f8700e1da90ddb P af75f3a57f6f48fba26736ad36aff045 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260809 12:10:53.722496 9721 ts_tablet_manager.cc:1905] T 5014bce7403a4d5f81f8700e1da90ddb P af75f3a57f6f48fba26736ad36aff045: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260809 12:10:53.734076 9721 ts_tablet_manager.cc:1918] T 5014bce7403a4d5f81f8700e1da90ddb P af75f3a57f6f48fba26736ad36aff045: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260809 12:10:53.734234 9721 log.cc:1196] T 5014bce7403a4d5f81f8700e1da90ddb P af75f3a57f6f48fba26736ad36aff045: Deleting WAL directory at /tmp/dist-test-taskgH5Pjm/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786277382073938-888-0/minicluster-data/ts-0/wal/wals/5014bce7403a4d5f81f8700e1da90ddb
I20260809 12:10:53.734679 9721 ts_tablet_manager.cc:1939] T 5014bce7403a4d5f81f8700e1da90ddb P af75f3a57f6f48fba26736ad36aff045: Deleting consensus metadata
I20260809 12:10:53.737085 9722 ts_tablet_manager.cc:1905] T 5014bce7403a4d5f81f8700e1da90ddb P cd3a4ffa0f4544a1b74b74fe5b6e3711: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260809 12:10:53.744341 9722 ts_tablet_manager.cc:1918] T 5014bce7403a4d5f81f8700e1da90ddb P cd3a4ffa0f4544a1b74b74fe5b6e3711: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260809 12:10:53.744495 9722 log.cc:1196] T 5014bce7403a4d5f81f8700e1da90ddb P cd3a4ffa0f4544a1b74b74fe5b6e3711: Deleting WAL directory at /tmp/dist-test-taskgH5Pjm/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786277382073938-888-0/minicluster-data/ts-1/wal/wals/5014bce7403a4d5f81f8700e1da90ddb
I20260809 12:10:53.744887 9722 ts_tablet_manager.cc:1939] T 5014bce7403a4d5f81f8700e1da90ddb P cd3a4ffa0f4544a1b74b74fe5b6e3711: Deleting consensus metadata
I20260809 12:10:53.746063 9068 catalog_manager.cc:4841] TS cd3a4ffa0f4544a1b74b74fe5b6e3711 (127.0.222.2:43017): tablet 5014bce7403a4d5f81f8700e1da90ddb (table test-table [id=0653f9f546f742b78cf0f7588bfb70c6]) successfully deleted
I20260809 12:10:53.747613 9723 ts_tablet_manager.cc:1905] T 5014bce7403a4d5f81f8700e1da90ddb P 040356c8db9f4f8e93ae96f8fcaecf29: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260809 12:10:53.754606 9723 ts_tablet_manager.cc:1918] T 5014bce7403a4d5f81f8700e1da90ddb P 040356c8db9f4f8e93ae96f8fcaecf29: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260809 12:10:53.754786 9723 log.cc:1196] T 5014bce7403a4d5f81f8700e1da90ddb P 040356c8db9f4f8e93ae96f8fcaecf29: Deleting WAL directory at /tmp/dist-test-taskgH5Pjm/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786277382073938-888-0/minicluster-data/ts-2/wal/wals/5014bce7403a4d5f81f8700e1da90ddb
I20260809 12:10:53.755343 9723 ts_tablet_manager.cc:1939] T 5014bce7403a4d5f81f8700e1da90ddb P 040356c8db9f4f8e93ae96f8fcaecf29: Deleting consensus metadata
I20260809 12:10:53.756681 9071 catalog_manager.cc:4841] TS 040356c8db9f4f8e93ae96f8fcaecf29 (127.0.222.3:33697): tablet 5014bce7403a4d5f81f8700e1da90ddb (table test-table [id=0653f9f546f742b78cf0f7588bfb70c6]) successfully deleted
I20260809 12:10:53.758168 9069 catalog_manager.cc:4841] TS af75f3a57f6f48fba26736ad36aff045 (127.0.222.1:41951): tablet 5014bce7403a4d5f81f8700e1da90ddb (table test-table [id=0653f9f546f742b78cf0f7588bfb70c6]) successfully deleted
Aug 09 12:10:53 dist-test-slave-2w3w krb5kdc[9036](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786277453, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.0.222.1@KRBTEST.COM
I20260809 12:10:53.775128 9694 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 12:10:53.728167 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49660 (local address 127.0.222.1:41951)
0809 12:10:53.728324 (+ 157us) server_negotiation.cc:206] Beginning negotiation
0809 12:10:53.728331 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0809 12:10:53.759309 (+ 30978us) server_negotiation.cc:407] Connection header received
0809 12:10:53.759555 (+ 246us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 12:10:53.759561 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 12:10:53.759658 (+ 97us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 12:10:53.759758 (+ 100us) server_negotiation.cc:226] Negotiated authn=SASL
0809 12:10:53.760970 (+ 1212us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 12:10:53.761955 (+ 985us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 12:10:53.763821 (+ 1866us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 12:10:53.764351 (+ 530us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 12:10:53.764518 (+ 167us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 12:10:53.768714 (+ 4196us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 12:10:53.768728 (+ 14us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 12:10:53.768743 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0809 12:10:53.768780 (+ 37us) server_negotiation.cc:922] calling sasl_server_start()
0809 12:10:53.772023 (+ 3243us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 12:10:53.772713 (+ 690us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 12:10:53.772720 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 12:10:53.772723 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0809 12:10:53.772874 (+ 151us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 12:10:53.773275 (+ 401us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 12:10:53.773279 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 12:10:53.773281 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0809 12:10:53.773784 (+ 503us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 12:10:53.774132 (+ 348us) server_negotiation.cc:1035] Waiting for connection context
0809 12:10:53.774424 (+ 292us) server_negotiation.cc:299] Negotiation successful
0809 12:10:53.774655 (+ 231us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":50}
I20260809 12:10:53.775801 9306 generic_service.cc:196] Checking for leaks (request via RPC)
I20260809 12:10:53.925628 888 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskgH5Pjm/build/asan/bin/kudu with pid 9164
I20260809 12:10:53.954577 888 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskgH5Pjm/build/asan/bin/kudu pid 9317
Aug 09 12:10:53 dist-test-slave-2w3w krb5kdc[9036](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786277453, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.0.222.2@KRBTEST.COM
I20260809 12:10:53.971443 9767 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 12:10:53.955415 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40542 (local address 127.0.222.2:43017)
0809 12:10:53.955919 (+ 504us) server_negotiation.cc:206] Beginning negotiation
0809 12:10:53.955924 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0809 12:10:53.955940 (+ 16us) server_negotiation.cc:407] Connection header received
0809 12:10:53.955997 (+ 57us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 12:10:53.956001 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 12:10:53.956061 (+ 60us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 12:10:53.956165 (+ 104us) server_negotiation.cc:226] Negotiated authn=SASL
0809 12:10:53.957584 (+ 1419us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 12:10:53.958428 (+ 844us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 12:10:53.959775 (+ 1347us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 12:10:53.960210 (+ 435us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 12:10:53.960322 (+ 112us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 12:10:53.964813 (+ 4491us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 12:10:53.964828 (+ 15us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 12:10:53.964845 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0809 12:10:53.964881 (+ 36us) server_negotiation.cc:922] calling sasl_server_start()
0809 12:10:53.968128 (+ 3247us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 12:10:53.968851 (+ 723us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 12:10:53.968861 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 12:10:53.968866 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0809 12:10:53.969088 (+ 222us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 12:10:53.969455 (+ 367us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 12:10:53.969461 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 12:10:53.969465 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0809 12:10:53.969953 (+ 488us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 12:10:53.970286 (+ 333us) server_negotiation.cc:1035] Waiting for connection context
0809 12:10:53.970463 (+ 177us) server_negotiation.cc:299] Negotiation successful
0809 12:10:53.970702 (+ 239us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":407,"thread_start_us":283,"threads_started":1}
I20260809 12:10:53.972203 9458 generic_service.cc:196] Checking for leaks (request via RPC)
I20260809 12:10:54.109273 888 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskgH5Pjm/build/asan/bin/kudu with pid 9317
I20260809 12:10:54.135108 888 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskgH5Pjm/build/asan/bin/kudu pid 9468
Aug 09 12:10:54 dist-test-slave-2w3w krb5kdc[9036](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786277453, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.0.222.3@KRBTEST.COM
I20260809 12:10:54.151049 9769 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 12:10:54.135947 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53454 (local address 127.0.222.3:33697)
0809 12:10:54.136451 (+ 504us) server_negotiation.cc:206] Beginning negotiation
0809 12:10:54.136455 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0809 12:10:54.136472 (+ 17us) server_negotiation.cc:407] Connection header received
0809 12:10:54.136529 (+ 57us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 12:10:54.136533 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 12:10:54.136599 (+ 66us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 12:10:54.136686 (+ 87us) server_negotiation.cc:226] Negotiated authn=SASL
0809 12:10:54.138100 (+ 1414us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 12:10:54.138823 (+ 723us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 12:10:54.140268 (+ 1445us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 12:10:54.140675 (+ 407us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 12:10:54.140770 (+ 95us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 12:10:54.145405 (+ 4635us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 12:10:54.145421 (+ 16us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 12:10:54.145438 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0809 12:10:54.145485 (+ 47us) server_negotiation.cc:922] calling sasl_server_start()
0809 12:10:54.148450 (+ 2965us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 12:10:54.149129 (+ 679us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 12:10:54.149136 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 12:10:54.149139 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0809 12:10:54.149277 (+ 138us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 12:10:54.149644 (+ 367us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 12:10:54.149647 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 12:10:54.149649 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0809 12:10:54.149998 (+ 349us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 12:10:54.150285 (+ 287us) server_negotiation.cc:1035] Waiting for connection context
0809 12:10:54.150522 (+ 237us) server_negotiation.cc:299] Negotiation successful
0809 12:10:54.150723 (+ 201us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":394,"thread_start_us":284,"threads_started":1}
I20260809 12:10:54.151706 9606 generic_service.cc:196] Checking for leaks (request via RPC)
I20260809 12:10:54.276302 888 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskgH5Pjm/build/asan/bin/kudu with pid 9468
I20260809 12:10:54.302346 888 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskgH5Pjm/build/asan/bin/kudu pid 9050
I20260809 12:10:54.303670 9128 generic_service.cc:196] Checking for leaks (request via RPC)
I20260809 12:10:54.424469 888 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskgH5Pjm/build/asan/bin/kudu with pid 9050
2026-08-09T12:10:54Z chronyd exiting
[ OK ] SecurityITest.TestNonDefaultPrincipal (6705 ms)
[ RUN ] SecurityITest.TestNonExistentPrincipal
Loading random data
Initializing database '/tmp/dist-test-taskgH5Pjm/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1786277382073938-888-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 09 12:10:54 dist-test-slave-2w3w krb5kdc[9774](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 09 12:10:54 dist-test-slave-2w3w krb5kdc[9774](info): set up 2 sockets
Aug 09 12:10:54 dist-test-slave-2w3w krb5kdc[9774](info): commencing operation
krb5kdc: starting...
W20260809 12:10:55.592967 888 mini_kdc.cc:121] Time spent starting KDC: real 1.119s user 0.002s sys 0.033s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskgH5Pjm/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1786277382073938-888-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskgH5Pjm/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1786277382073938-888-0/krb5kdc/test-user.keytab.
Aug 09 12:10:55 dist-test-slave-2w3w krb5kdc[9774](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786277455, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-09T12:10:55Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-09T12:10:55Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.222.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.222.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.222.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskgH5Pjm/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1786277382073938-888-0/krb5kdc/kudu_127.0.222.62.keytab.
Entry for principal kudu/127.0.222.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskgH5Pjm/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1786277382073938-888-0/krb5kdc/kudu_127.0.222.62.keytab.
I20260809 12:10:55.881996 888 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskgH5Pjm/build/asan/bin/kudu
/tmp/dist-test-taskgH5Pjm/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskgH5Pjm/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1786277382073938-888-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskgH5Pjm/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1786277382073938-888-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskgH5Pjm/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1786277382073938-888-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskgH5Pjm/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1786277382073938-888-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.222.62:33863
--webserver_interface=127.0.222.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskgH5Pjm/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1786277382073938-888-0/krb5kdc/kudu_127.0.222.62.keytab
--principal=kudu/127.0.222.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.222.20:36241
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.222.62:33863
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--principal=oryx with env {KRB5CCNAME=/tmp/dist-test-taskgH5Pjm/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1786277382073938-888-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskgH5Pjm/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1786277382073938-888-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskgH5Pjm/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1786277382073938-888-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260809 12:10:56.218802 9789 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260809 12:10:56.219182 9789 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260809 12:10:56.219352 9789 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260809 12:10:56.226536 9789 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260809 12:10:56.226693 9789 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260809 12:10:56.226799 9789 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260809 12:10:56.226905 9789 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260809 12:10:56.227027 9789 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260809 12:10:56.237068 9789 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.222.20:36241
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskgH5Pjm/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1786277382073938-888-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskgH5Pjm/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1786277382073938-888-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.222.62:33863
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskgH5Pjm/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1786277382073938-888-0/krb5kdc/kudu_127.0.222.62.keytab
--principal=oryx
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.222.62:33863
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskgH5Pjm/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1786277382073938-888-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.222.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskgH5Pjm/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1786277382073938-888-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 09 Aug 2026 11:06:41 UTC on ccycloud.quasar-bwoogy.root.comops.site
build id 4575991
ASAN enabled
I20260809 12:10:56.238445 9789 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260809 12:10:56.239815 9789 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260809 12:10:56.251431 9797 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 12:10:56.251632 9800 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 12:10:56.251652 9798 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260809 12:10:56.253245 9789 server_base.cc:1034] running on GCE node
I20260809 12:10:56.254009 9789 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260809 12:10:56.255473 9789 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260809 12:10:56.256704 9789 hybrid_clock.cc:648] HybridClock initialized: now 1786277456256660 us; error 36 us; skew 500 ppm
I20260809 12:10:56.257071 9789 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Runtime error: kinit failed: unable to login from keytab: Keytab contains no suitable keys for oryx@KRBTEST.COM
2026-08-09T12:10:56Z chronyd exiting
[ OK ] SecurityITest.TestNonExistentPrincipal (1915 ms)
[ RUN ] SecurityITest.TestMismatchingPrincipals
Loading random data
Initializing database '/tmp/dist-test-taskgH5Pjm/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786277382073938-888-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 09 12:10:56 dist-test-slave-2w3w krb5kdc[9807](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 09 12:10:56 dist-test-slave-2w3w krb5kdc[9807](info): set up 2 sockets
Aug 09 12:10:56 dist-test-slave-2w3w krb5kdc[9807](info): commencing operation
krb5kdc: starting...
W20260809 12:10:57.500286 888 mini_kdc.cc:121] Time spent starting KDC: real 1.111s user 0.002s sys 0.035s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskgH5Pjm/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786277382073938-888-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskgH5Pjm/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786277382073938-888-0/krb5kdc/test-user.keytab.
Aug 09 12:10:57 dist-test-slave-2w3w krb5kdc[9807](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786277457, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-09T12:10:57Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-09T12:10:57Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.222.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.222.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.222.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskgH5Pjm/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786277382073938-888-0/krb5kdc/kudu_127.0.222.62.keytab.
Entry for principal kudu/127.0.222.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskgH5Pjm/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786277382073938-888-0/krb5kdc/kudu_127.0.222.62.keytab.
I20260809 12:10:57.775635 888 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskgH5Pjm/build/asan/bin/kudu
/tmp/dist-test-taskgH5Pjm/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskgH5Pjm/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786277382073938-888-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskgH5Pjm/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786277382073938-888-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskgH5Pjm/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786277382073938-888-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskgH5Pjm/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786277382073938-888-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.222.62:44607
--webserver_interface=127.0.222.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskgH5Pjm/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786277382073938-888-0/krb5kdc/kudu_127.0.222.62.keytab
--principal=kudu/127.0.222.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.222.20:43745
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.222.62:44607
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskgH5Pjm/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786277382073938-888-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskgH5Pjm/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786277382073938-888-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskgH5Pjm/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786277382073938-888-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260809 12:10:58.115191 9820 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260809 12:10:58.115522 9820 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260809 12:10:58.115680 9820 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260809 12:10:58.122812 9820 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260809 12:10:58.122907 9820 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260809 12:10:58.123050 9820 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260809 12:10:58.123152 9820 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260809 12:10:58.123234 9820 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260809 12:10:58.132843 9820 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.222.20:43745
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskgH5Pjm/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786277382073938-888-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskgH5Pjm/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786277382073938-888-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.222.62:44607
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskgH5Pjm/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786277382073938-888-0/krb5kdc/kudu_127.0.222.62.keytab
--principal=kudu/127.0.222.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.222.62:44607
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskgH5Pjm/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786277382073938-888-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.222.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskgH5Pjm/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786277382073938-888-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 09 Aug 2026 11:06:41 UTC on ccycloud.quasar-bwoogy.root.comops.site
build id 4575991
ASAN enabled
I20260809 12:10:58.134178 9820 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260809 12:10:58.135514 9820 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260809 12:10:58.147938 9826 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 12:10:58.147998 9827 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 12:10:58.148315 9830 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260809 12:10:58.149222 9820 server_base.cc:1034] running on GCE node
I20260809 12:10:58.150068 9820 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260809 12:10:58.151460 9820 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260809 12:10:58.152665 9820 hybrid_clock.cc:648] HybridClock initialized: now 1786277458152599 us; error 53 us; skew 500 ppm
I20260809 12:10:58.153050 9820 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 09 12:10:58 dist-test-slave-2w3w krb5kdc[9807](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786277458, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.222.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260809 12:10:58.156250 9820 init.cc:377] Logged in from keytab as kudu/127.0.222.62@KRBTEST.COM (short username kudu)
I20260809 12:10:58.157691 9820 webserver.cc:460] Webserver started at http://127.0.222.62:42893/ using document root <none> and password file <none>
I20260809 12:10:58.158371 9820 fs_manager.cc:359] Metadata directory not provided
I20260809 12:10:58.158533 9820 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260809 12:10:58.158869 9820 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260809 12:10:58.161870 9820 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskgH5Pjm/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786277382073938-888-0/minicluster-data/master-0/data/instance:
uuid: "c52389485e85465f849e792576b5e75c"
format_stamp: "Formatted at 2026-08-09 12:10:58 on dist-test-slave-2w3w"
server_key: "ad9476eb168bb14d98dfe596b0b7c4d9"
server_key_iv: "0455fd4b3eb144bc266e76b120c224c2"
server_key_version: "encryptionkey@0"
I20260809 12:10:58.162675 9820 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskgH5Pjm/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786277382073938-888-0/minicluster-data/master-0/wal/instance:
uuid: "c52389485e85465f849e792576b5e75c"
format_stamp: "Formatted at 2026-08-09 12:10:58 on dist-test-slave-2w3w"
server_key: "ad9476eb168bb14d98dfe596b0b7c4d9"
server_key_iv: "0455fd4b3eb144bc266e76b120c224c2"
server_key_version: "encryptionkey@0"
I20260809 12:10:58.167851 9820 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.000s sys 0.003s
I20260809 12:10:58.171499 9838 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260809 12:10:58.172868 9820 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.002s
I20260809 12:10:58.173074 9820 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskgH5Pjm/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786277382073938-888-0/minicluster-data/master-0/data,/tmp/dist-test-taskgH5Pjm/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786277382073938-888-0/minicluster-data/master-0/wal
uuid: "c52389485e85465f849e792576b5e75c"
format_stamp: "Formatted at 2026-08-09 12:10:58 on dist-test-slave-2w3w"
server_key: "ad9476eb168bb14d98dfe596b0b7c4d9"
server_key_iv: "0455fd4b3eb144bc266e76b120c224c2"
server_key_version: "encryptionkey@0"
I20260809 12:10:58.173295 9820 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskgH5Pjm/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786277382073938-888-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskgH5Pjm/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786277382073938-888-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskgH5Pjm/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786277382073938-888-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260809 12:10:58.196208 9820 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260809 12:10:58.200191 9820 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260809 12:10:58.200496 9820 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260809 12:10:58.219918 9820 rpc_server.cc:307] RPC server started. Bound to: 127.0.222.62:44607
I20260809 12:10:58.219966 9895 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.222.62:44607 every 8 connection(s)
I20260809 12:10:58.221342 9820 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskgH5Pjm/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786277382073938-888-0/minicluster-data/master-0/data/info.pb
I20260809 12:10:58.225107 9896 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260809 12:10:58.231575 888 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskgH5Pjm/build/asan/bin/kudu as pid 9820
I20260809 12:10:58.231817 888 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskgH5Pjm/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786277382073938-888-0/minicluster-data/master-0/wal/instance
I20260809 12:10:58.232710 888 external_mini_cluster.cc:1428] Setting key 87be5cc13ca19b67b2f5cfbc9a9deef3
I20260809 12:10:58.236665 9896 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P c52389485e85465f849e792576b5e75c: Bootstrap starting.
I20260809 12:10:58.240406 9896 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P c52389485e85465f849e792576b5e75c: Neither blocks nor log segments found. Creating new log.
I20260809 12:10:58.241472 9896 log.cc:824] T 00000000000000000000000000000000 P c52389485e85465f849e792576b5e75c: Log is configured to *not* fsync() on all Append() calls
I20260809 12:10:58.244594 9896 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P c52389485e85465f849e792576b5e75c: No bootstrap required, opened a new log
Aug 09 12:10:58 dist-test-slave-2w3w krb5kdc[9807](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786277457, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.222.62@KRBTEST.COM
I20260809 12:10:58.253866 9896 raft_consensus.cc:348] T 00000000000000000000000000000000 P c52389485e85465f849e792576b5e75c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c52389485e85465f849e792576b5e75c" member_type: VOTER last_known_addr { host: "127.0.222.62" port: 44607 } }
I20260809 12:10:58.254211 9896 raft_consensus.cc:374] T 00000000000000000000000000000000 P c52389485e85465f849e792576b5e75c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260809 12:10:58.254343 9896 raft_consensus.cc:729] T 00000000000000000000000000000000 P c52389485e85465f849e792576b5e75c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c52389485e85465f849e792576b5e75c, State: Initialized, Role: FOLLOWER
I20260809 12:10:58.255090 9896 consensus_queue.cc:260] T 00000000000000000000000000000000 P c52389485e85465f849e792576b5e75c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c52389485e85465f849e792576b5e75c" member_type: VOTER last_known_addr { host: "127.0.222.62" port: 44607 } }
I20260809 12:10:58.255364 9896 raft_consensus.cc:388] T 00000000000000000000000000000000 P c52389485e85465f849e792576b5e75c [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260809 12:10:58.255510 9896 raft_consensus.cc:482] T 00000000000000000000000000000000 P c52389485e85465f849e792576b5e75c [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260809 12:10:58.255787 9896 raft_consensus.cc:3037] T 00000000000000000000000000000000 P c52389485e85465f849e792576b5e75c [term 0 FOLLOWER]: Advancing to term 1
I20260809 12:10:58.256847 9899 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 12:10:58.234871 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60814 (local address 127.0.222.62:44607)
0809 12:10:58.235747 (+ 876us) server_negotiation.cc:206] Beginning negotiation
0809 12:10:58.235760 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0809 12:10:58.235801 (+ 41us) server_negotiation.cc:407] Connection header received
0809 12:10:58.237541 (+ 1740us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 12:10:58.237564 (+ 23us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 12:10:58.237927 (+ 363us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 12:10:58.238246 (+ 319us) server_negotiation.cc:226] Negotiated authn=SASL
0809 12:10:58.240127 (+ 1881us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 12:10:58.241581 (+ 1454us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 12:10:58.243207 (+ 1626us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 12:10:58.243880 (+ 673us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 12:10:58.244014 (+ 134us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 12:10:58.249003 (+ 4989us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 12:10:58.249011 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 12:10:58.249028 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0809 12:10:58.249063 (+ 35us) server_negotiation.cc:922] calling sasl_server_start()
0809 12:10:58.252301 (+ 3238us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 12:10:58.253124 (+ 823us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 12:10:58.253130 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 12:10:58.253136 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0809 12:10:58.253306 (+ 170us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 12:10:58.253756 (+ 450us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 12:10:58.253760 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 12:10:58.253762 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0809 12:10:58.254453 (+ 691us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 12:10:58.254821 (+ 368us) server_negotiation.cc:1035] Waiting for connection context
0809 12:10:58.255348 (+ 527us) server_negotiation.cc:299] Negotiation successful
0809 12:10:58.255787 (+ 439us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":664,"thread_start_us":396,"threads_started":1}
I20260809 12:10:58.258267 9896 raft_consensus.cc:504] T 00000000000000000000000000000000 P c52389485e85465f849e792576b5e75c [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c52389485e85465f849e792576b5e75c" member_type: VOTER last_known_addr { host: "127.0.222.62" port: 44607 } }
I20260809 12:10:58.258780 9896 leader_election.cc:302] T 00000000000000000000000000000000 P c52389485e85465f849e792576b5e75c [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: c52389485e85465f849e792576b5e75c; no voters:
I20260809 12:10:58.259544 9896 leader_election.cc:288] T 00000000000000000000000000000000 P c52389485e85465f849e792576b5e75c [CANDIDATE]: Term 1 election: Requested vote from peers
I20260809 12:10:58.259670 9902 raft_consensus.cc:2781] T 00000000000000000000000000000000 P c52389485e85465f849e792576b5e75c [term 1 FOLLOWER]: Leader election won for term 1
I20260809 12:10:58.260396 9902 raft_consensus.cc:686] T 00000000000000000000000000000000 P c52389485e85465f849e792576b5e75c [term 1 LEADER]: Becoming Leader. State: Replica: c52389485e85465f849e792576b5e75c, State: Running, Role: LEADER
I20260809 12:10:58.261036 9902 consensus_queue.cc:237] T 00000000000000000000000000000000 P c52389485e85465f849e792576b5e75c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c52389485e85465f849e792576b5e75c" member_type: VOTER last_known_addr { host: "127.0.222.62" port: 44607 } }
I20260809 12:10:58.261828 9896 sys_catalog.cc:564] T 00000000000000000000000000000000 P c52389485e85465f849e792576b5e75c [sys.catalog]: configured and running, proceeding with master startup.
I20260809 12:10:58.263792 9904 sys_catalog.cc:455] T 00000000000000000000000000000000 P c52389485e85465f849e792576b5e75c [sys.catalog]: SysCatalogTable state changed. Reason: New leader c52389485e85465f849e792576b5e75c. Latest consensus state: current_term: 1 leader_uuid: "c52389485e85465f849e792576b5e75c" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c52389485e85465f849e792576b5e75c" member_type: VOTER last_known_addr { host: "127.0.222.62" port: 44607 } } }
I20260809 12:10:58.264169 9904 sys_catalog.cc:458] T 00000000000000000000000000000000 P c52389485e85465f849e792576b5e75c [sys.catalog]: This master's current role is: LEADER
I20260809 12:10:58.267252 9903 sys_catalog.cc:455] T 00000000000000000000000000000000 P c52389485e85465f849e792576b5e75c [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "c52389485e85465f849e792576b5e75c" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c52389485e85465f849e792576b5e75c" member_type: VOTER last_known_addr { host: "127.0.222.62" port: 44607 } } }
I20260809 12:10:58.267748 9903 sys_catalog.cc:458] T 00000000000000000000000000000000 P c52389485e85465f849e792576b5e75c [sys.catalog]: This master's current role is: LEADER
I20260809 12:10:58.268078 9907 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260809 12:10:58.274503 9907 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260809 12:10:58.284296 9907 catalog_manager.cc:1347] Generated new cluster ID: 9ebfe4d5d9644fc19ed95027331d83e2
I20260809 12:10:58.284433 9907 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260809 12:10:58.296586 9907 catalog_manager.cc:1370] Generated new certificate authority record
I20260809 12:10:58.297891 9907 catalog_manager.cc:1504] Loading token signing keys...
I20260809 12:10:58.315814 9907 catalog_manager.cc:5868] T 00000000000000000000000000000000 P c52389485e85465f849e792576b5e75c: Generated new TSK 0
I20260809 12:10:58.316682 9907 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.0.222.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.222.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.222.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskgH5Pjm/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786277382073938-888-0/krb5kdc/kudu_127.0.222.1.keytab.
Entry for principal kudu/127.0.222.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskgH5Pjm/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786277382073938-888-0/krb5kdc/kudu_127.0.222.1.keytab.
I20260809 12:10:58.388118 888 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskgH5Pjm/build/asan/bin/kudu
/tmp/dist-test-taskgH5Pjm/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskgH5Pjm/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786277382073938-888-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskgH5Pjm/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786277382073938-888-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskgH5Pjm/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786277382073938-888-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskgH5Pjm/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786277382073938-888-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.222.1:0
--local_ip_for_outbound_sockets=127.0.222.1
--webserver_interface=127.0.222.1
--webserver_port=0
--tserver_master_addrs=127.0.222.62:44607
--keytab_file=/tmp/dist-test-taskgH5Pjm/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786277382073938-888-0/krb5kdc/kudu_127.0.222.1.keytab
--principal=kudu/127.0.222.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.222.20:43745
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskgH5Pjm/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786277382073938-888-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskgH5Pjm/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786277382073938-888-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskgH5Pjm/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786277382073938-888-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260809 12:10:58.837086 9926 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260809 12:10:58.837728 9926 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260809 12:10:58.837879 9926 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260809 12:10:58.851459 9926 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260809 12:10:58.851605 9926 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260809 12:10:58.851795 9926 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.222.1
I20260809 12:10:58.867810 9926 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.222.20:43745
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskgH5Pjm/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786277382073938-888-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskgH5Pjm/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786277382073938-888-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskgH5Pjm/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786277382073938-888-0/krb5kdc/kudu_127.0.222.1.keytab
--principal=kudu/127.0.222.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.222.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskgH5Pjm/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786277382073938-888-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.222.1
--webserver_port=0
--tserver_master_addrs=127.0.222.62:44607
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.222.1
--log_dir=/tmp/dist-test-taskgH5Pjm/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786277382073938-888-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 09 Aug 2026 11:06:41 UTC on ccycloud.quasar-bwoogy.root.comops.site
build id 4575991
ASAN enabled
I20260809 12:10:58.869925 9926 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260809 12:10:58.872432 9926 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260809 12:10:58.892961 9934 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 12:10:58.893051 9932 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 12:10:58.893110 9931 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260809 12:10:58.899194 9926 server_base.cc:1034] running on GCE node
I20260809 12:10:58.900228 9926 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260809 12:10:58.901674 9926 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260809 12:10:58.903040 9926 hybrid_clock.cc:648] HybridClock initialized: now 1786277458902913 us; error 59 us; skew 500 ppm
I20260809 12:10:58.903542 9926 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 09 12:10:58 dist-test-slave-2w3w krb5kdc[9807](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786277458, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.222.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260809 12:10:58.907833 9926 init.cc:377] Logged in from keytab as kudu/127.0.222.1@KRBTEST.COM (short username kudu)
I20260809 12:10:58.909415 9926 webserver.cc:460] Webserver started at http://127.0.222.1:46535/ using document root <none> and password file <none>
I20260809 12:10:58.910173 9926 fs_manager.cc:359] Metadata directory not provided
I20260809 12:10:58.910302 9926 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260809 12:10:58.910651 9926 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260809 12:10:58.913884 9926 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskgH5Pjm/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786277382073938-888-0/minicluster-data/ts-0/data/instance:
uuid: "87e053888a014cebb5c16aa941bbf6f7"
format_stamp: "Formatted at 2026-08-09 12:10:58 on dist-test-slave-2w3w"
server_key: "271a6d02bdd4a416d7fb5c619edbda4e"
server_key_iv: "1e2ee96fbbaee301781e1f9a9be365e9"
server_key_version: "encryptionkey@0"
I20260809 12:10:58.914731 9926 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskgH5Pjm/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786277382073938-888-0/minicluster-data/ts-0/wal/instance:
uuid: "87e053888a014cebb5c16aa941bbf6f7"
format_stamp: "Formatted at 2026-08-09 12:10:58 on dist-test-slave-2w3w"
server_key: "271a6d02bdd4a416d7fb5c619edbda4e"
server_key_iv: "1e2ee96fbbaee301781e1f9a9be365e9"
server_key_version: "encryptionkey@0"
I20260809 12:10:58.920296 9926 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.003s sys 0.001s
I20260809 12:10:58.924242 9942 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260809 12:10:58.925616 9926 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.000s sys 0.004s
I20260809 12:10:58.925800 9926 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskgH5Pjm/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786277382073938-888-0/minicluster-data/ts-0/data,/tmp/dist-test-taskgH5Pjm/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786277382073938-888-0/minicluster-data/ts-0/wal
uuid: "87e053888a014cebb5c16aa941bbf6f7"
format_stamp: "Formatted at 2026-08-09 12:10:58 on dist-test-slave-2w3w"
server_key: "271a6d02bdd4a416d7fb5c619edbda4e"
server_key_iv: "1e2ee96fbbaee301781e1f9a9be365e9"
server_key_version: "encryptionkey@0"
I20260809 12:10:58.926035 9926 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskgH5Pjm/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786277382073938-888-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskgH5Pjm/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786277382073938-888-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskgH5Pjm/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786277382073938-888-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260809 12:10:58.944747 9926 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260809 12:10:58.948974 9926 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260809 12:10:58.949271 9926 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260809 12:10:58.950310 9926 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260809 12:10:58.951828 9926 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260809 12:10:58.951913 9926 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260809 12:10:58.952045 9926 ts_tablet_manager.cc:610] Registered 0 tablets
I20260809 12:10:58.952106 9926 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260809 12:10:58.987600 9926 rpc_server.cc:307] RPC server started. Bound to: 127.0.222.1:34291
I20260809 12:10:58.987679 10065 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.222.1:34291 every 8 connection(s)
I20260809 12:10:58.989001 9926 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskgH5Pjm/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786277382073938-888-0/minicluster-data/ts-0/data/info.pb
I20260809 12:10:58.994194 888 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskgH5Pjm/build/asan/bin/kudu as pid 9926
I20260809 12:10:58.994405 888 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskgH5Pjm/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786277382073938-888-0/minicluster-data/ts-0/wal/instance
I20260809 12:10:58.995153 888 external_mini_cluster.cc:1428] Setting key 0d30472897fe8e3cfdd1764bb4f1f064
Aug 09 12:10:58 dist-test-slave-2w3w krb5kdc[9807](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786277458, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.222.1@KRBTEST.COM for kudu/127.0.222.62@KRBTEST.COM
I20260809 12:10:59.010808 10068 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 12:10:58.991693 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.222.1:42633 (local address 127.0.222.62:44607)
0809 12:10:58.992154 (+ 461us) server_negotiation.cc:206] Beginning negotiation
0809 12:10:58.992159 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0809 12:10:58.993136 (+ 977us) server_negotiation.cc:407] Connection header received
0809 12:10:58.995168 (+ 2032us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 12:10:58.995174 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 12:10:58.995270 (+ 96us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 12:10:58.995407 (+ 137us) server_negotiation.cc:226] Negotiated authn=SASL
0809 12:10:58.997188 (+ 1781us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 12:10:58.997903 (+ 715us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 12:10:58.999578 (+ 1675us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 12:10:59.000011 (+ 433us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 12:10:59.000166 (+ 155us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 12:10:59.003822 (+ 3656us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 12:10:59.003830 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 12:10:59.003838 (+ 8us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0809 12:10:59.003885 (+ 47us) server_negotiation.cc:922] calling sasl_server_start()
0809 12:10:59.007458 (+ 3573us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 12:10:59.008364 (+ 906us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 12:10:59.008368 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 12:10:59.008370 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0809 12:10:59.008488 (+ 118us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 12:10:59.009014 (+ 526us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 12:10:59.009018 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 12:10:59.009020 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0809 12:10:59.009290 (+ 270us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 12:10:59.009416 (+ 126us) server_negotiation.cc:1035] Waiting for connection context
0809 12:10:59.010356 (+ 940us) server_negotiation.cc:299] Negotiation successful
0809 12:10:59.010538 (+ 182us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":343,"thread_start_us":236,"threads_started":1}
I20260809 12:10:59.012418 10070 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 12:10:58.992241 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.222.62:44607 (local address 127.0.222.1:42633)
0809 12:10:58.992959 (+ 718us) negotiation.cc:107] Waiting for socket to connect
0809 12:10:58.993004 (+ 45us) client_negotiation.cc:174] Beginning negotiation
0809 12:10:58.994840 (+ 1836us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 12:10:58.995658 (+ 818us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 12:10:58.995666 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 12:10:58.996260 (+ 594us) client_negotiation.cc:189] Negotiated authn=SASL
0809 12:10:58.997003 (+ 743us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 12:10:58.997023 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 12:10:58.998079 (+ 1056us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 12:10:58.998083 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 12:10:58.999425 (+ 1342us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 12:10:58.999438 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 12:10:59.000271 (+ 833us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 12:10:59.000279 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 12:10:59.000518 (+ 239us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 12:10:59.001224 (+ 706us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0809 12:10:59.001253 (+ 29us) client_negotiation.cc:656] calling sasl_client_start()
0809 12:10:59.003603 (+ 2350us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0809 12:10:59.007607 (+ 4004us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0809 12:10:59.007614 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0809 12:10:59.007628 (+ 14us) client_negotiation.cc:760] calling sasl_client_step()
0809 12:10:59.008218 (+ 590us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0809 12:10:59.008602 (+ 384us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0809 12:10:59.008608 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0809 12:10:59.008614 (+ 6us) client_negotiation.cc:760] calling sasl_client_step()
0809 12:10:59.008892 (+ 278us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0809 12:10:59.009444 (+ 552us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0809 12:10:59.009454 (+ 10us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0809 12:10:59.009929 (+ 475us) client_negotiation.cc:769] Sending connection context
0809 12:10:59.010240 (+ 311us) client_negotiation.cc:240] Negotiation successful
0809 12:10:59.010837 (+ 597us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":483,"thread_start_us":280,"threads_started":1}
I20260809 12:10:59.014015 10066 heartbeater.cc:347] Connected to a master server at 127.0.222.62:44607
I20260809 12:10:59.014321 10066 heartbeater.cc:464] Registering TS with master...
I20260809 12:10:59.015156 10066 heartbeater.cc:511] Master 127.0.222.62:44607 requested a full tablet report, sending...
I20260809 12:10:59.017242 9857 ts_manager.cc:194] Registered new tserver with Master: 87e053888a014cebb5c16aa941bbf6f7 (127.0.222.1:34291)
I20260809 12:10:59.018733 9857 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.222.1@KRBTEST.COM'} at 127.0.222.1:42633
WARNING: no policy specified for kudu/127.0.222.2@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.222.2@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.222.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskgH5Pjm/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786277382073938-888-0/krb5kdc/kudu_127.0.222.2.keytab.
Entry for principal kudu/127.0.222.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskgH5Pjm/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786277382073938-888-0/krb5kdc/kudu_127.0.222.2.keytab.
I20260809 12:10:59.060545 888 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskgH5Pjm/build/asan/bin/kudu
/tmp/dist-test-taskgH5Pjm/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskgH5Pjm/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786277382073938-888-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskgH5Pjm/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786277382073938-888-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskgH5Pjm/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786277382073938-888-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskgH5Pjm/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786277382073938-888-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.222.2:0
--local_ip_for_outbound_sockets=127.0.222.2
--webserver_interface=127.0.222.2
--webserver_port=0
--tserver_master_addrs=127.0.222.62:44607
--keytab_file=/tmp/dist-test-taskgH5Pjm/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786277382073938-888-0/krb5kdc/kudu_127.0.222.2.keytab
--principal=kudu/127.0.222.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.222.20:43745
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskgH5Pjm/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786277382073938-888-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskgH5Pjm/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786277382073938-888-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskgH5Pjm/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786277382073938-888-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260809 12:10:59.397090 10073 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260809 12:10:59.397365 10073 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260809 12:10:59.397461 10073 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260809 12:10:59.404735 10073 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260809 12:10:59.404855 10073 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260809 12:10:59.404978 10073 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.222.2
I20260809 12:10:59.415114 10073 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.222.20:43745
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskgH5Pjm/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786277382073938-888-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskgH5Pjm/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786277382073938-888-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskgH5Pjm/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786277382073938-888-0/krb5kdc/kudu_127.0.222.2.keytab
--principal=kudu/127.0.222.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.222.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskgH5Pjm/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786277382073938-888-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.222.2
--webserver_port=0
--tserver_master_addrs=127.0.222.62:44607
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.222.2
--log_dir=/tmp/dist-test-taskgH5Pjm/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786277382073938-888-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 09 Aug 2026 11:06:41 UTC on ccycloud.quasar-bwoogy.root.comops.site
build id 4575991
ASAN enabled
I20260809 12:10:59.416528 10073 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260809 12:10:59.417922 10073 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260809 12:10:59.431664 10079 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 12:10:59.431588 10080 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 12:10:59.431553 10082 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260809 12:10:59.432729 10073 server_base.cc:1034] running on GCE node
I20260809 12:10:59.433362 10073 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260809 12:10:59.434309 10073 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260809 12:10:59.435564 10073 hybrid_clock.cc:648] HybridClock initialized: now 1786277459435514 us; error 41 us; skew 500 ppm
I20260809 12:10:59.435920 10073 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 09 12:10:59 dist-test-slave-2w3w krb5kdc[9807](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786277459, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.222.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260809 12:10:59.439363 10073 init.cc:377] Logged in from keytab as kudu/127.0.222.2@KRBTEST.COM (short username kudu)
I20260809 12:10:59.440927 10073 webserver.cc:460] Webserver started at http://127.0.222.2:39473/ using document root <none> and password file <none>
I20260809 12:10:59.441644 10073 fs_manager.cc:359] Metadata directory not provided
I20260809 12:10:59.441830 10073 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260809 12:10:59.442173 10073 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260809 12:10:59.445250 10073 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskgH5Pjm/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786277382073938-888-0/minicluster-data/ts-1/data/instance:
uuid: "03830909e0584c7e90924ae9749b9849"
format_stamp: "Formatted at 2026-08-09 12:10:59 on dist-test-slave-2w3w"
server_key: "ab11351ef1ac9b2fb3afae599651734d"
server_key_iv: "88166c92df7d187ae91aca69147a8b17"
server_key_version: "encryptionkey@0"
I20260809 12:10:59.446076 10073 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskgH5Pjm/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786277382073938-888-0/minicluster-data/ts-1/wal/instance:
uuid: "03830909e0584c7e90924ae9749b9849"
format_stamp: "Formatted at 2026-08-09 12:10:59 on dist-test-slave-2w3w"
server_key: "ab11351ef1ac9b2fb3afae599651734d"
server_key_iv: "88166c92df7d187ae91aca69147a8b17"
server_key_version: "encryptionkey@0"
I20260809 12:10:59.451319 10073 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.007s sys 0.000s
I20260809 12:10:59.454831 10090 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260809 12:10:59.456233 10073 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260809 12:10:59.456416 10073 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskgH5Pjm/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786277382073938-888-0/minicluster-data/ts-1/data,/tmp/dist-test-taskgH5Pjm/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786277382073938-888-0/minicluster-data/ts-1/wal
uuid: "03830909e0584c7e90924ae9749b9849"
format_stamp: "Formatted at 2026-08-09 12:10:59 on dist-test-slave-2w3w"
server_key: "ab11351ef1ac9b2fb3afae599651734d"
server_key_iv: "88166c92df7d187ae91aca69147a8b17"
server_key_version: "encryptionkey@0"
I20260809 12:10:59.456640 10073 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskgH5Pjm/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786277382073938-888-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskgH5Pjm/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786277382073938-888-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskgH5Pjm/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786277382073938-888-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260809 12:10:59.531445 10073 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260809 12:10:59.536082 10073 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260809 12:10:59.536399 10073 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260809 12:10:59.537503 10073 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260809 12:10:59.538975 10073 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260809 12:10:59.539086 10073 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260809 12:10:59.539239 10073 ts_tablet_manager.cc:610] Registered 0 tablets
I20260809 12:10:59.539319 10073 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260809 12:10:59.573912 10073 rpc_server.cc:307] RPC server started. Bound to: 127.0.222.2:39033
I20260809 12:10:59.574008 10212 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.222.2:39033 every 8 connection(s)
I20260809 12:10:59.575328 10073 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskgH5Pjm/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786277382073938-888-0/minicluster-data/ts-1/data/info.pb
I20260809 12:10:59.579595 888 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskgH5Pjm/build/asan/bin/kudu as pid 10073
I20260809 12:10:59.579777 888 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskgH5Pjm/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786277382073938-888-0/minicluster-data/ts-1/wal/instance
I20260809 12:10:59.580247 888 external_mini_cluster.cc:1428] Setting key 813b1f34db86b10599858473bc7b5967
Aug 09 12:10:59 dist-test-slave-2w3w krb5kdc[9807](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786277459, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.222.2@KRBTEST.COM for kudu/127.0.222.62@KRBTEST.COM
I20260809 12:10:59.599282 10216 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 12:10:59.578188 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.222.2:50875 (local address 127.0.222.62:44607)
0809 12:10:59.578667 (+ 479us) server_negotiation.cc:206] Beginning negotiation
0809 12:10:59.578671 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0809 12:10:59.579511 (+ 840us) server_negotiation.cc:407] Connection header received
0809 12:10:59.581661 (+ 2150us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 12:10:59.581667 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 12:10:59.581759 (+ 92us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 12:10:59.581894 (+ 135us) server_negotiation.cc:226] Negotiated authn=SASL
0809 12:10:59.583957 (+ 2063us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 12:10:59.584704 (+ 747us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 12:10:59.587255 (+ 2551us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 12:10:59.587703 (+ 448us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 12:10:59.587806 (+ 103us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 12:10:59.592700 (+ 4894us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 12:10:59.592710 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 12:10:59.592717 (+ 7us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0809 12:10:59.592768 (+ 51us) server_negotiation.cc:922] calling sasl_server_start()
0809 12:10:59.595954 (+ 3186us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 12:10:59.596776 (+ 822us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 12:10:59.596786 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 12:10:59.596789 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0809 12:10:59.596927 (+ 138us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 12:10:59.597408 (+ 481us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 12:10:59.597412 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 12:10:59.597415 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0809 12:10:59.597702 (+ 287us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 12:10:59.597893 (+ 191us) server_negotiation.cc:1035] Waiting for connection context
0809 12:10:59.598621 (+ 728us) server_negotiation.cc:299] Negotiation successful
0809 12:10:59.598860 (+ 239us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":372,"thread_start_us":260,"threads_started":1}
I20260809 12:10:59.600301 10217 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 12:10:59.578613 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.222.62:44607 (local address 127.0.222.2:50875)
0809 12:10:59.579348 (+ 735us) negotiation.cc:107] Waiting for socket to connect
0809 12:10:59.579395 (+ 47us) client_negotiation.cc:174] Beginning negotiation
0809 12:10:59.581393 (+ 1998us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 12:10:59.582137 (+ 744us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 12:10:59.582145 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 12:10:59.582906 (+ 761us) client_negotiation.cc:189] Negotiated authn=SASL
0809 12:10:59.583745 (+ 839us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 12:10:59.583772 (+ 27us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 12:10:59.584891 (+ 1119us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 12:10:59.584900 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 12:10:59.587105 (+ 2205us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 12:10:59.587123 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 12:10:59.587954 (+ 831us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 12:10:59.587963 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 12:10:59.588302 (+ 339us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 12:10:59.589153 (+ 851us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0809 12:10:59.589189 (+ 36us) client_negotiation.cc:656] calling sasl_client_start()
0809 12:10:59.592372 (+ 3183us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0809 12:10:59.596103 (+ 3731us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0809 12:10:59.596110 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0809 12:10:59.596124 (+ 14us) client_negotiation.cc:760] calling sasl_client_step()
0809 12:10:59.596621 (+ 497us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0809 12:10:59.597057 (+ 436us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0809 12:10:59.597063 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0809 12:10:59.597067 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0809 12:10:59.597269 (+ 202us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0809 12:10:59.597850 (+ 581us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0809 12:10:59.597860 (+ 10us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0809 12:10:59.598228 (+ 368us) client_negotiation.cc:769] Sending connection context
0809 12:10:59.598512 (+ 284us) client_negotiation.cc:240] Negotiation successful
0809 12:10:59.599133 (+ 621us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":521,"thread_start_us":337,"threads_started":1}
I20260809 12:10:59.601893 10213 heartbeater.cc:347] Connected to a master server at 127.0.222.62:44607
I20260809 12:10:59.602249 10213 heartbeater.cc:464] Registering TS with master...
I20260809 12:10:59.603217 10213 heartbeater.cc:511] Master 127.0.222.62:44607 requested a full tablet report, sending...
I20260809 12:10:59.604935 9857 ts_manager.cc:194] Registered new tserver with Master: 03830909e0584c7e90924ae9749b9849 (127.0.222.2:39033)
I20260809 12:10:59.606240 9857 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.222.2@KRBTEST.COM'} at 127.0.222.2:50875
WARNING: no policy specified for kudu/127.0.222.3@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.222.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.222.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskgH5Pjm/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786277382073938-888-0/krb5kdc/kudu_127.0.222.3.keytab.
Entry for principal kudu/127.0.222.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskgH5Pjm/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786277382073938-888-0/krb5kdc/kudu_127.0.222.3.keytab.
I20260809 12:10:59.648012 888 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskgH5Pjm/build/asan/bin/kudu
/tmp/dist-test-taskgH5Pjm/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskgH5Pjm/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786277382073938-888-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskgH5Pjm/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786277382073938-888-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskgH5Pjm/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786277382073938-888-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskgH5Pjm/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786277382073938-888-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.222.3:0
--local_ip_for_outbound_sockets=127.0.222.3
--webserver_interface=127.0.222.3
--webserver_port=0
--tserver_master_addrs=127.0.222.62:44607
--keytab_file=/tmp/dist-test-taskgH5Pjm/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786277382073938-888-0/krb5kdc/kudu_127.0.222.3.keytab
--principal=kudu/127.0.222.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.222.20:43745
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskgH5Pjm/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786277382073938-888-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskgH5Pjm/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786277382073938-888-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskgH5Pjm/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786277382073938-888-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260809 12:10:59.982059 10220 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260809 12:10:59.982347 10220 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260809 12:10:59.982441 10220 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260809 12:10:59.991539 10220 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260809 12:10:59.991679 10220 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260809 12:10:59.991874 10220 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.222.3
I20260809 12:11:00.002503 10220 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.222.20:43745
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskgH5Pjm/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786277382073938-888-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskgH5Pjm/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786277382073938-888-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskgH5Pjm/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786277382073938-888-0/krb5kdc/kudu_127.0.222.3.keytab
--principal=kudu/127.0.222.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.222.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskgH5Pjm/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786277382073938-888-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.222.3
--webserver_port=0
--tserver_master_addrs=127.0.222.62:44607
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.222.3
--log_dir=/tmp/dist-test-taskgH5Pjm/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786277382073938-888-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 09 Aug 2026 11:06:41 UTC on ccycloud.quasar-bwoogy.root.comops.site
build id 4575991
ASAN enabled
I20260809 12:11:00.003921 10220 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260809 12:11:00.005198 10220 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260809 12:11:00.018468 10225 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 12:11:00.018647 10226 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 12:11:00.018671 10228 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260809 12:11:00.019662 10220 server_base.cc:1034] running on GCE node
I20260809 12:11:00.020244 10220 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260809 12:11:00.021174 10220 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260809 12:11:00.021708 10066 heartbeater.cc:503] Master 127.0.222.62:44607 was elected leader, sending a full tablet report...
I20260809 12:11:00.022418 10220 hybrid_clock.cc:648] HybridClock initialized: now 1786277460022363 us; error 51 us; skew 500 ppm
I20260809 12:11:00.022755 10220 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 09 12:11:00 dist-test-slave-2w3w krb5kdc[9807](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786277460, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.222.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260809 12:11:00.026033 10220 init.cc:377] Logged in from keytab as kudu/127.0.222.3@KRBTEST.COM (short username kudu)
I20260809 12:11:00.027683 10220 webserver.cc:460] Webserver started at http://127.0.222.3:35869/ using document root <none> and password file <none>
I20260809 12:11:00.028389 10220 fs_manager.cc:359] Metadata directory not provided
I20260809 12:11:00.028548 10220 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260809 12:11:00.028879 10220 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260809 12:11:00.032055 10220 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskgH5Pjm/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786277382073938-888-0/minicluster-data/ts-2/data/instance:
uuid: "6bf090a102124104ac7909e02e744847"
format_stamp: "Formatted at 2026-08-09 12:11:00 on dist-test-slave-2w3w"
server_key: "e4aa8907f8c8ade76fc1194c4d20bad1"
server_key_iv: "b762426b3ca726a8e6cfedbf872e0b8e"
server_key_version: "encryptionkey@0"
I20260809 12:11:00.032877 10220 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskgH5Pjm/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786277382073938-888-0/minicluster-data/ts-2/wal/instance:
uuid: "6bf090a102124104ac7909e02e744847"
format_stamp: "Formatted at 2026-08-09 12:11:00 on dist-test-slave-2w3w"
server_key: "e4aa8907f8c8ade76fc1194c4d20bad1"
server_key_iv: "b762426b3ca726a8e6cfedbf872e0b8e"
server_key_version: "encryptionkey@0"
I20260809 12:11:00.038319 10220 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.001s
I20260809 12:11:00.041816 10237 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260809 12:11:00.043344 10220 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.002s
I20260809 12:11:00.043538 10220 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskgH5Pjm/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786277382073938-888-0/minicluster-data/ts-2/data,/tmp/dist-test-taskgH5Pjm/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786277382073938-888-0/minicluster-data/ts-2/wal
uuid: "6bf090a102124104ac7909e02e744847"
format_stamp: "Formatted at 2026-08-09 12:11:00 on dist-test-slave-2w3w"
server_key: "e4aa8907f8c8ade76fc1194c4d20bad1"
server_key_iv: "b762426b3ca726a8e6cfedbf872e0b8e"
server_key_version: "encryptionkey@0"
I20260809 12:11:00.043754 10220 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskgH5Pjm/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786277382073938-888-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskgH5Pjm/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786277382073938-888-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskgH5Pjm/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786277382073938-888-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260809 12:11:00.079135 10220 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260809 12:11:00.083372 10220 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260809 12:11:00.083690 10220 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260809 12:11:00.084762 10220 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260809 12:11:00.086205 10220 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260809 12:11:00.086329 10220 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260809 12:11:00.086485 10220 ts_tablet_manager.cc:610] Registered 0 tablets
I20260809 12:11:00.086560 10220 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260809 12:11:00.120577 10220 rpc_server.cc:307] RPC server started. Bound to: 127.0.222.3:35897
I20260809 12:11:00.120615 10357 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.222.3:35897 every 8 connection(s)
I20260809 12:11:00.121927 10220 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskgH5Pjm/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786277382073938-888-0/minicluster-data/ts-2/data/info.pb
I20260809 12:11:00.125602 888 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskgH5Pjm/build/asan/bin/kudu as pid 10220
I20260809 12:11:00.125813 888 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskgH5Pjm/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786277382073938-888-0/minicluster-data/ts-2/wal/instance
I20260809 12:11:00.126351 888 external_mini_cluster.cc:1428] Setting key ce80a32dd2e287cd45eb3366670a90fb
Aug 09 12:11:00 dist-test-slave-2w3w krb5kdc[9807](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786277460, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.222.3@KRBTEST.COM for kudu/127.0.222.62@KRBTEST.COM
I20260809 12:11:00.144018 10360 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 12:11:00.124683 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.222.3:51925 (local address 127.0.222.62:44607)
0809 12:11:00.125244 (+ 561us) server_negotiation.cc:206] Beginning negotiation
0809 12:11:00.125250 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0809 12:11:00.126226 (+ 976us) server_negotiation.cc:407] Connection header received
0809 12:11:00.128605 (+ 2379us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 12:11:00.128611 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 12:11:00.128728 (+ 117us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 12:11:00.128938 (+ 210us) server_negotiation.cc:226] Negotiated authn=SASL
0809 12:11:00.130771 (+ 1833us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 12:11:00.131633 (+ 862us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 12:11:00.133214 (+ 1581us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 12:11:00.133662 (+ 448us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 12:11:00.133800 (+ 138us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 12:11:00.137456 (+ 3656us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 12:11:00.137461 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 12:11:00.137465 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0809 12:11:00.137500 (+ 35us) server_negotiation.cc:922] calling sasl_server_start()
0809 12:11:00.140299 (+ 2799us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 12:11:00.141406 (+ 1107us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 12:11:00.141412 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 12:11:00.141415 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0809 12:11:00.141575 (+ 160us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 12:11:00.142129 (+ 554us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 12:11:00.142133 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 12:11:00.142139 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0809 12:11:00.142528 (+ 389us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 12:11:00.142678 (+ 150us) server_negotiation.cc:1035] Waiting for connection context
0809 12:11:00.143505 (+ 827us) server_negotiation.cc:299] Negotiation successful
0809 12:11:00.143691 (+ 186us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":422,"thread_start_us":263,"threads_started":1}
I20260809 12:11:00.145028 10361 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 12:11:00.125193 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.222.62:44607 (local address 127.0.222.3:51925)
0809 12:11:00.126043 (+ 850us) negotiation.cc:107] Waiting for socket to connect
0809 12:11:00.126088 (+ 45us) client_negotiation.cc:174] Beginning negotiation
0809 12:11:00.128268 (+ 2180us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 12:11:00.129267 (+ 999us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 12:11:00.129280 (+ 13us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 12:11:00.130026 (+ 746us) client_negotiation.cc:189] Negotiated authn=SASL
0809 12:11:00.130614 (+ 588us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 12:11:00.130631 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 12:11:00.131775 (+ 1144us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 12:11:00.131779 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 12:11:00.133037 (+ 1258us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 12:11:00.133057 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 12:11:00.133866 (+ 809us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 12:11:00.133871 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 12:11:00.134084 (+ 213us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 12:11:00.134822 (+ 738us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0809 12:11:00.134852 (+ 30us) client_negotiation.cc:656] calling sasl_client_start()
0809 12:11:00.137211 (+ 2359us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0809 12:11:00.140585 (+ 3374us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0809 12:11:00.140594 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0809 12:11:00.140611 (+ 17us) client_negotiation.cc:760] calling sasl_client_step()
0809 12:11:00.141230 (+ 619us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0809 12:11:00.141696 (+ 466us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0809 12:11:00.141699 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0809 12:11:00.141701 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0809 12:11:00.141991 (+ 290us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0809 12:11:00.142668 (+ 677us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0809 12:11:00.142675 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0809 12:11:00.143131 (+ 456us) client_negotiation.cc:769] Sending connection context
0809 12:11:00.143386 (+ 255us) client_negotiation.cc:240] Negotiation successful
0809 12:11:00.143816 (+ 430us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":575,"thread_start_us":363,"threads_started":1}
I20260809 12:11:00.146601 10358 heartbeater.cc:347] Connected to a master server at 127.0.222.62:44607
I20260809 12:11:00.146989 10358 heartbeater.cc:464] Registering TS with master...
I20260809 12:11:00.147769 10358 heartbeater.cc:511] Master 127.0.222.62:44607 requested a full tablet report, sending...
I20260809 12:11:00.149259 9857 ts_manager.cc:194] Registered new tserver with Master: 6bf090a102124104ac7909e02e744847 (127.0.222.3:35897)
I20260809 12:11:00.150102 9857 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.222.3@KRBTEST.COM'} at 127.0.222.3:51925
I20260809 12:11:00.154884 888 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
WARNING: no policy specified for oryx/127.0.222.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.0.222.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.0.222.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskgH5Pjm/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786277382073938-888-0/krb5kdc/oryx_127.0.222.1.keytab.
Entry for principal oryx/127.0.222.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskgH5Pjm/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786277382073938-888-0/krb5kdc/oryx_127.0.222.1.keytab.
WARNING: no policy specified for oryx/127.0.222.2@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.0.222.2@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.0.222.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskgH5Pjm/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786277382073938-888-0/krb5kdc/oryx_127.0.222.2.keytab.
Entry for principal oryx/127.0.222.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskgH5Pjm/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786277382073938-888-0/krb5kdc/oryx_127.0.222.2.keytab.
WARNING: no policy specified for oryx/127.0.222.3@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.0.222.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.0.222.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskgH5Pjm/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786277382073938-888-0/krb5kdc/oryx_127.0.222.3.keytab.
Entry for principal oryx/127.0.222.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskgH5Pjm/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786277382073938-888-0/krb5kdc/oryx_127.0.222.3.keytab.
I20260809 12:11:00.341519 888 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskgH5Pjm/build/asan/bin/kudu pid 9926
Aug 09 12:11:00 dist-test-slave-2w3w krb5kdc[9807](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786277457, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.222.1@KRBTEST.COM
I20260809 12:11:00.358837 10369 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 12:11:00.342655 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36570 (local address 127.0.222.1:34291)
0809 12:11:00.343453 (+ 798us) server_negotiation.cc:206] Beginning negotiation
0809 12:11:00.343461 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0809 12:11:00.343648 (+ 187us) server_negotiation.cc:407] Connection header received
0809 12:11:00.343795 (+ 147us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 12:11:00.343802 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 12:11:00.344036 (+ 234us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 12:11:00.344177 (+ 141us) server_negotiation.cc:226] Negotiated authn=SASL
0809 12:11:00.345750 (+ 1573us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 12:11:00.346803 (+ 1053us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 12:11:00.348198 (+ 1395us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 12:11:00.348897 (+ 699us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 12:11:00.349005 (+ 108us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 12:11:00.353201 (+ 4196us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 12:11:00.353212 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 12:11:00.353227 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0809 12:11:00.353265 (+ 38us) server_negotiation.cc:922] calling sasl_server_start()
0809 12:11:00.356211 (+ 2946us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 12:11:00.356876 (+ 665us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 12:11:00.356882 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 12:11:00.356884 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0809 12:11:00.357032 (+ 148us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 12:11:00.357360 (+ 328us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 12:11:00.357363 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 12:11:00.357365 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0809 12:11:00.357693 (+ 328us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 12:11:00.357914 (+ 221us) server_negotiation.cc:1035] Waiting for connection context
0809 12:11:00.358219 (+ 305us) server_negotiation.cc:299] Negotiation successful
0809 12:11:00.358455 (+ 236us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":602,"thread_start_us":382,"threads_started":1}
I20260809 12:11:00.360352 10061 generic_service.cc:196] Checking for leaks (request via RPC)
I20260809 12:11:00.446178 888 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskgH5Pjm/build/asan/bin/kudu with pid 9926
I20260809 12:11:00.467154 888 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskgH5Pjm/build/asan/bin/kudu pid 10073
Aug 09 12:11:00 dist-test-slave-2w3w krb5kdc[9807](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786277457, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.222.2@KRBTEST.COM
I20260809 12:11:00.483788 10372 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 12:11:00.468069 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40002 (local address 127.0.222.2:39033)
0809 12:11:00.468655 (+ 586us) server_negotiation.cc:206] Beginning negotiation
0809 12:11:00.468663 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0809 12:11:00.468688 (+ 25us) server_negotiation.cc:407] Connection header received
0809 12:11:00.468751 (+ 63us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 12:11:00.468758 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 12:11:00.468990 (+ 232us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 12:11:00.469118 (+ 128us) server_negotiation.cc:226] Negotiated authn=SASL
0809 12:11:00.470524 (+ 1406us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 12:11:00.471759 (+ 1235us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 12:11:00.473039 (+ 1280us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 12:11:00.473624 (+ 585us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 12:11:00.473723 (+ 99us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 12:11:00.477950 (+ 4227us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 12:11:00.477962 (+ 12us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 12:11:00.477976 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0809 12:11:00.478011 (+ 35us) server_negotiation.cc:922] calling sasl_server_start()
0809 12:11:00.480877 (+ 2866us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 12:11:00.481565 (+ 688us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 12:11:00.481572 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 12:11:00.481575 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0809 12:11:00.481698 (+ 123us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 12:11:00.482087 (+ 389us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 12:11:00.482093 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 12:11:00.482096 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0809 12:11:00.482428 (+ 332us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 12:11:00.482748 (+ 320us) server_negotiation.cc:1035] Waiting for connection context
0809 12:11:00.483187 (+ 439us) server_negotiation.cc:299] Negotiation successful
0809 12:11:00.483451 (+ 264us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":450,"thread_start_us":302,"threads_started":1}
I20260809 12:11:00.485064 10208 generic_service.cc:196] Checking for leaks (request via RPC)
I20260809 12:11:00.569922 888 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskgH5Pjm/build/asan/bin/kudu with pid 10073
I20260809 12:11:00.589218 888 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskgH5Pjm/build/asan/bin/kudu pid 10220
Aug 09 12:11:00 dist-test-slave-2w3w krb5kdc[9807](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786277457, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.222.3@KRBTEST.COM
I20260809 12:11:00.606423 10374 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 12:11:00.590290 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46164 (local address 127.0.222.3:35897)
0809 12:11:00.590785 (+ 495us) server_negotiation.cc:206] Beginning negotiation
0809 12:11:00.590792 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0809 12:11:00.590831 (+ 39us) server_negotiation.cc:407] Connection header received
0809 12:11:00.590893 (+ 62us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 12:11:00.590899 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 12:11:00.591207 (+ 308us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 12:11:00.591392 (+ 185us) server_negotiation.cc:226] Negotiated authn=SASL
0809 12:11:00.592826 (+ 1434us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 12:11:00.593792 (+ 966us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 12:11:00.595351 (+ 1559us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 12:11:00.595920 (+ 569us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 12:11:00.596067 (+ 147us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 12:11:00.600458 (+ 4391us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 12:11:00.600469 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 12:11:00.600482 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0809 12:11:00.600512 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0809 12:11:00.603576 (+ 3064us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 12:11:00.604241 (+ 665us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 12:11:00.604255 (+ 14us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 12:11:00.604258 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0809 12:11:00.604377 (+ 119us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 12:11:00.604772 (+ 395us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 12:11:00.604777 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 12:11:00.604781 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0809 12:11:00.605131 (+ 350us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 12:11:00.605458 (+ 327us) server_negotiation.cc:1035] Waiting for connection context
0809 12:11:00.605770 (+ 312us) server_negotiation.cc:299] Negotiation successful
0809 12:11:00.606050 (+ 280us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":347,"thread_start_us":206,"threads_started":1}
I20260809 12:11:00.607808 10351 generic_service.cc:196] Checking for leaks (request via RPC)
I20260809 12:11:00.691098 888 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskgH5Pjm/build/asan/bin/kudu with pid 10220
I20260809 12:11:00.710400 888 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskgH5Pjm/build/asan/bin/kudu pid 9820
I20260809 12:11:00.711735 9890 generic_service.cc:196] Checking for leaks (request via RPC)
I20260809 12:11:00.818280 888 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskgH5Pjm/build/asan/bin/kudu with pid 9820
I20260809 12:11:00.844375 888 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskgH5Pjm/build/asan/bin/kudu
/tmp/dist-test-taskgH5Pjm/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskgH5Pjm/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786277382073938-888-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskgH5Pjm/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786277382073938-888-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskgH5Pjm/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786277382073938-888-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskgH5Pjm/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786277382073938-888-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.222.62:44607
--webserver_interface=127.0.222.62
--webserver_port=42893
--keytab_file=/tmp/dist-test-taskgH5Pjm/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786277382073938-888-0/krb5kdc/kudu_127.0.222.62.keytab
--principal=kudu/127.0.222.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.222.20:43745
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.222.62:44607
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskgH5Pjm/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786277382073938-888-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskgH5Pjm/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786277382073938-888-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskgH5Pjm/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786277382073938-888-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260809 12:11:01.188226 10377 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260809 12:11:01.188535 10377 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260809 12:11:01.188676 10377 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260809 12:11:01.196143 10377 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260809 12:11:01.196262 10377 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260809 12:11:01.196309 10377 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260809 12:11:01.196360 10377 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260809 12:11:01.196406 10377 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260809 12:11:01.206247 10377 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.222.20:43745
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskgH5Pjm/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786277382073938-888-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskgH5Pjm/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786277382073938-888-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.222.62:44607
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskgH5Pjm/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786277382073938-888-0/krb5kdc/kudu_127.0.222.62.keytab
--principal=kudu/127.0.222.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.222.62:44607
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskgH5Pjm/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786277382073938-888-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.222.62
--webserver_port=42893
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskgH5Pjm/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786277382073938-888-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 09 Aug 2026 11:06:41 UTC on ccycloud.quasar-bwoogy.root.comops.site
build id 4575991
ASAN enabled
I20260809 12:11:01.207741 10377 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260809 12:11:01.209029 10377 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260809 12:11:01.221060 10383 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 12:11:01.221184 10384 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 12:11:01.221138 10386 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260809 12:11:01.222441 10377 server_base.cc:1034] running on GCE node
I20260809 12:11:01.223271 10377 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260809 12:11:01.224701 10377 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260809 12:11:01.225924 10377 hybrid_clock.cc:648] HybridClock initialized: now 1786277461225881 us; error 38 us; skew 500 ppm
I20260809 12:11:01.226230 10377 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 09 12:11:01 dist-test-slave-2w3w krb5kdc[9807](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786277461, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.222.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260809 12:11:01.229411 10377 init.cc:377] Logged in from keytab as kudu/127.0.222.62@KRBTEST.COM (short username kudu)
I20260809 12:11:01.231026 10377 webserver.cc:460] Webserver started at http://127.0.222.62:42893/ using document root <none> and password file <none>
I20260809 12:11:01.231750 10377 fs_manager.cc:359] Metadata directory not provided
I20260809 12:11:01.231917 10377 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260809 12:11:01.237460 10377 fs_manager.cc:709] Time spent opening directory manager: real 0.004s user 0.003s sys 0.003s
I20260809 12:11:01.240335 10394 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260809 12:11:01.241698 10377 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260809 12:11:01.241904 10377 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskgH5Pjm/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786277382073938-888-0/minicluster-data/master-0/data,/tmp/dist-test-taskgH5Pjm/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786277382073938-888-0/minicluster-data/master-0/wal
uuid: "c52389485e85465f849e792576b5e75c"
format_stamp: "Formatted at 2026-08-09 12:10:58 on dist-test-slave-2w3w"
server_key: "ad9476eb168bb14d98dfe596b0b7c4d9"
server_key_iv: "0455fd4b3eb144bc266e76b120c224c2"
server_key_version: "encryptionkey@0"
I20260809 12:11:01.242892 10377 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskgH5Pjm/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786277382073938-888-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskgH5Pjm/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786277382073938-888-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskgH5Pjm/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786277382073938-888-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260809 12:11:01.271483 10377 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260809 12:11:01.275671 10377 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260809 12:11:01.275969 10377 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260809 12:11:01.295449 10377 rpc_server.cc:307] RPC server started. Bound to: 127.0.222.62:44607
I20260809 12:11:01.295467 10454 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.222.62:44607 every 8 connection(s)
I20260809 12:11:01.297009 10377 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskgH5Pjm/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786277382073938-888-0/minicluster-data/master-0/data/info.pb
I20260809 12:11:01.302394 888 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskgH5Pjm/build/asan/bin/kudu as pid 10377
I20260809 12:11:01.303464 888 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskgH5Pjm/build/asan/bin/kudu
/tmp/dist-test-taskgH5Pjm/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskgH5Pjm/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786277382073938-888-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskgH5Pjm/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786277382073938-888-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskgH5Pjm/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786277382073938-888-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskgH5Pjm/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786277382073938-888-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.222.1:34291
--local_ip_for_outbound_sockets=127.0.222.1
--tserver_master_addrs=127.0.222.62:44607
--webserver_port=46535
--webserver_interface=127.0.222.1
--keytab_file=/tmp/dist-test-taskgH5Pjm/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786277382073938-888-0/krb5kdc/kudu_127.0.222.1.keytab
--principal=kudu/127.0.222.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.222.20:43745
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--principal=oryx/127.0.222.1
--keytab_file=/tmp/dist-test-taskgH5Pjm/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786277382073938-888-0/krb5kdc/oryx_127.0.222.1.keytab with env {KRB5CCNAME=/tmp/dist-test-taskgH5Pjm/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786277382073938-888-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskgH5Pjm/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786277382073938-888-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskgH5Pjm/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786277382073938-888-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260809 12:11:01.304972 10455 sys_catalog.cc:263] Verifying existing consensus state
I20260809 12:11:01.310310 10455 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P c52389485e85465f849e792576b5e75c: Bootstrap starting.
I20260809 12:11:01.331404 10455 log.cc:824] T 00000000000000000000000000000000 P c52389485e85465f849e792576b5e75c: Log is configured to *not* fsync() on all Append() calls
I20260809 12:11:01.340541 10455 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P c52389485e85465f849e792576b5e75c: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=3 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260809 12:11:01.341085 10455 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P c52389485e85465f849e792576b5e75c: Bootstrap complete.
I20260809 12:11:01.350847 10455 raft_consensus.cc:348] T 00000000000000000000000000000000 P c52389485e85465f849e792576b5e75c [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c52389485e85465f849e792576b5e75c" member_type: VOTER last_known_addr { host: "127.0.222.62" port: 44607 } }
I20260809 12:11:01.351816 10455 raft_consensus.cc:729] T 00000000000000000000000000000000 P c52389485e85465f849e792576b5e75c [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: c52389485e85465f849e792576b5e75c, State: Initialized, Role: FOLLOWER
I20260809 12:11:01.352428 10455 consensus_queue.cc:260] T 00000000000000000000000000000000 P c52389485e85465f849e792576b5e75c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c52389485e85465f849e792576b5e75c" member_type: VOTER last_known_addr { host: "127.0.222.62" port: 44607 } }
I20260809 12:11:01.352705 10455 raft_consensus.cc:388] T 00000000000000000000000000000000 P c52389485e85465f849e792576b5e75c [term 1 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260809 12:11:01.352845 10455 raft_consensus.cc:482] T 00000000000000000000000000000000 P c52389485e85465f849e792576b5e75c [term 1 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260809 12:11:01.353029 10455 raft_consensus.cc:3037] T 00000000000000000000000000000000 P c52389485e85465f849e792576b5e75c [term 1 FOLLOWER]: Advancing to term 2
I20260809 12:11:01.357043 10455 raft_consensus.cc:504] T 00000000000000000000000000000000 P c52389485e85465f849e792576b5e75c [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c52389485e85465f849e792576b5e75c" member_type: VOTER last_known_addr { host: "127.0.222.62" port: 44607 } }
I20260809 12:11:01.357556 10455 leader_election.cc:302] T 00000000000000000000000000000000 P c52389485e85465f849e792576b5e75c [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: c52389485e85465f849e792576b5e75c; no voters:
I20260809 12:11:01.358255 10455 leader_election.cc:288] T 00000000000000000000000000000000 P c52389485e85465f849e792576b5e75c [CANDIDATE]: Term 2 election: Requested vote from peers
I20260809 12:11:01.358403 10459 raft_consensus.cc:2781] T 00000000000000000000000000000000 P c52389485e85465f849e792576b5e75c [term 2 FOLLOWER]: Leader election won for term 2
I20260809 12:11:01.359760 10459 raft_consensus.cc:686] T 00000000000000000000000000000000 P c52389485e85465f849e792576b5e75c [term 2 LEADER]: Becoming Leader. State: Replica: c52389485e85465f849e792576b5e75c, State: Running, Role: LEADER
I20260809 12:11:01.360327 10459 consensus_queue.cc:237] T 00000000000000000000000000000000 P c52389485e85465f849e792576b5e75c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 2, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c52389485e85465f849e792576b5e75c" member_type: VOTER last_known_addr { host: "127.0.222.62" port: 44607 } }
I20260809 12:11:01.360718 10455 sys_catalog.cc:564] T 00000000000000000000000000000000 P c52389485e85465f849e792576b5e75c [sys.catalog]: configured and running, proceeding with master startup.
I20260809 12:11:01.363157 10460 sys_catalog.cc:455] T 00000000000000000000000000000000 P c52389485e85465f849e792576b5e75c [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 2 leader_uuid: "c52389485e85465f849e792576b5e75c" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c52389485e85465f849e792576b5e75c" member_type: VOTER last_known_addr { host: "127.0.222.62" port: 44607 } } }
I20260809 12:11:01.363613 10460 sys_catalog.cc:458] T 00000000000000000000000000000000 P c52389485e85465f849e792576b5e75c [sys.catalog]: This master's current role is: LEADER
I20260809 12:11:01.368346 10461 sys_catalog.cc:455] T 00000000000000000000000000000000 P c52389485e85465f849e792576b5e75c [sys.catalog]: SysCatalogTable state changed. Reason: New leader c52389485e85465f849e792576b5e75c. Latest consensus state: current_term: 2 leader_uuid: "c52389485e85465f849e792576b5e75c" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c52389485e85465f849e792576b5e75c" member_type: VOTER last_known_addr { host: "127.0.222.62" port: 44607 } } }
I20260809 12:11:01.368753 10461 sys_catalog.cc:458] T 00000000000000000000000000000000 P c52389485e85465f849e792576b5e75c [sys.catalog]: This master's current role is: LEADER
I20260809 12:11:01.371426 10465 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260809 12:11:01.377573 10465 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260809 12:11:01.379091 10465 catalog_manager.cc:1259] Loaded cluster ID: 9ebfe4d5d9644fc19ed95027331d83e2
I20260809 12:11:01.379262 10465 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260809 12:11:01.383925 10465 catalog_manager.cc:1504] Loading token signing keys...
I20260809 12:11:01.389268 10465 catalog_manager.cc:5879] T 00000000000000000000000000000000 P c52389485e85465f849e792576b5e75c: Loaded TSK: 0
I20260809 12:11:01.391274 10465 catalog_manager.cc:1514] Initializing in-progress tserver states...
W20260809 12:11:01.677595 10457 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260809 12:11:01.677897 10457 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260809 12:11:01.678049 10457 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260809 12:11:01.685604 10457 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260809 12:11:01.685719 10457 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260809 12:11:01.685899 10457 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.222.1
I20260809 12:11:01.696164 10457 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.222.20:43745
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskgH5Pjm/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786277382073938-888-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskgH5Pjm/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786277382073938-888-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskgH5Pjm/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786277382073938-888-0/krb5kdc/oryx_127.0.222.1.keytab
--principal=oryx/127.0.222.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.222.1:34291
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskgH5Pjm/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786277382073938-888-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.222.1
--webserver_port=46535
--tserver_master_addrs=127.0.222.62:44607
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.222.1
--log_dir=/tmp/dist-test-taskgH5Pjm/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786277382073938-888-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 09 Aug 2026 11:06:41 UTC on ccycloud.quasar-bwoogy.root.comops.site
build id 4575991
ASAN enabled
I20260809 12:11:01.697595 10457 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260809 12:11:01.698932 10457 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260809 12:11:01.713991 10484 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 12:11:01.714080 10486 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 12:11:01.714879 10483 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260809 12:11:01.715340 10457 server_base.cc:1034] running on GCE node
I20260809 12:11:01.716009 10457 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260809 12:11:01.716998 10457 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260809 12:11:01.718236 10457 hybrid_clock.cc:648] HybridClock initialized: now 1786277461718179 us; error 49 us; skew 500 ppm
I20260809 12:11:01.718566 10457 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 09 12:11:01 dist-test-slave-2w3w krb5kdc[9807](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786277461, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.222.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260809 12:11:01.721890 10457 init.cc:377] Logged in from keytab as oryx/127.0.222.1@KRBTEST.COM (short username oryx)
I20260809 12:11:01.723533 10457 webserver.cc:460] Webserver started at http://127.0.222.1:46535/ using document root <none> and password file <none>
I20260809 12:11:01.724252 10457 fs_manager.cc:359] Metadata directory not provided
I20260809 12:11:01.724428 10457 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260809 12:11:01.730113 10457 fs_manager.cc:709] Time spent opening directory manager: real 0.004s user 0.005s sys 0.000s
I20260809 12:11:01.733229 10493 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260809 12:11:01.734645 10457 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260809 12:11:01.734843 10457 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskgH5Pjm/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786277382073938-888-0/minicluster-data/ts-0/data,/tmp/dist-test-taskgH5Pjm/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786277382073938-888-0/minicluster-data/ts-0/wal
uuid: "87e053888a014cebb5c16aa941bbf6f7"
format_stamp: "Formatted at 2026-08-09 12:10:58 on dist-test-slave-2w3w"
server_key: "271a6d02bdd4a416d7fb5c619edbda4e"
server_key_iv: "1e2ee96fbbaee301781e1f9a9be365e9"
server_key_version: "encryptionkey@0"
I20260809 12:11:01.735872 10457 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskgH5Pjm/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786277382073938-888-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskgH5Pjm/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786277382073938-888-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskgH5Pjm/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786277382073938-888-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260809 12:11:01.749811 10457 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260809 12:11:01.754216 10457 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260809 12:11:01.754505 10457 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260809 12:11:01.755604 10457 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260809 12:11:01.757112 10457 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260809 12:11:01.757239 10457 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260809 12:11:01.757367 10457 ts_tablet_manager.cc:610] Registered 0 tablets
I20260809 12:11:01.757454 10457 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260809 12:11:01.792991 10457 rpc_server.cc:307] RPC server started. Bound to: 127.0.222.1:34291
I20260809 12:11:01.793087 10615 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.222.1:34291 every 8 connection(s)
I20260809 12:11:01.794531 10457 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskgH5Pjm/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786277382073938-888-0/minicluster-data/ts-0/data/info.pb
I20260809 12:11:01.796156 888 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskgH5Pjm/build/asan/bin/kudu as pid 10457
I20260809 12:11:01.796854 888 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskgH5Pjm/build/asan/bin/kudu
/tmp/dist-test-taskgH5Pjm/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskgH5Pjm/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786277382073938-888-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskgH5Pjm/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786277382073938-888-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskgH5Pjm/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786277382073938-888-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskgH5Pjm/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786277382073938-888-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.222.2:39033
--local_ip_for_outbound_sockets=127.0.222.2
--tserver_master_addrs=127.0.222.62:44607
--webserver_port=39473
--webserver_interface=127.0.222.2
--keytab_file=/tmp/dist-test-taskgH5Pjm/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786277382073938-888-0/krb5kdc/kudu_127.0.222.2.keytab
--principal=kudu/127.0.222.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.222.20:43745
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--principal=oryx/127.0.222.2
--keytab_file=/tmp/dist-test-taskgH5Pjm/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786277382073938-888-0/krb5kdc/oryx_127.0.222.2.keytab with env {KRB5CCNAME=/tmp/dist-test-taskgH5Pjm/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786277382073938-888-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskgH5Pjm/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786277382073938-888-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskgH5Pjm/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786277382073938-888-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Aug 09 12:11:01 dist-test-slave-2w3w krb5kdc[9807](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.222.1@KRBTEST.COM for oryx/127.0.222.62@KRBTEST.COM, Server not found in Kerberos database
Aug 09 12:11:01 dist-test-slave-2w3w krb5kdc[9807](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.222.1@KRBTEST.COM for oryx/127.0.222.62@KRBTEST.COM, Server not found in Kerberos database
I20260809 12:11:01.816116 10619 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 12:11:01.797982 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.222.62:44607 (local address 127.0.222.1:58167)
0809 12:11:01.799068 (+ 1086us) negotiation.cc:107] Waiting for socket to connect
0809 12:11:01.799122 (+ 54us) client_negotiation.cc:174] Beginning negotiation
0809 12:11:01.801205 (+ 2083us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 12:11:01.802846 (+ 1641us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 12:11:01.802854 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 12:11:01.803504 (+ 650us) client_negotiation.cc:189] Negotiated authn=SASL
0809 12:11:01.804066 (+ 562us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 12:11:01.804084 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 12:11:01.806509 (+ 2425us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 12:11:01.806513 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 12:11:01.807955 (+ 1442us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 12:11:01.807969 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 12:11:01.809165 (+ 1196us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 12:11:01.809174 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 12:11:01.809484 (+ 310us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 12:11:01.810233 (+ 749us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0809 12:11:01.810273 (+ 40us) client_negotiation.cc:656] calling sasl_client_start()
0809 12:11:01.814616 (+ 4343us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.222.62:44607: Server oryx/127.0.222.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":517,"thread_start_us":515,"threads_started":1}
W20260809 12:11:01.816517 10620 negotiation.cc:337] Failed RPC negotiation. Trace:
0809 12:11:01.799259 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.222.1:58167 (local address 127.0.222.62:44607)
0809 12:11:01.800129 (+ 870us) server_negotiation.cc:206] Beginning negotiation
0809 12:11:01.800146 (+ 17us) server_negotiation.cc:399] Waiting for connection header
0809 12:11:01.800200 (+ 54us) server_negotiation.cc:407] Connection header received
0809 12:11:01.801819 (+ 1619us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 12:11:01.801840 (+ 21us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 12:11:01.802271 (+ 431us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 12:11:01.802668 (+ 397us) server_negotiation.cc:226] Negotiated authn=SASL
0809 12:11:01.804279 (+ 1611us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 12:11:01.806317 (+ 2038us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 12:11:01.808139 (+ 1822us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 12:11:01.808897 (+ 758us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 12:11:01.809129 (+ 232us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 12:11:01.815186 (+ 6057us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.222.1:58167: BlockingRecv error: recv got EOF from 127.0.222.1:58167 (error 108)
Metrics: {"server-negotiator.queue_time_us":569,"thread_start_us":298,"threads_started":1}
W20260809 12:11:01.817215 10617 heartbeater.cc:644] Failed to heartbeat to 127.0.222.62:44607 (0 consecutive failures): Not authorized: Failed to ping master at 127.0.222.62:44607: Client connection negotiation failed: client connection to 127.0.222.62:44607: Server oryx/127.0.222.62@KRBTEST.COM not found in Kerberos database
Aug 09 12:11:01 dist-test-slave-2w3w krb5kdc[9807](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.222.1@KRBTEST.COM for oryx/127.0.222.62@KRBTEST.COM, Server not found in Kerberos database
Aug 09 12:11:01 dist-test-slave-2w3w krb5kdc[9807](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.222.1@KRBTEST.COM for oryx/127.0.222.62@KRBTEST.COM, Server not found in Kerberos database
I20260809 12:11:01.827205 10619 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 12:11:01.817923 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.222.62:44607 (local address 127.0.222.1:52891)
0809 12:11:01.818143 (+ 220us) negotiation.cc:107] Waiting for socket to connect
0809 12:11:01.818165 (+ 22us) client_negotiation.cc:174] Beginning negotiation
0809 12:11:01.818292 (+ 127us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 12:11:01.818756 (+ 464us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 12:11:01.818759 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 12:11:01.819182 (+ 423us) client_negotiation.cc:189] Negotiated authn=SASL
0809 12:11:01.819324 (+ 142us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 12:11:01.819333 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 12:11:01.821013 (+ 1680us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 12:11:01.821017 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 12:11:01.822023 (+ 1006us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 12:11:01.822036 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 12:11:01.823144 (+ 1108us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 12:11:01.823149 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 12:11:01.823291 (+ 142us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 12:11:01.823952 (+ 661us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0809 12:11:01.823969 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0809 12:11:01.826950 (+ 2981us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.222.62:44607: Server oryx/127.0.222.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":68}
W20260809 12:11:01.827734 10620 negotiation.cc:337] Failed RPC negotiation. Trace:
0809 12:11:01.818208 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.222.1:52891 (local address 127.0.222.62:44607)
0809 12:11:01.818423 (+ 215us) server_negotiation.cc:206] Beginning negotiation
0809 12:11:01.818432 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0809 12:11:01.818453 (+ 21us) server_negotiation.cc:407] Connection header received
0809 12:11:01.818524 (+ 71us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 12:11:01.818532 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 12:11:01.818616 (+ 84us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 12:11:01.818725 (+ 109us) server_negotiation.cc:226] Negotiated authn=SASL
0809 12:11:01.819473 (+ 748us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 12:11:01.820831 (+ 1358us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 12:11:01.822188 (+ 1357us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 12:11:01.822990 (+ 802us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 12:11:01.823145 (+ 155us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 12:11:01.827273 (+ 4128us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.222.1:52891: BlockingRecv error: recv got EOF from 127.0.222.1:52891 (error 108)
Metrics: {"server-negotiator.queue_time_us":69}
Aug 09 12:11:01 dist-test-slave-2w3w krb5kdc[9807](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.222.1@KRBTEST.COM for oryx/127.0.222.62@KRBTEST.COM, Server not found in Kerberos database
Aug 09 12:11:01 dist-test-slave-2w3w krb5kdc[9807](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.222.1@KRBTEST.COM for oryx/127.0.222.62@KRBTEST.COM, Server not found in Kerberos database
I20260809 12:11:01.837118 10619 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 12:11:01.827976 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.222.62:44607 (local address 127.0.222.1:43051)
0809 12:11:01.828142 (+ 166us) negotiation.cc:107] Waiting for socket to connect
0809 12:11:01.828162 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0809 12:11:01.828258 (+ 96us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 12:11:01.828646 (+ 388us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 12:11:01.828651 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 12:11:01.829027 (+ 376us) client_negotiation.cc:189] Negotiated authn=SASL
0809 12:11:01.829211 (+ 184us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 12:11:01.829221 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 12:11:01.830307 (+ 1086us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 12:11:01.830313 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 12:11:01.831844 (+ 1531us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 12:11:01.831859 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 12:11:01.832531 (+ 672us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 12:11:01.832537 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 12:11:01.832751 (+ 214us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 12:11:01.833404 (+ 653us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0809 12:11:01.833426 (+ 22us) client_negotiation.cc:656] calling sasl_client_start()
0809 12:11:01.836819 (+ 3393us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.222.62:44607: Server oryx/127.0.222.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":45}
W20260809 12:11:01.837281 10620 negotiation.cc:337] Failed RPC negotiation. Trace:
0809 12:11:01.828204 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.222.1:43051 (local address 127.0.222.62:44607)
0809 12:11:01.828370 (+ 166us) server_negotiation.cc:206] Beginning negotiation
0809 12:11:01.828374 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0809 12:11:01.828388 (+ 14us) server_negotiation.cc:407] Connection header received
0809 12:11:01.828428 (+ 40us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 12:11:01.828432 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 12:11:01.828501 (+ 69us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 12:11:01.828586 (+ 85us) server_negotiation.cc:226] Negotiated authn=SASL
0809 12:11:01.829356 (+ 770us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 12:11:01.830173 (+ 817us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 12:11:01.831983 (+ 1810us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 12:11:01.832380 (+ 397us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 12:11:01.832480 (+ 100us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 12:11:01.837039 (+ 4559us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.222.1:43051: BlockingRecv error: recv got EOF from 127.0.222.1:43051 (error 108)
Metrics: {"server-negotiator.queue_time_us":57}
W20260809 12:11:01.837579 10617 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
W20260809 12:11:02.143435 10621 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260809 12:11:02.143779 10621 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260809 12:11:02.143952 10621 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260809 12:11:02.151445 10621 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260809 12:11:02.151607 10621 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260809 12:11:02.151796 10621 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.222.2
I20260809 12:11:02.161444 10621 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.222.20:43745
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskgH5Pjm/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786277382073938-888-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskgH5Pjm/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786277382073938-888-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskgH5Pjm/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786277382073938-888-0/krb5kdc/oryx_127.0.222.2.keytab
--principal=oryx/127.0.222.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.222.2:39033
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskgH5Pjm/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786277382073938-888-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.222.2
--webserver_port=39473
--tserver_master_addrs=127.0.222.62:44607
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.222.2
--log_dir=/tmp/dist-test-taskgH5Pjm/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786277382073938-888-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 09 Aug 2026 11:06:41 UTC on ccycloud.quasar-bwoogy.root.comops.site
build id 4575991
ASAN enabled
I20260809 12:11:02.162839 10621 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260809 12:11:02.164183 10621 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260809 12:11:02.177644 10628 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 12:11:02.177695 10632 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 12:11:02.177610 10629 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260809 12:11:02.178534 10621 server_base.cc:1034] running on GCE node
I20260809 12:11:02.179204 10621 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260809 12:11:02.180186 10621 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260809 12:11:02.181437 10621 hybrid_clock.cc:648] HybridClock initialized: now 1786277462181386 us; error 66 us; skew 500 ppm
I20260809 12:11:02.181785 10621 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 09 12:11:02 dist-test-slave-2w3w krb5kdc[9807](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786277462, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.222.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260809 12:11:02.185159 10621 init.cc:377] Logged in from keytab as oryx/127.0.222.2@KRBTEST.COM (short username oryx)
I20260809 12:11:02.186801 10621 webserver.cc:460] Webserver started at http://127.0.222.2:39473/ using document root <none> and password file <none>
I20260809 12:11:02.187598 10621 fs_manager.cc:359] Metadata directory not provided
I20260809 12:11:02.187768 10621 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260809 12:11:02.193545 10621 fs_manager.cc:709] Time spent opening directory manager: real 0.004s user 0.005s sys 0.000s
I20260809 12:11:02.196535 10639 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260809 12:11:02.197971 10621 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.000s sys 0.003s
I20260809 12:11:02.198184 10621 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskgH5Pjm/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786277382073938-888-0/minicluster-data/ts-1/data,/tmp/dist-test-taskgH5Pjm/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786277382073938-888-0/minicluster-data/ts-1/wal
uuid: "03830909e0584c7e90924ae9749b9849"
format_stamp: "Formatted at 2026-08-09 12:10:59 on dist-test-slave-2w3w"
server_key: "ab11351ef1ac9b2fb3afae599651734d"
server_key_iv: "88166c92df7d187ae91aca69147a8b17"
server_key_version: "encryptionkey@0"
I20260809 12:11:02.199290 10621 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskgH5Pjm/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786277382073938-888-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskgH5Pjm/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786277382073938-888-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskgH5Pjm/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786277382073938-888-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260809 12:11:02.224509 10621 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260809 12:11:02.228888 10621 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260809 12:11:02.229195 10621 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260809 12:11:02.230297 10621 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260809 12:11:02.231766 10621 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260809 12:11:02.231894 10621 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260809 12:11:02.232029 10621 ts_tablet_manager.cc:610] Registered 0 tablets
I20260809 12:11:02.232128 10621 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260809 12:11:02.269510 10621 rpc_server.cc:307] RPC server started. Bound to: 127.0.222.2:39033
I20260809 12:11:02.269608 10765 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.222.2:39033 every 8 connection(s)
I20260809 12:11:02.271147 10621 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskgH5Pjm/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786277382073938-888-0/minicluster-data/ts-1/data/info.pb
I20260809 12:11:02.278581 888 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskgH5Pjm/build/asan/bin/kudu as pid 10621
I20260809 12:11:02.279423 888 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskgH5Pjm/build/asan/bin/kudu
/tmp/dist-test-taskgH5Pjm/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskgH5Pjm/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786277382073938-888-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskgH5Pjm/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786277382073938-888-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskgH5Pjm/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786277382073938-888-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskgH5Pjm/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786277382073938-888-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.222.3:35897
--local_ip_for_outbound_sockets=127.0.222.3
--tserver_master_addrs=127.0.222.62:44607
--webserver_port=35869
--webserver_interface=127.0.222.3
--keytab_file=/tmp/dist-test-taskgH5Pjm/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786277382073938-888-0/krb5kdc/kudu_127.0.222.3.keytab
--principal=kudu/127.0.222.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.222.20:43745
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--principal=oryx/127.0.222.3
--keytab_file=/tmp/dist-test-taskgH5Pjm/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786277382073938-888-0/krb5kdc/oryx_127.0.222.3.keytab with env {KRB5CCNAME=/tmp/dist-test-taskgH5Pjm/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786277382073938-888-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskgH5Pjm/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786277382073938-888-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskgH5Pjm/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786277382073938-888-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Aug 09 12:11:02 dist-test-slave-2w3w krb5kdc[9807](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.222.2@KRBTEST.COM for oryx/127.0.222.62@KRBTEST.COM, Server not found in Kerberos database
Aug 09 12:11:02 dist-test-slave-2w3w krb5kdc[9807](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.222.2@KRBTEST.COM for oryx/127.0.222.62@KRBTEST.COM, Server not found in Kerberos database
I20260809 12:11:02.291819 10768 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 12:11:02.274466 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.222.62:44607 (local address 127.0.222.2:53467)
0809 12:11:02.275237 (+ 771us) negotiation.cc:107] Waiting for socket to connect
0809 12:11:02.275283 (+ 46us) client_negotiation.cc:174] Beginning negotiation
0809 12:11:02.277018 (+ 1735us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 12:11:02.277745 (+ 727us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 12:11:02.277753 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 12:11:02.278397 (+ 644us) client_negotiation.cc:189] Negotiated authn=SASL
0809 12:11:02.279219 (+ 822us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 12:11:02.279246 (+ 27us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 12:11:02.280549 (+ 1303us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 12:11:02.280558 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 12:11:02.282738 (+ 2180us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 12:11:02.282756 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 12:11:02.283981 (+ 1225us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 12:11:02.283990 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 12:11:02.284337 (+ 347us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 12:11:02.285161 (+ 824us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0809 12:11:02.285195 (+ 34us) client_negotiation.cc:656] calling sasl_client_start()
0809 12:11:02.290222 (+ 5027us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.222.62:44607: Server oryx/127.0.222.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":527,"thread_start_us":306,"threads_started":1}
W20260809 12:11:02.291976 10620 negotiation.cc:337] Failed RPC negotiation. Trace:
0809 12:11:02.273973 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.222.2:53467 (local address 127.0.222.62:44607)
0809 12:11:02.274156 (+ 183us) server_negotiation.cc:206] Beginning negotiation
0809 12:11:02.274161 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0809 12:11:02.275429 (+ 1268us) server_negotiation.cc:407] Connection header received
0809 12:11:02.277296 (+ 1867us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 12:11:02.277302 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 12:11:02.277394 (+ 92us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 12:11:02.277510 (+ 116us) server_negotiation.cc:226] Negotiated authn=SASL
0809 12:11:02.279449 (+ 1939us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 12:11:02.280377 (+ 928us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 12:11:02.282914 (+ 2537us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 12:11:02.283714 (+ 800us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 12:11:02.283863 (+ 149us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 12:11:02.291667 (+ 7804us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.222.2:53467: BlockingRecv error: recv got EOF from 127.0.222.2:53467 (error 108)
Metrics: {"server-negotiator.queue_time_us":60}
W20260809 12:11:02.292696 10766 heartbeater.cc:644] Failed to heartbeat to 127.0.222.62:44607 (0 consecutive failures): Not authorized: Failed to ping master at 127.0.222.62:44607: Client connection negotiation failed: client connection to 127.0.222.62:44607: Server oryx/127.0.222.62@KRBTEST.COM not found in Kerberos database
Aug 09 12:11:02 dist-test-slave-2w3w krb5kdc[9807](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.222.2@KRBTEST.COM for oryx/127.0.222.62@KRBTEST.COM, Server not found in Kerberos database
Aug 09 12:11:02 dist-test-slave-2w3w krb5kdc[9807](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.222.2@KRBTEST.COM for oryx/127.0.222.62@KRBTEST.COM, Server not found in Kerberos database
I20260809 12:11:02.304189 10768 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 12:11:02.293417 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.222.62:44607 (local address 127.0.222.2:43281)
0809 12:11:02.293611 (+ 194us) negotiation.cc:107] Waiting for socket to connect
0809 12:11:02.293628 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0809 12:11:02.293712 (+ 84us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 12:11:02.294098 (+ 386us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 12:11:02.294102 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 12:11:02.294428 (+ 326us) client_negotiation.cc:189] Negotiated authn=SASL
0809 12:11:02.294624 (+ 196us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 12:11:02.294635 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 12:11:02.296401 (+ 1766us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 12:11:02.296407 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 12:11:02.297827 (+ 1420us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 12:11:02.297842 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 12:11:02.299134 (+ 1292us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 12:11:02.299141 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 12:11:02.299346 (+ 205us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 12:11:02.300046 (+ 700us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0809 12:11:02.300070 (+ 24us) client_negotiation.cc:656] calling sasl_client_start()
0809 12:11:02.303874 (+ 3804us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.222.62:44607: Server oryx/127.0.222.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":64}
W20260809 12:11:02.304432 10620 negotiation.cc:337] Failed RPC negotiation. Trace:
0809 12:11:02.293537 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.222.2:43281 (local address 127.0.222.62:44607)
0809 12:11:02.293766 (+ 229us) server_negotiation.cc:206] Beginning negotiation
0809 12:11:02.293774 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0809 12:11:02.293797 (+ 23us) server_negotiation.cc:407] Connection header received
0809 12:11:02.293860 (+ 63us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 12:11:02.293868 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 12:11:02.293956 (+ 88us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 12:11:02.294073 (+ 117us) server_negotiation.cc:226] Negotiated authn=SASL
0809 12:11:02.294830 (+ 757us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 12:11:02.296223 (+ 1393us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 12:11:02.298001 (+ 1778us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 12:11:02.298902 (+ 901us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 12:11:02.299146 (+ 244us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 12:11:02.304084 (+ 4938us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.222.2:43281: BlockingRecv error: recv got EOF from 127.0.222.2:43281 (error 108)
Metrics: {"server-negotiator.queue_time_us":68}
Aug 09 12:11:02 dist-test-slave-2w3w krb5kdc[9807](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.222.2@KRBTEST.COM for oryx/127.0.222.62@KRBTEST.COM, Server not found in Kerberos database
Aug 09 12:11:02 dist-test-slave-2w3w krb5kdc[9807](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.222.2@KRBTEST.COM for oryx/127.0.222.62@KRBTEST.COM, Server not found in Kerberos database
I20260809 12:11:02.314036 10768 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 12:11:02.305090 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.222.62:44607 (local address 127.0.222.2:41403)
0809 12:11:02.305471 (+ 381us) negotiation.cc:107] Waiting for socket to connect
0809 12:11:02.305495 (+ 24us) client_negotiation.cc:174] Beginning negotiation
0809 12:11:02.305619 (+ 124us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 12:11:02.306004 (+ 385us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 12:11:02.306012 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 12:11:02.306501 (+ 489us) client_negotiation.cc:189] Negotiated authn=SASL
0809 12:11:02.306777 (+ 276us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 12:11:02.306795 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 12:11:02.308105 (+ 1310us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 12:11:02.308110 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 12:11:02.309036 (+ 926us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 12:11:02.309047 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 12:11:02.309753 (+ 706us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 12:11:02.309759 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 12:11:02.309939 (+ 180us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 12:11:02.310589 (+ 650us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0809 12:11:02.310605 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0809 12:11:02.313713 (+ 3108us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.222.62:44607: Server oryx/127.0.222.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":218}
W20260809 12:11:02.314265 10620 negotiation.cc:337] Failed RPC negotiation. Trace:
0809 12:11:02.305520 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.222.2:41403 (local address 127.0.222.62:44607)
0809 12:11:02.305675 (+ 155us) server_negotiation.cc:206] Beginning negotiation
0809 12:11:02.305681 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0809 12:11:02.305698 (+ 17us) server_negotiation.cc:407] Connection header received
0809 12:11:02.305760 (+ 62us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 12:11:02.305766 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 12:11:02.305834 (+ 68us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 12:11:02.305929 (+ 95us) server_negotiation.cc:226] Negotiated authn=SASL
0809 12:11:02.306950 (+ 1021us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 12:11:02.307957 (+ 1007us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 12:11:02.309183 (+ 1226us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 12:11:02.309594 (+ 411us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 12:11:02.309729 (+ 135us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 12:11:02.313926 (+ 4197us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.222.2:41403: BlockingRecv error: recv got EOF from 127.0.222.2:41403 (error 108)
Metrics: {"server-negotiator.queue_time_us":57}
W20260809 12:11:02.314455 10766 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
W20260809 12:11:02.635807 10769 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260809 12:11:02.636086 10769 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260809 12:11:02.636181 10769 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260809 12:11:02.643790 10769 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260809 12:11:02.643913 10769 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260809 12:11:02.644029 10769 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.222.3
I20260809 12:11:02.653975 10769 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.222.20:43745
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskgH5Pjm/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786277382073938-888-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskgH5Pjm/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786277382073938-888-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskgH5Pjm/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786277382073938-888-0/krb5kdc/oryx_127.0.222.3.keytab
--principal=oryx/127.0.222.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.222.3:35897
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskgH5Pjm/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786277382073938-888-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.222.3
--webserver_port=35869
--tserver_master_addrs=127.0.222.62:44607
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.222.3
--log_dir=/tmp/dist-test-taskgH5Pjm/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786277382073938-888-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 09 Aug 2026 11:06:41 UTC on ccycloud.quasar-bwoogy.root.comops.site
build id 4575991
ASAN enabled
I20260809 12:11:02.655300 10769 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260809 12:11:02.656580 10769 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260809 12:11:02.669749 10775 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 12:11:02.669697 10776 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 12:11:02.669651 10778 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260809 12:11:02.671000 10769 server_base.cc:1034] running on GCE node
I20260809 12:11:02.671624 10769 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260809 12:11:02.672624 10769 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260809 12:11:02.673862 10769 hybrid_clock.cc:648] HybridClock initialized: now 1786277462673800 us; error 50 us; skew 500 ppm
I20260809 12:11:02.674187 10769 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 09 12:11:02 dist-test-slave-2w3w krb5kdc[9807](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786277462, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.222.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260809 12:11:02.677399 10769 init.cc:377] Logged in from keytab as oryx/127.0.222.3@KRBTEST.COM (short username oryx)
I20260809 12:11:02.678853 10769 webserver.cc:460] Webserver started at http://127.0.222.3:35869/ using document root <none> and password file <none>
I20260809 12:11:02.679582 10769 fs_manager.cc:359] Metadata directory not provided
I20260809 12:11:02.679745 10769 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260809 12:11:02.685032 10769 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.005s sys 0.000s
I20260809 12:11:02.687747 10786 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260809 12:11:02.689034 10769 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260809 12:11:02.689227 10769 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskgH5Pjm/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786277382073938-888-0/minicluster-data/ts-2/data,/tmp/dist-test-taskgH5Pjm/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786277382073938-888-0/minicluster-data/ts-2/wal
uuid: "6bf090a102124104ac7909e02e744847"
format_stamp: "Formatted at 2026-08-09 12:11:00 on dist-test-slave-2w3w"
server_key: "e4aa8907f8c8ade76fc1194c4d20bad1"
server_key_iv: "b762426b3ca726a8e6cfedbf872e0b8e"
server_key_version: "encryptionkey@0"
I20260809 12:11:02.690179 10769 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskgH5Pjm/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786277382073938-888-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskgH5Pjm/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786277382073938-888-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskgH5Pjm/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786277382073938-888-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260809 12:11:02.706432 10769 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260809 12:11:02.710909 10769 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260809 12:11:02.711345 10769 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260809 12:11:02.712466 10769 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260809 12:11:02.713996 10769 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260809 12:11:02.714092 10769 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260809 12:11:02.714262 10769 ts_tablet_manager.cc:610] Registered 0 tablets
I20260809 12:11:02.714385 10769 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260809 12:11:02.750372 10769 rpc_server.cc:307] RPC server started. Bound to: 127.0.222.3:35897
I20260809 12:11:02.750435 10908 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.222.3:35897 every 8 connection(s)
I20260809 12:11:02.751995 10769 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskgH5Pjm/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786277382073938-888-0/minicluster-data/ts-2/data/info.pb
I20260809 12:11:02.753074 888 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskgH5Pjm/build/asan/bin/kudu as pid 10769
Aug 09 12:11:02 dist-test-slave-2w3w krb5kdc[9807](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.222.3@KRBTEST.COM for oryx/127.0.222.62@KRBTEST.COM, Server not found in Kerberos database
Aug 09 12:11:02 dist-test-slave-2w3w krb5kdc[9807](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.222.3@KRBTEST.COM for oryx/127.0.222.62@KRBTEST.COM, Server not found in Kerberos database
W20260809 12:11:02.768697 10914 negotiation.cc:337] Failed RPC negotiation. Trace:
0809 12:11:02.754843 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.222.3:47779 (local address 127.0.222.62:44607)
0809 12:11:02.757057 (+ 2214us) server_negotiation.cc:206] Beginning negotiation
0809 12:11:02.757063 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0809 12:11:02.757083 (+ 20us) server_negotiation.cc:407] Connection header received
0809 12:11:02.758439 (+ 1356us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 12:11:02.758443 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 12:11:02.758532 (+ 89us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 12:11:02.758649 (+ 117us) server_negotiation.cc:226] Negotiated authn=SASL
0809 12:11:02.760190 (+ 1541us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 12:11:02.761046 (+ 856us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 12:11:02.762790 (+ 1744us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 12:11:02.763257 (+ 467us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 12:11:02.763394 (+ 137us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 12:11:02.768387 (+ 4993us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.222.3:47779: BlockingRecv error: recv got EOF from 127.0.222.3:47779 (error 108)
Metrics: {"mutex_wait_us":1,"server-negotiator.queue_time_us":2082,"thread_start_us":1938,"threads_started":1}
I20260809 12:11:02.769320 10915 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 12:11:02.755462 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.222.62:44607 (local address 127.0.222.3:47779)
0809 12:11:02.756288 (+ 826us) negotiation.cc:107] Waiting for socket to connect
0809 12:11:02.756345 (+ 57us) client_negotiation.cc:174] Beginning negotiation
0809 12:11:02.758182 (+ 1837us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 12:11:02.758908 (+ 726us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 12:11:02.758916 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 12:11:02.759458 (+ 542us) client_negotiation.cc:189] Negotiated authn=SASL
0809 12:11:02.760005 (+ 547us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 12:11:02.760022 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 12:11:02.761207 (+ 1185us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 12:11:02.761212 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 12:11:02.762647 (+ 1435us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 12:11:02.762660 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 12:11:02.763500 (+ 840us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 12:11:02.763510 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 12:11:02.763737 (+ 227us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 12:11:02.764402 (+ 665us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0809 12:11:02.764422 (+ 20us) client_negotiation.cc:656] calling sasl_client_start()
0809 12:11:02.768133 (+ 3711us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.222.62:44607: Server oryx/127.0.222.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":558,"thread_start_us":395,"threads_started":1}
W20260809 12:11:02.770063 10909 heartbeater.cc:644] Failed to heartbeat to 127.0.222.62:44607 (0 consecutive failures): Not authorized: Failed to ping master at 127.0.222.62:44607: Client connection negotiation failed: client connection to 127.0.222.62:44607: Server oryx/127.0.222.62@KRBTEST.COM not found in Kerberos database
I20260809 12:11:02.772213 10620 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 12:11:02.754128 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54032 (local address 127.0.222.62:44607)
0809 12:11:02.754331 (+ 203us) server_negotiation.cc:206] Beginning negotiation
0809 12:11:02.754336 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0809 12:11:02.756178 (+ 1842us) server_negotiation.cc:407] Connection header received
0809 12:11:02.756603 (+ 425us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 12:11:02.756614 (+ 11us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 12:11:02.756778 (+ 164us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 12:11:02.756869 (+ 91us) server_negotiation.cc:226] Negotiated authn=SASL
0809 12:11:02.758753 (+ 1884us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 12:11:02.759668 (+ 915us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 12:11:02.761180 (+ 1512us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 12:11:02.761668 (+ 488us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 12:11:02.761826 (+ 158us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 12:11:02.765021 (+ 3195us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 12:11:02.765029 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 12:11:02.765044 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0809 12:11:02.765081 (+ 37us) server_negotiation.cc:922] calling sasl_server_start()
0809 12:11:02.768383 (+ 3302us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 12:11:02.769279 (+ 896us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 12:11:02.769292 (+ 13us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 12:11:02.769299 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0809 12:11:02.769533 (+ 234us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 12:11:02.769944 (+ 411us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 12:11:02.769950 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 12:11:02.769954 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0809 12:11:02.770477 (+ 523us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 12:11:02.770873 (+ 396us) server_negotiation.cc:1035] Waiting for connection context
0809 12:11:02.771341 (+ 468us) server_negotiation.cc:299] Negotiation successful
0809 12:11:02.771762 (+ 421us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":24,"server-negotiator.queue_time_us":67}
Aug 09 12:11:02 dist-test-slave-2w3w krb5kdc[9807](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.222.3@KRBTEST.COM for oryx/127.0.222.62@KRBTEST.COM, Server not found in Kerberos database
Aug 09 12:11:02 dist-test-slave-2w3w krb5kdc[9807](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.222.3@KRBTEST.COM for oryx/127.0.222.62@KRBTEST.COM, Server not found in Kerberos database
I20260809 12:11:02.779500 10915 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 12:11:02.770653 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.222.62:44607 (local address 127.0.222.3:60947)
0809 12:11:02.770893 (+ 240us) negotiation.cc:107] Waiting for socket to connect
0809 12:11:02.770913 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0809 12:11:02.771048 (+ 135us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 12:11:02.771432 (+ 384us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 12:11:02.771437 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 12:11:02.771785 (+ 348us) client_negotiation.cc:189] Negotiated authn=SASL
0809 12:11:02.772027 (+ 242us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 12:11:02.772036 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 12:11:02.773723 (+ 1687us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 12:11:02.773728 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 12:11:02.775040 (+ 1312us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 12:11:02.775057 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 12:11:02.775887 (+ 830us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 12:11:02.775893 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 12:11:02.776076 (+ 183us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 12:11:02.776688 (+ 612us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0809 12:11:02.776704 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0809 12:11:02.779311 (+ 2607us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.222.62:44607: Server oryx/127.0.222.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":133}
W20260809 12:11:02.779799 10914 negotiation.cc:337] Failed RPC negotiation. Trace:
0809 12:11:02.770779 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.222.3:60947 (local address 127.0.222.62:44607)
0809 12:11:02.771009 (+ 230us) server_negotiation.cc:206] Beginning negotiation
0809 12:11:02.771018 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0809 12:11:02.771038 (+ 20us) server_negotiation.cc:407] Connection header received
0809 12:11:02.771177 (+ 139us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 12:11:02.771182 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 12:11:02.771259 (+ 77us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 12:11:02.771381 (+ 122us) server_negotiation.cc:226] Negotiated authn=SASL
0809 12:11:02.772200 (+ 819us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 12:11:02.773543 (+ 1343us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 12:11:02.775182 (+ 1639us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 12:11:02.775737 (+ 555us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 12:11:02.775839 (+ 102us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 12:11:02.779514 (+ 3675us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.222.3:60947: BlockingRecv error: recv got EOF from 127.0.222.3:60947 (error 108)
Metrics: {"server-negotiator.queue_time_us":65}
Aug 09 12:11:02 dist-test-slave-2w3w krb5kdc[9807](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.222.3@KRBTEST.COM for oryx/127.0.222.62@KRBTEST.COM, Server not found in Kerberos database
Aug 09 12:11:02 dist-test-slave-2w3w krb5kdc[9807](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.222.3@KRBTEST.COM for oryx/127.0.222.62@KRBTEST.COM, Server not found in Kerberos database
I20260809 12:11:02.787751 10915 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 12:11:02.780272 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.222.62:44607 (local address 127.0.222.3:54933)
0809 12:11:02.780471 (+ 199us) negotiation.cc:107] Waiting for socket to connect
0809 12:11:02.780491 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0809 12:11:02.780582 (+ 91us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 12:11:02.780901 (+ 319us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 12:11:02.780904 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 12:11:02.781169 (+ 265us) client_negotiation.cc:189] Negotiated authn=SASL
0809 12:11:02.781303 (+ 134us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 12:11:02.781311 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 12:11:02.782306 (+ 995us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 12:11:02.782310 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 12:11:02.783254 (+ 944us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 12:11:02.783266 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 12:11:02.783991 (+ 725us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 12:11:02.783995 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 12:11:02.784147 (+ 152us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 12:11:02.784645 (+ 498us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0809 12:11:02.784661 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0809 12:11:02.787454 (+ 2793us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.222.62:44607: Server oryx/127.0.222.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":61}
W20260809 12:11:02.787906 10914 negotiation.cc:337] Failed RPC negotiation. Trace:
0809 12:11:02.780373 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.222.3:54933 (local address 127.0.222.62:44607)
0809 12:11:02.780605 (+ 232us) server_negotiation.cc:206] Beginning negotiation
0809 12:11:02.780609 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0809 12:11:02.780624 (+ 15us) server_negotiation.cc:407] Connection header received
0809 12:11:02.780697 (+ 73us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 12:11:02.780702 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 12:11:02.780768 (+ 66us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 12:11:02.780866 (+ 98us) server_negotiation.cc:226] Negotiated authn=SASL
0809 12:11:02.781425 (+ 559us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 12:11:02.782178 (+ 753us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 12:11:02.783393 (+ 1215us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 12:11:02.783861 (+ 468us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 12:11:02.783962 (+ 101us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 12:11:02.787639 (+ 3677us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.222.3:54933: BlockingRecv error: recv got EOF from 127.0.222.3:54933 (error 108)
Metrics: {"server-negotiator.queue_time_us":130}
W20260809 12:11:02.788234 10909 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
Aug 09 12:11:02 dist-test-slave-2w3w krb5kdc[9807](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.222.1@KRBTEST.COM for oryx/127.0.222.62@KRBTEST.COM, Server not found in Kerberos database
Aug 09 12:11:02 dist-test-slave-2w3w krb5kdc[9807](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.222.1@KRBTEST.COM for oryx/127.0.222.62@KRBTEST.COM, Server not found in Kerberos database
I20260809 12:11:02.849272 10916 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 12:11:02.838316 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.222.62:44607 (local address 127.0.222.1:56257)
0809 12:11:02.839002 (+ 686us) negotiation.cc:107] Waiting for socket to connect
0809 12:11:02.839028 (+ 26us) client_negotiation.cc:174] Beginning negotiation
0809 12:11:02.839212 (+ 184us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 12:11:02.839636 (+ 424us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 12:11:02.839643 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 12:11:02.840329 (+ 686us) client_negotiation.cc:189] Negotiated authn=SASL
0809 12:11:02.840620 (+ 291us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 12:11:02.840634 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 12:11:02.842244 (+ 1610us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 12:11:02.842270 (+ 26us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 12:11:02.843799 (+ 1529us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 12:11:02.843825 (+ 26us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 12:11:02.844843 (+ 1018us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 12:11:02.844849 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 12:11:02.845039 (+ 190us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 12:11:02.845725 (+ 686us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0809 12:11:02.845743 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0809 12:11:02.848954 (+ 3211us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.222.62:44607: Server oryx/127.0.222.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":471,"thread_start_us":299,"threads_started":1}
W20260809 12:11:02.849700 10914 negotiation.cc:337] Failed RPC negotiation. Trace:
0809 12:11:02.838785 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.222.1:56257 (local address 127.0.222.62:44607)
0809 12:11:02.839006 (+ 221us) server_negotiation.cc:206] Beginning negotiation
0809 12:11:02.839012 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0809 12:11:02.839127 (+ 115us) server_negotiation.cc:407] Connection header received
0809 12:11:02.839374 (+ 247us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 12:11:02.839379 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 12:11:02.839479 (+ 100us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 12:11:02.839587 (+ 108us) server_negotiation.cc:226] Negotiated authn=SASL
0809 12:11:02.840777 (+ 1190us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 12:11:02.842053 (+ 1276us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 12:11:02.844002 (+ 1949us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 12:11:02.844611 (+ 609us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 12:11:02.844767 (+ 156us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 12:11:02.849314 (+ 4547us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.222.1:56257: BlockingRecv error: recv got EOF from 127.0.222.1:56257 (error 108)
Metrics: {"server-negotiator.queue_time_us":79}
Aug 09 12:11:03 dist-test-slave-2w3w krb5kdc[9807](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.222.2@KRBTEST.COM for oryx/127.0.222.62@KRBTEST.COM, Server not found in Kerberos database
Aug 09 12:11:03 dist-test-slave-2w3w krb5kdc[9807](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.222.2@KRBTEST.COM for oryx/127.0.222.62@KRBTEST.COM, Server not found in Kerberos database
I20260809 12:11:03.323916 10917 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 12:11:03.315122 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.222.62:44607 (local address 127.0.222.2:34833)
0809 12:11:03.315648 (+ 526us) negotiation.cc:107] Waiting for socket to connect
0809 12:11:03.315664 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0809 12:11:03.315825 (+ 161us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 12:11:03.316174 (+ 349us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 12:11:03.316178 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 12:11:03.316678 (+ 500us) client_negotiation.cc:189] Negotiated authn=SASL
0809 12:11:03.316872 (+ 194us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 12:11:03.316881 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 12:11:03.317960 (+ 1079us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 12:11:03.317966 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 12:11:03.318949 (+ 983us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 12:11:03.318993 (+ 44us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 12:11:03.319758 (+ 765us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 12:11:03.319766 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 12:11:03.319941 (+ 175us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 12:11:03.320526 (+ 585us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0809 12:11:03.320544 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0809 12:11:03.323562 (+ 3018us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.222.62:44607: Server oryx/127.0.222.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":422,"thread_start_us":277,"threads_started":1}
W20260809 12:11:03.324106 10914 negotiation.cc:337] Failed RPC negotiation. Trace:
0809 12:11:03.315171 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.222.2:34833 (local address 127.0.222.62:44607)
0809 12:11:03.315397 (+ 226us) server_negotiation.cc:206] Beginning negotiation
0809 12:11:03.315406 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0809 12:11:03.315762 (+ 356us) server_negotiation.cc:407] Connection header received
0809 12:11:03.315959 (+ 197us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 12:11:03.315963 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 12:11:03.316036 (+ 73us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 12:11:03.316144 (+ 108us) server_negotiation.cc:226] Negotiated authn=SASL
0809 12:11:03.317034 (+ 890us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 12:11:03.317809 (+ 775us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 12:11:03.319130 (+ 1321us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 12:11:03.319571 (+ 441us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 12:11:03.319721 (+ 150us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 12:11:03.323808 (+ 4087us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.222.2:34833: BlockingRecv error: recv got EOF from 127.0.222.2:34833 (error 108)
Metrics: {"server-negotiator.queue_time_us":60}
Aug 09 12:11:03 dist-test-slave-2w3w krb5kdc[9807](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.222.3@KRBTEST.COM for oryx/127.0.222.62@KRBTEST.COM, Server not found in Kerberos database
Aug 09 12:11:03 dist-test-slave-2w3w krb5kdc[9807](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.222.3@KRBTEST.COM for oryx/127.0.222.62@KRBTEST.COM, Server not found in Kerberos database
I20260809 12:11:03.798120 10918 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 12:11:03.788884 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.222.62:44607 (local address 127.0.222.3:38363)
0809 12:11:03.789486 (+ 602us) negotiation.cc:107] Waiting for socket to connect
0809 12:11:03.789505 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0809 12:11:03.789683 (+ 178us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 12:11:03.790059 (+ 376us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 12:11:03.790063 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 12:11:03.790568 (+ 505us) client_negotiation.cc:189] Negotiated authn=SASL
0809 12:11:03.790763 (+ 195us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 12:11:03.790772 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 12:11:03.791876 (+ 1104us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 12:11:03.791883 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 12:11:03.792970 (+ 1087us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 12:11:03.792982 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 12:11:03.793897 (+ 915us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 12:11:03.793904 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 12:11:03.794083 (+ 179us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 12:11:03.794862 (+ 779us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0809 12:11:03.794880 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0809 12:11:03.797767 (+ 2887us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.222.62:44607: Server oryx/127.0.222.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":502,"thread_start_us":334,"threads_started":1}
W20260809 12:11:03.798283 10914 negotiation.cc:337] Failed RPC negotiation. Trace:
0809 12:11:03.788977 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.222.3:38363 (local address 127.0.222.62:44607)
0809 12:11:03.789287 (+ 310us) server_negotiation.cc:206] Beginning negotiation
0809 12:11:03.789295 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0809 12:11:03.789611 (+ 316us) server_negotiation.cc:407] Connection header received
0809 12:11:03.789818 (+ 207us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 12:11:03.789823 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 12:11:03.789910 (+ 87us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 12:11:03.790022 (+ 112us) server_negotiation.cc:226] Negotiated authn=SASL
0809 12:11:03.790949 (+ 927us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 12:11:03.791725 (+ 776us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 12:11:03.793120 (+ 1395us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 12:11:03.793678 (+ 558us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 12:11:03.793850 (+ 172us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 12:11:03.798028 (+ 4178us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.222.3:38363: BlockingRecv error: recv got EOF from 127.0.222.3:38363 (error 108)
Metrics: {"server-negotiator.queue_time_us":144}
Aug 09 12:11:03 dist-test-slave-2w3w krb5kdc[9807](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.222.1@KRBTEST.COM for oryx/127.0.222.62@KRBTEST.COM, Server not found in Kerberos database
Aug 09 12:11:03 dist-test-slave-2w3w krb5kdc[9807](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.222.1@KRBTEST.COM for oryx/127.0.222.62@KRBTEST.COM, Server not found in Kerberos database
I20260809 12:11:03.858724 10919 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 12:11:03.850211 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.222.62:44607 (local address 127.0.222.1:60921)
0809 12:11:03.850789 (+ 578us) negotiation.cc:107] Waiting for socket to connect
0809 12:11:03.850805 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0809 12:11:03.850993 (+ 188us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 12:11:03.851367 (+ 374us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 12:11:03.851374 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 12:11:03.851807 (+ 433us) client_negotiation.cc:189] Negotiated authn=SASL
0809 12:11:03.851954 (+ 147us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 12:11:03.851965 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 12:11:03.852988 (+ 1023us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 12:11:03.852998 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 12:11:03.853904 (+ 906us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 12:11:03.853916 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 12:11:03.854624 (+ 708us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 12:11:03.854630 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 12:11:03.854851 (+ 221us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 12:11:03.855516 (+ 665us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0809 12:11:03.855534 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0809 12:11:03.858444 (+ 2910us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.222.62:44607: Server oryx/127.0.222.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":475,"thread_start_us":318,"threads_started":1}
W20260809 12:11:03.858966 10914 negotiation.cc:337] Failed RPC negotiation. Trace:
0809 12:11:03.850273 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.222.1:60921 (local address 127.0.222.62:44607)
0809 12:11:03.850573 (+ 300us) server_negotiation.cc:206] Beginning negotiation
0809 12:11:03.850582 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0809 12:11:03.850898 (+ 316us) server_negotiation.cc:407] Connection header received
0809 12:11:03.851137 (+ 239us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 12:11:03.851147 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 12:11:03.851212 (+ 65us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 12:11:03.851328 (+ 116us) server_negotiation.cc:226] Negotiated authn=SASL
0809 12:11:03.852121 (+ 793us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 12:11:03.852825 (+ 704us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 12:11:03.854058 (+ 1233us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 12:11:03.854459 (+ 401us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 12:11:03.854607 (+ 148us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 12:11:03.858704 (+ 4097us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.222.1:60921: BlockingRecv error: recv got EOF from 127.0.222.1:60921 (error 108)
Metrics: {"server-negotiator.queue_time_us":95}
Aug 09 12:11:04 dist-test-slave-2w3w krb5kdc[9807](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.222.2@KRBTEST.COM for oryx/127.0.222.62@KRBTEST.COM, Server not found in Kerberos database
Aug 09 12:11:04 dist-test-slave-2w3w krb5kdc[9807](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.222.2@KRBTEST.COM for oryx/127.0.222.62@KRBTEST.COM, Server not found in Kerberos database
I20260809 12:11:04.333228 10920 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 12:11:04.324873 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.222.62:44607 (local address 127.0.222.2:46869)
0809 12:11:04.325459 (+ 586us) negotiation.cc:107] Waiting for socket to connect
0809 12:11:04.325475 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0809 12:11:04.325610 (+ 135us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 12:11:04.325942 (+ 332us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 12:11:04.325945 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 12:11:04.326356 (+ 411us) client_negotiation.cc:189] Negotiated authn=SASL
0809 12:11:04.326520 (+ 164us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 12:11:04.326528 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 12:11:04.327610 (+ 1082us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 12:11:04.327614 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 12:11:04.328508 (+ 894us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 12:11:04.328519 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 12:11:04.329207 (+ 688us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 12:11:04.329211 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 12:11:04.329320 (+ 109us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 12:11:04.329904 (+ 584us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0809 12:11:04.329922 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0809 12:11:04.332867 (+ 2945us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.222.62:44607: Server oryx/127.0.222.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":482,"thread_start_us":388,"threads_started":1}
W20260809 12:11:04.333421 10914 negotiation.cc:337] Failed RPC negotiation. Trace:
0809 12:11:04.324943 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.222.2:46869 (local address 127.0.222.62:44607)
0809 12:11:04.325278 (+ 335us) server_negotiation.cc:206] Beginning negotiation
0809 12:11:04.325285 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0809 12:11:04.325581 (+ 296us) server_negotiation.cc:407] Connection header received
0809 12:11:04.325712 (+ 131us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 12:11:04.325715 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 12:11:04.325810 (+ 95us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 12:11:04.325897 (+ 87us) server_negotiation.cc:226] Negotiated authn=SASL
0809 12:11:04.326658 (+ 761us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 12:11:04.327494 (+ 836us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 12:11:04.328636 (+ 1142us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 12:11:04.329069 (+ 433us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 12:11:04.329169 (+ 100us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 12:11:04.333152 (+ 3983us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.222.2:46869: BlockingRecv error: recv got EOF from 127.0.222.2:46869 (error 108)
Metrics: {"server-negotiator.queue_time_us":139}
Aug 09 12:11:04 dist-test-slave-2w3w krb5kdc[9807](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.222.3@KRBTEST.COM for oryx/127.0.222.62@KRBTEST.COM, Server not found in Kerberos database
Aug 09 12:11:04 dist-test-slave-2w3w krb5kdc[9807](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.222.3@KRBTEST.COM for oryx/127.0.222.62@KRBTEST.COM, Server not found in Kerberos database
I20260809 12:11:04.808120 10922 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 12:11:04.799084 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.222.62:44607 (local address 127.0.222.3:42769)
0809 12:11:04.799634 (+ 550us) negotiation.cc:107] Waiting for socket to connect
0809 12:11:04.799656 (+ 22us) client_negotiation.cc:174] Beginning negotiation
0809 12:11:04.799885 (+ 229us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 12:11:04.800202 (+ 317us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 12:11:04.800205 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 12:11:04.800589 (+ 384us) client_negotiation.cc:189] Negotiated authn=SASL
0809 12:11:04.800733 (+ 144us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 12:11:04.800742 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 12:11:04.801846 (+ 1104us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 12:11:04.801850 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 12:11:04.802779 (+ 929us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 12:11:04.802799 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 12:11:04.803740 (+ 941us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 12:11:04.803750 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 12:11:04.803924 (+ 174us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 12:11:04.804498 (+ 574us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0809 12:11:04.804515 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0809 12:11:04.807732 (+ 3217us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.222.62:44607: Server oryx/127.0.222.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":430,"thread_start_us":308,"threads_started":1}
W20260809 12:11:04.808254 10914 negotiation.cc:337] Failed RPC negotiation. Trace:
0809 12:11:04.799370 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.222.3:42769 (local address 127.0.222.62:44607)
0809 12:11:04.799553 (+ 183us) server_negotiation.cc:206] Beginning negotiation
0809 12:11:04.799558 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0809 12:11:04.799797 (+ 239us) server_negotiation.cc:407] Connection header received
0809 12:11:04.799992 (+ 195us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 12:11:04.799996 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 12:11:04.800072 (+ 76us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 12:11:04.800160 (+ 88us) server_negotiation.cc:226] Negotiated authn=SASL
0809 12:11:04.800929 (+ 769us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 12:11:04.801684 (+ 755us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 12:11:04.802918 (+ 1234us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 12:11:04.803569 (+ 651us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 12:11:04.803669 (+ 100us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 12:11:04.807924 (+ 4255us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.222.3:42769: BlockingRecv error: recv got EOF from 127.0.222.3:42769 (error 108)
Metrics: {"server-negotiator.queue_time_us":65}
Aug 09 12:11:04 dist-test-slave-2w3w krb5kdc[9807](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.222.1@KRBTEST.COM for oryx/127.0.222.62@KRBTEST.COM, Server not found in Kerberos database
Aug 09 12:11:04 dist-test-slave-2w3w krb5kdc[9807](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.222.1@KRBTEST.COM for oryx/127.0.222.62@KRBTEST.COM, Server not found in Kerberos database
I20260809 12:11:04.868746 10923 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 12:11:04.859854 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.222.62:44607 (local address 127.0.222.1:41103)
0809 12:11:04.860340 (+ 486us) negotiation.cc:107] Waiting for socket to connect
0809 12:11:04.860357 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0809 12:11:04.860654 (+ 297us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 12:11:04.860974 (+ 320us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 12:11:04.860978 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 12:11:04.861328 (+ 350us) client_negotiation.cc:189] Negotiated authn=SASL
0809 12:11:04.861516 (+ 188us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 12:11:04.861526 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 12:11:04.862513 (+ 987us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 12:11:04.862517 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 12:11:04.863548 (+ 1031us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 12:11:04.863561 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 12:11:04.864197 (+ 636us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 12:11:04.864201 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 12:11:04.864337 (+ 136us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 12:11:04.864906 (+ 569us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0809 12:11:04.864924 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0809 12:11:04.868406 (+ 3482us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.222.62:44607: Server oryx/127.0.222.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":383,"thread_start_us":274,"threads_started":1}
W20260809 12:11:04.868925 10914 negotiation.cc:337] Failed RPC negotiation. Trace:
0809 12:11:04.859899 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.222.1:41103 (local address 127.0.222.62:44607)
0809 12:11:04.860150 (+ 251us) server_negotiation.cc:206] Beginning negotiation
0809 12:11:04.860157 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0809 12:11:04.860611 (+ 454us) server_negotiation.cc:407] Connection header received
0809 12:11:04.860762 (+ 151us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 12:11:04.860766 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 12:11:04.860837 (+ 71us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 12:11:04.860918 (+ 81us) server_negotiation.cc:226] Negotiated authn=SASL
0809 12:11:04.861628 (+ 710us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 12:11:04.862377 (+ 749us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 12:11:04.863681 (+ 1304us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 12:11:04.864065 (+ 384us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 12:11:04.864158 (+ 93us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 12:11:04.868670 (+ 4512us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.222.1:41103: BlockingRecv error: recv got EOF from 127.0.222.1:41103 (error 108)
Metrics: {"server-negotiator.queue_time_us":69}
Aug 09 12:11:05 dist-test-slave-2w3w krb5kdc[9807](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.222.2@KRBTEST.COM for oryx/127.0.222.62@KRBTEST.COM, Server not found in Kerberos database
Aug 09 12:11:05 dist-test-slave-2w3w krb5kdc[9807](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.222.2@KRBTEST.COM for oryx/127.0.222.62@KRBTEST.COM, Server not found in Kerberos database
I20260809 12:11:05.342988 10925 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 12:11:05.334257 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.222.62:44607 (local address 127.0.222.2:34463)
0809 12:11:05.334886 (+ 629us) negotiation.cc:107] Waiting for socket to connect
0809 12:11:05.334907 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0809 12:11:05.335108 (+ 201us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 12:11:05.335477 (+ 369us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 12:11:05.335481 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 12:11:05.335879 (+ 398us) client_negotiation.cc:189] Negotiated authn=SASL
0809 12:11:05.336050 (+ 171us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 12:11:05.336059 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 12:11:05.337118 (+ 1059us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 12:11:05.337125 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 12:11:05.338040 (+ 915us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 12:11:05.338052 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 12:11:05.338789 (+ 737us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 12:11:05.338802 (+ 13us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 12:11:05.339017 (+ 215us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 12:11:05.339677 (+ 660us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0809 12:11:05.339694 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0809 12:11:05.342562 (+ 2868us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.222.62:44607: Server oryx/127.0.222.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":466,"thread_start_us":310,"threads_started":1}
W20260809 12:11:05.343042 10914 negotiation.cc:337] Failed RPC negotiation. Trace:
0809 12:11:05.334547 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.222.2:34463 (local address 127.0.222.62:44607)
0809 12:11:05.334781 (+ 234us) server_negotiation.cc:206] Beginning negotiation
0809 12:11:05.334787 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0809 12:11:05.335020 (+ 233us) server_negotiation.cc:407] Connection header received
0809 12:11:05.335245 (+ 225us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 12:11:05.335262 (+ 17us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 12:11:05.335327 (+ 65us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 12:11:05.335436 (+ 109us) server_negotiation.cc:226] Negotiated authn=SASL
0809 12:11:05.336203 (+ 767us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 12:11:05.336970 (+ 767us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 12:11:05.338191 (+ 1221us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 12:11:05.338618 (+ 427us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 12:11:05.338757 (+ 139us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 12:11:05.342769 (+ 4012us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.222.2:34463: BlockingRecv error: recv got EOF from 127.0.222.2:34463 (error 108)
Metrics: {"server-negotiator.queue_time_us":81}
Aug 09 12:11:05 dist-test-slave-2w3w krb5kdc[9807](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.222.3@KRBTEST.COM for oryx/127.0.222.62@KRBTEST.COM, Server not found in Kerberos database
Aug 09 12:11:05 dist-test-slave-2w3w krb5kdc[9807](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.222.3@KRBTEST.COM for oryx/127.0.222.62@KRBTEST.COM, Server not found in Kerberos database
I20260809 12:11:05.817909 10927 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 12:11:05.809253 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.222.62:44607 (local address 127.0.222.3:54439)
0809 12:11:05.809763 (+ 510us) negotiation.cc:107] Waiting for socket to connect
0809 12:11:05.809785 (+ 22us) client_negotiation.cc:174] Beginning negotiation
0809 12:11:05.809965 (+ 180us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 12:11:05.810353 (+ 388us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 12:11:05.810358 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 12:11:05.810736 (+ 378us) client_negotiation.cc:189] Negotiated authn=SASL
0809 12:11:05.810912 (+ 176us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 12:11:05.810921 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 12:11:05.812009 (+ 1088us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 12:11:05.812016 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 12:11:05.812930 (+ 914us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 12:11:05.812942 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 12:11:05.813703 (+ 761us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 12:11:05.813708 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 12:11:05.813900 (+ 192us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 12:11:05.814588 (+ 688us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0809 12:11:05.814605 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0809 12:11:05.817522 (+ 2917us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.222.62:44607: Server oryx/127.0.222.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":414,"thread_start_us":277,"threads_started":1}
W20260809 12:11:05.818033 10914 negotiation.cc:337] Failed RPC negotiation. Trace:
0809 12:11:05.809308 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.222.3:54439 (local address 127.0.222.62:44607)
0809 12:11:05.809561 (+ 253us) server_negotiation.cc:206] Beginning negotiation
0809 12:11:05.809569 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0809 12:11:05.809894 (+ 325us) server_negotiation.cc:407] Connection header received
0809 12:11:05.810101 (+ 207us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 12:11:05.810107 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 12:11:05.810195 (+ 88us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 12:11:05.810305 (+ 110us) server_negotiation.cc:226] Negotiated authn=SASL
0809 12:11:05.811096 (+ 791us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 12:11:05.811868 (+ 772us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 12:11:05.813080 (+ 1212us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 12:11:05.813537 (+ 457us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 12:11:05.813674 (+ 137us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 12:11:05.817768 (+ 4094us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.222.3:54439: BlockingRecv error: recv got EOF from 127.0.222.3:54439 (error 108)
Metrics: {"server-negotiator.queue_time_us":69}
Aug 09 12:11:05 dist-test-slave-2w3w krb5kdc[9807](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.222.1@KRBTEST.COM for oryx/127.0.222.62@KRBTEST.COM, Server not found in Kerberos database
Aug 09 12:11:05 dist-test-slave-2w3w krb5kdc[9807](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.222.1@KRBTEST.COM for oryx/127.0.222.62@KRBTEST.COM, Server not found in Kerberos database
I20260809 12:11:05.878291 10928 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 12:11:05.869912 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.222.62:44607 (local address 127.0.222.1:36415)
0809 12:11:05.870588 (+ 676us) negotiation.cc:107] Waiting for socket to connect
0809 12:11:05.870605 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0809 12:11:05.870738 (+ 133us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 12:11:05.871073 (+ 335us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 12:11:05.871077 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 12:11:05.871441 (+ 364us) client_negotiation.cc:189] Negotiated authn=SASL
0809 12:11:05.871608 (+ 167us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 12:11:05.871617 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 12:11:05.872637 (+ 1020us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 12:11:05.872645 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 12:11:05.873695 (+ 1050us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 12:11:05.873708 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 12:11:05.874421 (+ 713us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 12:11:05.874425 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 12:11:05.874562 (+ 137us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 12:11:05.875236 (+ 674us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0809 12:11:05.875253 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0809 12:11:05.878052 (+ 2799us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.222.62:44607: Server oryx/127.0.222.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":528,"thread_start_us":419,"threads_started":1}
W20260809 12:11:05.878500 10914 negotiation.cc:337] Failed RPC negotiation. Trace:
0809 12:11:05.869973 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.222.1:36415 (local address 127.0.222.62:44607)
0809 12:11:05.870205 (+ 232us) server_negotiation.cc:206] Beginning negotiation
0809 12:11:05.870212 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0809 12:11:05.870709 (+ 497us) server_negotiation.cc:407] Connection header received
0809 12:11:05.870845 (+ 136us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 12:11:05.870848 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 12:11:05.870913 (+ 65us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 12:11:05.871058 (+ 145us) server_negotiation.cc:226] Negotiated authn=SASL
0809 12:11:05.871729 (+ 671us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 12:11:05.872503 (+ 774us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 12:11:05.873865 (+ 1362us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 12:11:05.874295 (+ 430us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 12:11:05.874388 (+ 93us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 12:11:05.878249 (+ 3861us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.222.1:36415: BlockingRecv error: recv got EOF from 127.0.222.1:36415 (error 108)
Metrics: {"server-negotiator.queue_time_us":73}
Aug 09 12:11:06 dist-test-slave-2w3w krb5kdc[9807](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.222.2@KRBTEST.COM for oryx/127.0.222.62@KRBTEST.COM, Server not found in Kerberos database
Aug 09 12:11:06 dist-test-slave-2w3w krb5kdc[9807](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.222.2@KRBTEST.COM for oryx/127.0.222.62@KRBTEST.COM, Server not found in Kerberos database
I20260809 12:11:06.355634 10929 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 12:11:06.344090 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.222.62:44607 (local address 127.0.222.2:48955)
0809 12:11:06.344628 (+ 538us) negotiation.cc:107] Waiting for socket to connect
0809 12:11:06.344646 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0809 12:11:06.344792 (+ 146us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 12:11:06.345237 (+ 445us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 12:11:06.345242 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 12:11:06.345744 (+ 502us) client_negotiation.cc:189] Negotiated authn=SASL
0809 12:11:06.345913 (+ 169us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 12:11:06.345922 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 12:11:06.347124 (+ 1202us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 12:11:06.347129 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 12:11:06.348223 (+ 1094us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 12:11:06.348237 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 12:11:06.348894 (+ 657us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 12:11:06.348898 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 12:11:06.349022 (+ 124us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 12:11:06.352435 (+ 3413us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0809 12:11:06.352454 (+ 19us) client_negotiation.cc:656] calling sasl_client_start()
0809 12:11:06.355390 (+ 2936us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.222.62:44607: Server oryx/127.0.222.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":418,"thread_start_us":300,"threads_started":1}
W20260809 12:11:06.355875 10914 negotiation.cc:337] Failed RPC negotiation. Trace:
0809 12:11:06.344152 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.222.2:48955 (local address 127.0.222.62:44607)
0809 12:11:06.344345 (+ 193us) server_negotiation.cc:206] Beginning negotiation
0809 12:11:06.344349 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0809 12:11:06.344742 (+ 393us) server_negotiation.cc:407] Connection header received
0809 12:11:06.344918 (+ 176us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 12:11:06.344922 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 12:11:06.345050 (+ 128us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 12:11:06.345165 (+ 115us) server_negotiation.cc:226] Negotiated authn=SASL
0809 12:11:06.346048 (+ 883us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 12:11:06.346976 (+ 928us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 12:11:06.348355 (+ 1379us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 12:11:06.348768 (+ 413us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 12:11:06.348865 (+ 97us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 12:11:06.355597 (+ 6732us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.222.2:48955: BlockingRecv error: recv got EOF from 127.0.222.2:48955 (error 108)
Metrics: {"server-negotiator.queue_time_us":58}
Aug 09 12:11:06 dist-test-slave-2w3w krb5kdc[9807](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.222.3@KRBTEST.COM for oryx/127.0.222.62@KRBTEST.COM, Server not found in Kerberos database
Aug 09 12:11:06 dist-test-slave-2w3w krb5kdc[9807](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.222.3@KRBTEST.COM for oryx/127.0.222.62@KRBTEST.COM, Server not found in Kerberos database
I20260809 12:11:06.827741 10930 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 12:11:06.819048 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.222.62:44607 (local address 127.0.222.3:38537)
0809 12:11:06.819608 (+ 560us) negotiation.cc:107] Waiting for socket to connect
0809 12:11:06.819640 (+ 32us) client_negotiation.cc:174] Beginning negotiation
0809 12:11:06.819804 (+ 164us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 12:11:06.820192 (+ 388us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 12:11:06.820198 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 12:11:06.820617 (+ 419us) client_negotiation.cc:189] Negotiated authn=SASL
0809 12:11:06.820768 (+ 151us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 12:11:06.820777 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 12:11:06.821829 (+ 1052us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 12:11:06.821837 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 12:11:06.822833 (+ 996us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 12:11:06.822845 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 12:11:06.823625 (+ 780us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 12:11:06.823631 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 12:11:06.823813 (+ 182us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 12:11:06.824490 (+ 677us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0809 12:11:06.824508 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0809 12:11:06.827510 (+ 3002us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.222.62:44607: Server oryx/127.0.222.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":437,"thread_start_us":319,"threads_started":1}
W20260809 12:11:06.828202 10914 negotiation.cc:337] Failed RPC negotiation. Trace:
0809 12:11:06.819105 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.222.3:38537 (local address 127.0.222.62:44607)
0809 12:11:06.819336 (+ 231us) server_negotiation.cc:206] Beginning negotiation
0809 12:11:06.819344 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0809 12:11:06.819735 (+ 391us) server_negotiation.cc:407] Connection header received
0809 12:11:06.819945 (+ 210us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 12:11:06.819950 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 12:11:06.820050 (+ 100us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 12:11:06.820157 (+ 107us) server_negotiation.cc:226] Negotiated authn=SASL
0809 12:11:06.820948 (+ 791us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 12:11:06.821685 (+ 737us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 12:11:06.823010 (+ 1325us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 12:11:06.823459 (+ 449us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 12:11:06.823603 (+ 144us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 12:11:06.827748 (+ 4145us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.222.3:38537: BlockingRecv error: recv got EOF from 127.0.222.3:38537 (error 108)
Metrics: {"server-negotiator.queue_time_us":61}
Aug 09 12:11:06 dist-test-slave-2w3w krb5kdc[9807](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.222.1@KRBTEST.COM for oryx/127.0.222.62@KRBTEST.COM, Server not found in Kerberos database
Aug 09 12:11:06 dist-test-slave-2w3w krb5kdc[9807](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.222.1@KRBTEST.COM for oryx/127.0.222.62@KRBTEST.COM, Server not found in Kerberos database
I20260809 12:11:06.888059 10931 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 12:11:06.879498 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.222.62:44607 (local address 127.0.222.1:41271)
0809 12:11:06.880069 (+ 571us) negotiation.cc:107] Waiting for socket to connect
0809 12:11:06.880086 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0809 12:11:06.880250 (+ 164us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 12:11:06.880613 (+ 363us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 12:11:06.880617 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 12:11:06.881126 (+ 509us) client_negotiation.cc:189] Negotiated authn=SASL
0809 12:11:06.881308 (+ 182us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 12:11:06.881319 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 12:11:06.882331 (+ 1012us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 12:11:06.882337 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 12:11:06.883331 (+ 994us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 12:11:06.883344 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 12:11:06.884055 (+ 711us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 12:11:06.884061 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 12:11:06.884238 (+ 177us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 12:11:06.884807 (+ 569us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0809 12:11:06.884826 (+ 19us) client_negotiation.cc:656] calling sasl_client_start()
0809 12:11:06.887702 (+ 2876us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.222.62:44607: Server oryx/127.0.222.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":456,"thread_start_us":289,"threads_started":1}
W20260809 12:11:06.888302 10914 negotiation.cc:337] Failed RPC negotiation. Trace:
0809 12:11:06.879585 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.222.1:41271 (local address 127.0.222.62:44607)
0809 12:11:06.879793 (+ 208us) server_negotiation.cc:206] Beginning negotiation
0809 12:11:06.879799 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0809 12:11:06.880180 (+ 381us) server_negotiation.cc:407] Connection header received
0809 12:11:06.880386 (+ 206us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 12:11:06.880392 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 12:11:06.880469 (+ 77us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 12:11:06.880575 (+ 106us) server_negotiation.cc:226] Negotiated authn=SASL
0809 12:11:06.881458 (+ 883us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 12:11:06.882172 (+ 714us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 12:11:06.883474 (+ 1302us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 12:11:06.883894 (+ 420us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 12:11:06.884028 (+ 134us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 12:11:06.887957 (+ 3929us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.222.1:41271: BlockingRecv error: recv got EOF from 127.0.222.1:41271 (error 108)
Metrics: {"server-negotiator.queue_time_us":55}
Aug 09 12:11:07 dist-test-slave-2w3w krb5kdc[9807](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.222.2@KRBTEST.COM for oryx/127.0.222.62@KRBTEST.COM, Server not found in Kerberos database
Aug 09 12:11:07 dist-test-slave-2w3w krb5kdc[9807](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.222.2@KRBTEST.COM for oryx/127.0.222.62@KRBTEST.COM, Server not found in Kerberos database
I20260809 12:11:07.365866 10932 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 12:11:07.356960 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.222.62:44607 (local address 127.0.222.2:50123)
0809 12:11:07.357528 (+ 568us) negotiation.cc:107] Waiting for socket to connect
0809 12:11:07.357545 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0809 12:11:07.357660 (+ 115us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 12:11:07.357995 (+ 335us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 12:11:07.357999 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 12:11:07.358419 (+ 420us) client_negotiation.cc:189] Negotiated authn=SASL
0809 12:11:07.358582 (+ 163us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 12:11:07.358592 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 12:11:07.359673 (+ 1081us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 12:11:07.359677 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 12:11:07.360651 (+ 974us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 12:11:07.360663 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 12:11:07.361334 (+ 671us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 12:11:07.361338 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 12:11:07.361469 (+ 131us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 12:11:07.362117 (+ 648us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0809 12:11:07.362145 (+ 28us) client_negotiation.cc:656] calling sasl_client_start()
0809 12:11:07.365611 (+ 3466us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.222.62:44607: Server oryx/127.0.222.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":448,"thread_start_us":292,"threads_started":1}
W20260809 12:11:07.366012 10914 negotiation.cc:337] Failed RPC negotiation. Trace:
0809 12:11:07.357166 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.222.2:50123 (local address 127.0.222.62:44607)
0809 12:11:07.357340 (+ 174us) server_negotiation.cc:206] Beginning negotiation
0809 12:11:07.357344 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0809 12:11:07.357625 (+ 281us) server_negotiation.cc:407] Connection header received
0809 12:11:07.357759 (+ 134us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 12:11:07.357763 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 12:11:07.357824 (+ 61us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 12:11:07.357908 (+ 84us) server_negotiation.cc:226] Negotiated authn=SASL
0809 12:11:07.358713 (+ 805us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 12:11:07.359531 (+ 818us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 12:11:07.360781 (+ 1250us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 12:11:07.361204 (+ 423us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 12:11:07.361297 (+ 93us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 12:11:07.365795 (+ 4498us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.222.2:50123: BlockingRecv error: recv got EOF from 127.0.222.2:50123 (error 108)
Metrics: {"server-negotiator.queue_time_us":63}
Aug 09 12:11:07 dist-test-slave-2w3w krb5kdc[9807](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.222.3@KRBTEST.COM for oryx/127.0.222.62@KRBTEST.COM, Server not found in Kerberos database
Aug 09 12:11:07 dist-test-slave-2w3w krb5kdc[9807](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.222.3@KRBTEST.COM for oryx/127.0.222.62@KRBTEST.COM, Server not found in Kerberos database
I20260809 12:11:07.837879 10934 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 12:11:07.828821 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.222.62:44607 (local address 127.0.222.3:46459)
0809 12:11:07.829405 (+ 584us) negotiation.cc:107] Waiting for socket to connect
0809 12:11:07.829426 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0809 12:11:07.829554 (+ 128us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 12:11:07.829905 (+ 351us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 12:11:07.829908 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 12:11:07.830324 (+ 416us) client_negotiation.cc:189] Negotiated authn=SASL
0809 12:11:07.830501 (+ 177us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 12:11:07.830509 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 12:11:07.831732 (+ 1223us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 12:11:07.831736 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 12:11:07.832634 (+ 898us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 12:11:07.832644 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 12:11:07.833374 (+ 730us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 12:11:07.833378 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 12:11:07.833521 (+ 143us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 12:11:07.834201 (+ 680us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0809 12:11:07.834219 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0809 12:11:07.837624 (+ 3405us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.222.62:44607: Server oryx/127.0.222.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":471,"thread_start_us":303,"threads_started":1}
W20260809 12:11:07.838095 10914 negotiation.cc:337] Failed RPC negotiation. Trace:
0809 12:11:07.828960 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.222.3:46459 (local address 127.0.222.62:44607)
0809 12:11:07.829188 (+ 228us) server_negotiation.cc:206] Beginning negotiation
0809 12:11:07.829195 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0809 12:11:07.829522 (+ 327us) server_negotiation.cc:407] Connection header received
0809 12:11:07.829674 (+ 152us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 12:11:07.829678 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 12:11:07.829764 (+ 86us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 12:11:07.829860 (+ 96us) server_negotiation.cc:226] Negotiated authn=SASL
0809 12:11:07.830668 (+ 808us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 12:11:07.831590 (+ 922us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 12:11:07.832784 (+ 1194us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 12:11:07.833214 (+ 430us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 12:11:07.833347 (+ 133us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 12:11:07.837845 (+ 4498us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.222.3:46459: BlockingRecv error: recv got EOF from 127.0.222.3:46459 (error 108)
Metrics: {"server-negotiator.queue_time_us":85}
Aug 09 12:11:07 dist-test-slave-2w3w krb5kdc[9807](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.222.1@KRBTEST.COM for oryx/127.0.222.62@KRBTEST.COM, Server not found in Kerberos database
Aug 09 12:11:07 dist-test-slave-2w3w krb5kdc[9807](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.222.1@KRBTEST.COM for oryx/127.0.222.62@KRBTEST.COM, Server not found in Kerberos database
I20260809 12:11:07.898048 10935 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 12:11:07.889061 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.222.62:44607 (local address 127.0.222.1:39785)
0809 12:11:07.889618 (+ 557us) negotiation.cc:107] Waiting for socket to connect
0809 12:11:07.889635 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0809 12:11:07.889750 (+ 115us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 12:11:07.890120 (+ 370us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 12:11:07.890123 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 12:11:07.890503 (+ 380us) client_negotiation.cc:189] Negotiated authn=SASL
0809 12:11:07.890667 (+ 164us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 12:11:07.890676 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 12:11:07.891879 (+ 1203us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 12:11:07.891883 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 12:11:07.892793 (+ 910us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 12:11:07.892805 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 12:11:07.893602 (+ 797us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 12:11:07.893606 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 12:11:07.893736 (+ 130us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 12:11:07.894447 (+ 711us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0809 12:11:07.894474 (+ 27us) client_negotiation.cc:656] calling sasl_client_start()
0809 12:11:07.897787 (+ 3313us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.222.62:44607: Server oryx/127.0.222.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":448,"thread_start_us":318,"threads_started":1}
W20260809 12:11:07.898543 10914 negotiation.cc:337] Failed RPC negotiation. Trace:
0809 12:11:07.889117 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.222.1:39785 (local address 127.0.222.62:44607)
0809 12:11:07.889352 (+ 235us) server_negotiation.cc:206] Beginning negotiation
0809 12:11:07.889357 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0809 12:11:07.889715 (+ 358us) server_negotiation.cc:407] Connection header received
0809 12:11:07.889908 (+ 193us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 12:11:07.889913 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 12:11:07.889984 (+ 71us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 12:11:07.890081 (+ 97us) server_negotiation.cc:226] Negotiated authn=SASL
0809 12:11:07.890822 (+ 741us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 12:11:07.891735 (+ 913us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 12:11:07.892939 (+ 1204us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 12:11:07.893448 (+ 509us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 12:11:07.893555 (+ 107us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 12:11:07.898276 (+ 4721us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.222.1:39785: BlockingRecv error: recv got EOF from 127.0.222.1:39785 (error 108)
Metrics: {"server-negotiator.queue_time_us":83}
Aug 09 12:11:08 dist-test-slave-2w3w krb5kdc[9807](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.222.2@KRBTEST.COM for oryx/127.0.222.62@KRBTEST.COM, Server not found in Kerberos database
Aug 09 12:11:08 dist-test-slave-2w3w krb5kdc[9807](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.222.2@KRBTEST.COM for oryx/127.0.222.62@KRBTEST.COM, Server not found in Kerberos database
I20260809 12:11:08.374922 10936 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 12:11:08.366895 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.222.62:44607 (local address 127.0.222.2:56683)
0809 12:11:08.367466 (+ 571us) negotiation.cc:107] Waiting for socket to connect
0809 12:11:08.367483 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0809 12:11:08.367601 (+ 118us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 12:11:08.367910 (+ 309us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 12:11:08.367914 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 12:11:08.368287 (+ 373us) client_negotiation.cc:189] Negotiated authn=SASL
0809 12:11:08.368453 (+ 166us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 12:11:08.368462 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 12:11:08.369388 (+ 926us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 12:11:08.369392 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 12:11:08.370290 (+ 898us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 12:11:08.370310 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 12:11:08.371037 (+ 727us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 12:11:08.371042 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 12:11:08.371164 (+ 122us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 12:11:08.371749 (+ 585us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0809 12:11:08.371765 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0809 12:11:08.374688 (+ 2923us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.222.62:44607: Server oryx/127.0.222.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":476,"thread_start_us":346,"threads_started":1}
W20260809 12:11:08.375159 10914 negotiation.cc:337] Failed RPC negotiation. Trace:
0809 12:11:08.366945 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.222.2:56683 (local address 127.0.222.62:44607)
0809 12:11:08.367182 (+ 237us) server_negotiation.cc:206] Beginning negotiation
0809 12:11:08.367192 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0809 12:11:08.367571 (+ 379us) server_negotiation.cc:407] Connection header received
0809 12:11:08.367699 (+ 128us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 12:11:08.367703 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 12:11:08.367781 (+ 78us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 12:11:08.367862 (+ 81us) server_negotiation.cc:226] Negotiated authn=SASL
0809 12:11:08.368575 (+ 713us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 12:11:08.369259 (+ 684us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 12:11:08.370424 (+ 1165us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 12:11:08.370852 (+ 428us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 12:11:08.370944 (+ 92us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 12:11:08.374900 (+ 3956us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.222.2:56683: BlockingRecv error: recv got EOF from 127.0.222.2:56683 (error 108)
Metrics: {"server-negotiator.queue_time_us":72}
Aug 09 12:11:08 dist-test-slave-2w3w krb5kdc[9807](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.222.3@KRBTEST.COM for oryx/127.0.222.62@KRBTEST.COM, Server not found in Kerberos database
Aug 09 12:11:08 dist-test-slave-2w3w krb5kdc[9807](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.222.3@KRBTEST.COM for oryx/127.0.222.62@KRBTEST.COM, Server not found in Kerberos database
I20260809 12:11:08.847585 10937 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 12:11:08.838866 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.222.62:44607 (local address 127.0.222.3:50835)
0809 12:11:08.839494 (+ 628us) negotiation.cc:107] Waiting for socket to connect
0809 12:11:08.839517 (+ 23us) client_negotiation.cc:174] Beginning negotiation
0809 12:11:08.839684 (+ 167us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 12:11:08.840182 (+ 498us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 12:11:08.840186 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 12:11:08.840731 (+ 545us) client_negotiation.cc:189] Negotiated authn=SASL
0809 12:11:08.840921 (+ 190us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 12:11:08.840930 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 12:11:08.841940 (+ 1010us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 12:11:08.841943 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 12:11:08.842842 (+ 899us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 12:11:08.842852 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 12:11:08.843592 (+ 740us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 12:11:08.843597 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 12:11:08.843721 (+ 124us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 12:11:08.844375 (+ 654us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0809 12:11:08.844393 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0809 12:11:08.847362 (+ 2969us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.222.62:44607: Server oryx/127.0.222.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":509,"thread_start_us":379,"threads_started":1}
W20260809 12:11:08.847797 10914 negotiation.cc:337] Failed RPC negotiation. Trace:
0809 12:11:08.838905 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.222.3:50835 (local address 127.0.222.62:44607)
0809 12:11:08.839148 (+ 243us) server_negotiation.cc:206] Beginning negotiation
0809 12:11:08.839155 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0809 12:11:08.839609 (+ 454us) server_negotiation.cc:407] Connection header received
0809 12:11:08.839901 (+ 292us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 12:11:08.839907 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 12:11:08.840001 (+ 94us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 12:11:08.840123 (+ 122us) server_negotiation.cc:226] Negotiated authn=SASL
0809 12:11:08.841053 (+ 930us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 12:11:08.841822 (+ 769us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 12:11:08.842993 (+ 1171us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 12:11:08.843449 (+ 456us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 12:11:08.843550 (+ 101us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 12:11:08.847554 (+ 4004us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.222.3:50835: BlockingRecv error: recv got EOF from 127.0.222.3:50835 (error 108)
Metrics: {"server-negotiator.queue_time_us":77}
Aug 09 12:11:08 dist-test-slave-2w3w krb5kdc[9807](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.222.1@KRBTEST.COM for oryx/127.0.222.62@KRBTEST.COM, Server not found in Kerberos database
Aug 09 12:11:08 dist-test-slave-2w3w krb5kdc[9807](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.222.1@KRBTEST.COM for oryx/127.0.222.62@KRBTEST.COM, Server not found in Kerberos database
I20260809 12:11:08.908569 10938 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 12:11:08.899299 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.222.62:44607 (local address 127.0.222.1:36509)
0809 12:11:08.900108 (+ 809us) negotiation.cc:107] Waiting for socket to connect
0809 12:11:08.900128 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0809 12:11:08.900315 (+ 187us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 12:11:08.900771 (+ 456us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 12:11:08.900776 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 12:11:08.901308 (+ 532us) client_negotiation.cc:189] Negotiated authn=SASL
0809 12:11:08.901495 (+ 187us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 12:11:08.901504 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 12:11:08.902614 (+ 1110us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 12:11:08.902617 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 12:11:08.903860 (+ 1243us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 12:11:08.903872 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 12:11:08.904619 (+ 747us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 12:11:08.904623 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 12:11:08.904735 (+ 112us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 12:11:08.905341 (+ 606us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0809 12:11:08.905359 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0809 12:11:08.908309 (+ 2950us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.222.62:44607: Server oryx/127.0.222.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":655,"thread_start_us":345,"threads_started":1}
W20260809 12:11:08.908742 10914 negotiation.cc:337] Failed RPC negotiation. Trace:
0809 12:11:08.899411 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.222.1:36509 (local address 127.0.222.62:44607)
0809 12:11:08.899657 (+ 246us) server_negotiation.cc:206] Beginning negotiation
0809 12:11:08.899667 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0809 12:11:08.900227 (+ 560us) server_negotiation.cc:407] Connection header received
0809 12:11:08.900512 (+ 285us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 12:11:08.900517 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 12:11:08.900598 (+ 81us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 12:11:08.900705 (+ 107us) server_negotiation.cc:226] Negotiated authn=SASL
0809 12:11:08.901659 (+ 954us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 12:11:08.902482 (+ 823us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 12:11:08.904046 (+ 1564us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 12:11:08.904481 (+ 435us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 12:11:08.904581 (+ 100us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 12:11:08.908526 (+ 3945us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.222.1:36509: BlockingRecv error: recv got EOF from 127.0.222.1:36509 (error 108)
Metrics: {"server-negotiator.queue_time_us":71}
Aug 09 12:11:09 dist-test-slave-2w3w krb5kdc[9807](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.222.2@KRBTEST.COM for oryx/127.0.222.62@KRBTEST.COM, Server not found in Kerberos database
Aug 09 12:11:09 dist-test-slave-2w3w krb5kdc[9807](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.222.2@KRBTEST.COM for oryx/127.0.222.62@KRBTEST.COM, Server not found in Kerberos database
I20260809 12:11:09.384438 10939 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 12:11:09.375912 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.222.62:44607 (local address 127.0.222.2:50047)
0809 12:11:09.376462 (+ 550us) negotiation.cc:107] Waiting for socket to connect
0809 12:11:09.376492 (+ 30us) client_negotiation.cc:174] Beginning negotiation
0809 12:11:09.376610 (+ 118us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 12:11:09.376902 (+ 292us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 12:11:09.376905 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 12:11:09.377366 (+ 461us) client_negotiation.cc:189] Negotiated authn=SASL
0809 12:11:09.377531 (+ 165us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 12:11:09.377539 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 12:11:09.378526 (+ 987us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 12:11:09.378530 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 12:11:09.379484 (+ 954us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 12:11:09.379496 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 12:11:09.380230 (+ 734us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 12:11:09.380234 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 12:11:09.380350 (+ 116us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 12:11:09.380943 (+ 593us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0809 12:11:09.380960 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0809 12:11:09.384090 (+ 3130us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.222.62:44607: Server oryx/127.0.222.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":446,"thread_start_us":257,"threads_started":1}
W20260809 12:11:09.384701 10914 negotiation.cc:337] Failed RPC negotiation. Trace:
0809 12:11:09.375976 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.222.2:50047 (local address 127.0.222.62:44607)
0809 12:11:09.376273 (+ 297us) server_negotiation.cc:206] Beginning negotiation
0809 12:11:09.376279 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0809 12:11:09.376580 (+ 301us) server_negotiation.cc:407] Connection header received
0809 12:11:09.376708 (+ 128us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 12:11:09.376712 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 12:11:09.376775 (+ 63us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 12:11:09.376861 (+ 86us) server_negotiation.cc:226] Negotiated authn=SASL
0809 12:11:09.377660 (+ 799us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 12:11:09.378393 (+ 733us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 12:11:09.379626 (+ 1233us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 12:11:09.380085 (+ 459us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 12:11:09.380181 (+ 96us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 12:11:09.384415 (+ 4234us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.222.2:50047: BlockingRecv error: recv got EOF from 127.0.222.2:50047 (error 108)
Metrics: {"server-negotiator.queue_time_us":123}
Aug 09 12:11:09 dist-test-slave-2w3w krb5kdc[9807](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.222.3@KRBTEST.COM for oryx/127.0.222.62@KRBTEST.COM, Server not found in Kerberos database
Aug 09 12:11:09 dist-test-slave-2w3w krb5kdc[9807](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.222.3@KRBTEST.COM for oryx/127.0.222.62@KRBTEST.COM, Server not found in Kerberos database
I20260809 12:11:09.857326 10940 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 12:11:09.848659 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.222.62:44607 (local address 127.0.222.3:37385)
0809 12:11:09.849259 (+ 600us) negotiation.cc:107] Waiting for socket to connect
0809 12:11:09.849280 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0809 12:11:09.849403 (+ 123us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 12:11:09.849764 (+ 361us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 12:11:09.849767 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 12:11:09.850222 (+ 455us) client_negotiation.cc:189] Negotiated authn=SASL
0809 12:11:09.850390 (+ 168us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 12:11:09.850398 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 12:11:09.851445 (+ 1047us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 12:11:09.851448 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 12:11:09.852423 (+ 975us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 12:11:09.852435 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 12:11:09.853172 (+ 737us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 12:11:09.853177 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 12:11:09.853294 (+ 117us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 12:11:09.854009 (+ 715us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0809 12:11:09.854027 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0809 12:11:09.857048 (+ 3021us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.222.62:44607: Server oryx/127.0.222.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":483,"thread_start_us":294,"threads_started":1}
W20260809 12:11:09.857559 10914 negotiation.cc:337] Failed RPC negotiation. Trace:
0809 12:11:09.848706 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.222.3:37385 (local address 127.0.222.62:44607)
0809 12:11:09.848915 (+ 209us) server_negotiation.cc:206] Beginning negotiation
0809 12:11:09.848920 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0809 12:11:09.849368 (+ 448us) server_negotiation.cc:407] Connection header received
0809 12:11:09.849491 (+ 123us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 12:11:09.849495 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 12:11:09.849574 (+ 79us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 12:11:09.849729 (+ 155us) server_negotiation.cc:226] Negotiated authn=SASL
0809 12:11:09.850536 (+ 807us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 12:11:09.851320 (+ 784us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 12:11:09.852552 (+ 1232us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 12:11:09.853024 (+ 472us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 12:11:09.853129 (+ 105us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 12:11:09.857292 (+ 4163us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.222.3:37385: BlockingRecv error: recv got EOF from 127.0.222.3:37385 (error 108)
Metrics: {"server-negotiator.queue_time_us":69}
Aug 09 12:11:09 dist-test-slave-2w3w krb5kdc[9807](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.222.1@KRBTEST.COM for oryx/127.0.222.62@KRBTEST.COM, Server not found in Kerberos database
Aug 09 12:11:09 dist-test-slave-2w3w krb5kdc[9807](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.222.1@KRBTEST.COM for oryx/127.0.222.62@KRBTEST.COM, Server not found in Kerberos database
I20260809 12:11:09.918601 10941 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 12:11:09.909743 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.222.62:44607 (local address 127.0.222.1:56027)
0809 12:11:09.910371 (+ 628us) negotiation.cc:107] Waiting for socket to connect
0809 12:11:09.910387 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0809 12:11:09.910547 (+ 160us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 12:11:09.910918 (+ 371us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 12:11:09.910923 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 12:11:09.911390 (+ 467us) client_negotiation.cc:189] Negotiated authn=SASL
0809 12:11:09.911546 (+ 156us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 12:11:09.911555 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 12:11:09.912646 (+ 1091us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 12:11:09.912652 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 12:11:09.913586 (+ 934us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 12:11:09.913598 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 12:11:09.914397 (+ 799us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 12:11:09.914405 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 12:11:09.914566 (+ 161us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 12:11:09.915276 (+ 710us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0809 12:11:09.915294 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0809 12:11:09.918340 (+ 3046us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.222.62:44607: Server oryx/127.0.222.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":494,"thread_start_us":333,"threads_started":1}
W20260809 12:11:09.918831 10914 negotiation.cc:337] Failed RPC negotiation. Trace:
0809 12:11:09.909972 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.222.1:56027 (local address 127.0.222.62:44607)
0809 12:11:09.910211 (+ 239us) server_negotiation.cc:206] Beginning negotiation
0809 12:11:09.910217 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0809 12:11:09.910499 (+ 282us) server_negotiation.cc:407] Connection header received
0809 12:11:09.910694 (+ 195us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 12:11:09.910700 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 12:11:09.910777 (+ 77us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 12:11:09.910883 (+ 106us) server_negotiation.cc:226] Negotiated authn=SASL
0809 12:11:09.911705 (+ 822us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 12:11:09.912497 (+ 792us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 12:11:09.913741 (+ 1244us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 12:11:09.914196 (+ 455us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 12:11:09.914331 (+ 135us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 12:11:09.918557 (+ 4226us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.222.1:56027: BlockingRecv error: recv got EOF from 127.0.222.1:56027 (error 108)
Metrics: {"server-negotiator.queue_time_us":86}
Aug 09 12:11:10 dist-test-slave-2w3w krb5kdc[9807](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.222.2@KRBTEST.COM for oryx/127.0.222.62@KRBTEST.COM, Server not found in Kerberos database
Aug 09 12:11:10 dist-test-slave-2w3w krb5kdc[9807](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.222.2@KRBTEST.COM for oryx/127.0.222.62@KRBTEST.COM, Server not found in Kerberos database
I20260809 12:11:10.395618 10942 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 12:11:10.385725 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.222.62:44607 (local address 127.0.222.2:48585)
0809 12:11:10.386367 (+ 642us) negotiation.cc:107] Waiting for socket to connect
0809 12:11:10.386383 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0809 12:11:10.386582 (+ 199us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 12:11:10.386949 (+ 367us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 12:11:10.386953 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 12:11:10.387428 (+ 475us) client_negotiation.cc:189] Negotiated authn=SASL
0809 12:11:10.387586 (+ 158us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 12:11:10.387596 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 12:11:10.389168 (+ 1572us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 12:11:10.389173 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 12:11:10.390510 (+ 1337us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 12:11:10.390525 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 12:11:10.391307 (+ 782us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 12:11:10.391313 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 12:11:10.391439 (+ 126us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 12:11:10.392069 (+ 630us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0809 12:11:10.392088 (+ 19us) client_negotiation.cc:656] calling sasl_client_start()
0809 12:11:10.395263 (+ 3175us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.222.62:44607: Server oryx/127.0.222.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":524,"thread_start_us":340,"threads_started":1}
W20260809 12:11:10.395803 10914 negotiation.cc:337] Failed RPC negotiation. Trace:
0809 12:11:10.385856 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.222.2:48585 (local address 127.0.222.62:44607)
0809 12:11:10.386112 (+ 256us) server_negotiation.cc:206] Beginning negotiation
0809 12:11:10.386122 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0809 12:11:10.386524 (+ 402us) server_negotiation.cc:407] Connection header received
0809 12:11:10.386725 (+ 201us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 12:11:10.386735 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 12:11:10.386799 (+ 64us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 12:11:10.386913 (+ 114us) server_negotiation.cc:226] Negotiated authn=SASL
0809 12:11:10.387809 (+ 896us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 12:11:10.388966 (+ 1157us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 12:11:10.390654 (+ 1688us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 12:11:10.391131 (+ 477us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 12:11:10.391230 (+ 99us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 12:11:10.395519 (+ 4289us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.222.2:48585: BlockingRecv error: recv got EOF from 127.0.222.2:48585 (error 108)
Metrics: {"server-negotiator.queue_time_us":66}
Aug 09 12:11:10 dist-test-slave-2w3w krb5kdc[9807](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.222.3@KRBTEST.COM for oryx/127.0.222.62@KRBTEST.COM, Server not found in Kerberos database
Aug 09 12:11:10 dist-test-slave-2w3w krb5kdc[9807](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.222.3@KRBTEST.COM for oryx/127.0.222.62@KRBTEST.COM, Server not found in Kerberos database
I20260809 12:11:10.867254 10943 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 12:11:10.858400 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.222.62:44607 (local address 127.0.222.3:59587)
0809 12:11:10.859094 (+ 694us) negotiation.cc:107] Waiting for socket to connect
0809 12:11:10.859115 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0809 12:11:10.859232 (+ 117us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 12:11:10.859690 (+ 458us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 12:11:10.859696 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 12:11:10.860275 (+ 579us) client_negotiation.cc:189] Negotiated authn=SASL
0809 12:11:10.860549 (+ 274us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 12:11:10.860568 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 12:11:10.861622 (+ 1054us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 12:11:10.861626 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 12:11:10.862622 (+ 996us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 12:11:10.862636 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 12:11:10.863411 (+ 775us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 12:11:10.863416 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 12:11:10.863534 (+ 118us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 12:11:10.864154 (+ 620us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0809 12:11:10.864171 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0809 12:11:10.867014 (+ 2843us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.222.62:44607: Server oryx/127.0.222.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":519,"thread_start_us":405,"threads_started":1}
W20260809 12:11:10.867503 10914 negotiation.cc:337] Failed RPC negotiation. Trace:
0809 12:11:10.858451 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.222.3:59587 (local address 127.0.222.62:44607)
0809 12:11:10.858631 (+ 180us) server_negotiation.cc:206] Beginning negotiation
0809 12:11:10.858636 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0809 12:11:10.859180 (+ 544us) server_negotiation.cc:407] Connection header received
0809 12:11:10.859352 (+ 172us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 12:11:10.859357 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 12:11:10.859465 (+ 108us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 12:11:10.859596 (+ 131us) server_negotiation.cc:226] Negotiated authn=SASL
0809 12:11:10.860700 (+ 1104us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 12:11:10.861488 (+ 788us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 12:11:10.862754 (+ 1266us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 12:11:10.863239 (+ 485us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 12:11:10.863336 (+ 97us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 12:11:10.867257 (+ 3921us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.222.3:59587: BlockingRecv error: recv got EOF from 127.0.222.3:59587 (error 108)
Metrics: {"server-negotiator.queue_time_us":60}
Aug 09 12:11:10 dist-test-slave-2w3w krb5kdc[9807](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.222.1@KRBTEST.COM for oryx/127.0.222.62@KRBTEST.COM, Server not found in Kerberos database
Aug 09 12:11:10 dist-test-slave-2w3w krb5kdc[9807](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.222.1@KRBTEST.COM for oryx/127.0.222.62@KRBTEST.COM, Server not found in Kerberos database
I20260809 12:11:10.928401 10944 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 12:11:10.919656 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.222.62:44607 (local address 127.0.222.1:34981)
0809 12:11:10.920287 (+ 631us) negotiation.cc:107] Waiting for socket to connect
0809 12:11:10.920316 (+ 29us) client_negotiation.cc:174] Beginning negotiation
0809 12:11:10.920428 (+ 112us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 12:11:10.920809 (+ 381us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 12:11:10.920814 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 12:11:10.921227 (+ 413us) client_negotiation.cc:189] Negotiated authn=SASL
0809 12:11:10.921393 (+ 166us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 12:11:10.921402 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 12:11:10.922500 (+ 1098us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 12:11:10.922510 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 12:11:10.923509 (+ 999us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 12:11:10.923522 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 12:11:10.924257 (+ 735us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 12:11:10.924263 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 12:11:10.924446 (+ 183us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 12:11:10.925012 (+ 566us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0809 12:11:10.925046 (+ 34us) client_negotiation.cc:656] calling sasl_client_start()
0809 12:11:10.928031 (+ 2985us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.222.62:44607: Server oryx/127.0.222.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":519,"thread_start_us":338,"threads_started":1}
W20260809 12:11:10.928506 10914 negotiation.cc:337] Failed RPC negotiation. Trace:
0809 12:11:10.919717 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.222.1:34981 (local address 127.0.222.62:44607)
0809 12:11:10.919897 (+ 180us) server_negotiation.cc:206] Beginning negotiation
0809 12:11:10.919902 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0809 12:11:10.920388 (+ 486us) server_negotiation.cc:407] Connection header received
0809 12:11:10.920566 (+ 178us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 12:11:10.920572 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 12:11:10.920650 (+ 78us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 12:11:10.920756 (+ 106us) server_negotiation.cc:226] Negotiated authn=SASL
0809 12:11:10.921536 (+ 780us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 12:11:10.922344 (+ 808us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 12:11:10.923654 (+ 1310us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 12:11:10.924094 (+ 440us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 12:11:10.924233 (+ 139us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 12:11:10.928252 (+ 4019us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.222.1:34981: BlockingRecv error: recv got EOF from 127.0.222.1:34981 (error 108)
Metrics: {"server-negotiator.queue_time_us":44}
Aug 09 12:11:11 dist-test-slave-2w3w krb5kdc[9807](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.222.2@KRBTEST.COM for oryx/127.0.222.62@KRBTEST.COM, Server not found in Kerberos database
Aug 09 12:11:11 dist-test-slave-2w3w krb5kdc[9807](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.222.2@KRBTEST.COM for oryx/127.0.222.62@KRBTEST.COM, Server not found in Kerberos database
I20260809 12:11:11.413076 10945 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 12:11:11.396547 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.222.62:44607 (local address 127.0.222.2:39677)
0809 12:11:11.397111 (+ 564us) negotiation.cc:107] Waiting for socket to connect
0809 12:11:11.397127 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0809 12:11:11.397407 (+ 280us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 12:11:11.397760 (+ 353us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 12:11:11.397764 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 12:11:11.398244 (+ 480us) client_negotiation.cc:189] Negotiated authn=SASL
0809 12:11:11.398420 (+ 176us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 12:11:11.398429 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 12:11:11.399626 (+ 1197us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 12:11:11.399632 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 12:11:11.400593 (+ 961us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 12:11:11.400605 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 12:11:11.401352 (+ 747us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 12:11:11.401359 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 12:11:11.401541 (+ 182us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 12:11:11.409711 (+ 8170us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0809 12:11:11.409732 (+ 21us) client_negotiation.cc:656] calling sasl_client_start()
0809 12:11:11.412824 (+ 3092us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.222.62:44607: Server oryx/127.0.222.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":470,"thread_start_us":292,"threads_started":1}
W20260809 12:11:11.413425 10914 negotiation.cc:337] Failed RPC negotiation. Trace:
0809 12:11:11.396630 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.222.2:39677 (local address 127.0.222.62:44607)
0809 12:11:11.396882 (+ 252us) server_negotiation.cc:206] Beginning negotiation
0809 12:11:11.396890 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0809 12:11:11.397308 (+ 418us) server_negotiation.cc:407] Connection header received
0809 12:11:11.397551 (+ 243us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 12:11:11.397560 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 12:11:11.397621 (+ 61us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 12:11:11.397731 (+ 110us) server_negotiation.cc:226] Negotiated authn=SASL
0809 12:11:11.398573 (+ 842us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 12:11:11.399492 (+ 919us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 12:11:11.400748 (+ 1256us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 12:11:11.401175 (+ 427us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 12:11:11.401312 (+ 137us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 12:11:11.413106 (+ 11794us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.222.2:39677: BlockingRecv error: recv got EOF from 127.0.222.2:39677 (error 108)
Metrics: {"server-negotiator.queue_time_us":92}
Aug 09 12:11:11 dist-test-slave-2w3w krb5kdc[9807](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.222.3@KRBTEST.COM for oryx/127.0.222.62@KRBTEST.COM, Server not found in Kerberos database
Aug 09 12:11:11 dist-test-slave-2w3w krb5kdc[9807](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.222.3@KRBTEST.COM for oryx/127.0.222.62@KRBTEST.COM, Server not found in Kerberos database
I20260809 12:11:11.877420 10946 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 12:11:11.868250 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.222.62:44607 (local address 127.0.222.3:32903)
0809 12:11:11.868926 (+ 676us) negotiation.cc:107] Waiting for socket to connect
0809 12:11:11.868947 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0809 12:11:11.869127 (+ 180us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 12:11:11.869493 (+ 366us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 12:11:11.869498 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 12:11:11.869943 (+ 445us) client_negotiation.cc:189] Negotiated authn=SASL
0809 12:11:11.870190 (+ 247us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 12:11:11.870201 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 12:11:11.871289 (+ 1088us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 12:11:11.871296 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 12:11:11.872325 (+ 1029us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 12:11:11.872338 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 12:11:11.873082 (+ 744us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 12:11:11.873088 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 12:11:11.873288 (+ 200us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 12:11:11.873955 (+ 667us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0809 12:11:11.873973 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0809 12:11:11.877022 (+ 3049us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.222.62:44607: Server oryx/127.0.222.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":582,"thread_start_us":382,"threads_started":1}
W20260809 12:11:11.877614 10914 negotiation.cc:337] Failed RPC negotiation. Trace:
0809 12:11:11.868441 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.222.3:32903 (local address 127.0.222.62:44607)
0809 12:11:11.868708 (+ 267us) server_negotiation.cc:206] Beginning negotiation
0809 12:11:11.868717 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0809 12:11:11.869051 (+ 334us) server_negotiation.cc:407] Connection header received
0809 12:11:11.869262 (+ 211us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 12:11:11.869269 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 12:11:11.869337 (+ 68us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 12:11:11.869448 (+ 111us) server_negotiation.cc:226] Negotiated authn=SASL
0809 12:11:11.870342 (+ 894us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 12:11:11.871144 (+ 802us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 12:11:11.872469 (+ 1325us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 12:11:11.872905 (+ 436us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 12:11:11.873066 (+ 161us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 12:11:11.877368 (+ 4302us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.222.3:32903: BlockingRecv error: recv got EOF from 127.0.222.3:32903 (error 108)
Metrics: {"server-negotiator.queue_time_us":92}
Aug 09 12:11:11 dist-test-slave-2w3w krb5kdc[9807](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.222.1@KRBTEST.COM for oryx/127.0.222.62@KRBTEST.COM, Server not found in Kerberos database
Aug 09 12:11:11 dist-test-slave-2w3w krb5kdc[9807](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.222.1@KRBTEST.COM for oryx/127.0.222.62@KRBTEST.COM, Server not found in Kerberos database
I20260809 12:11:11.937685 10948 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 12:11:11.929443 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.222.62:44607 (local address 127.0.222.1:49399)
0809 12:11:11.930015 (+ 572us) negotiation.cc:107] Waiting for socket to connect
0809 12:11:11.930031 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0809 12:11:11.930144 (+ 113us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 12:11:11.930454 (+ 310us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 12:11:11.930457 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 12:11:11.930846 (+ 389us) client_negotiation.cc:189] Negotiated authn=SASL
0809 12:11:11.931064 (+ 218us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 12:11:11.931075 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 12:11:11.932079 (+ 1004us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 12:11:11.932083 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 12:11:11.933015 (+ 932us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 12:11:11.933026 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 12:11:11.933717 (+ 691us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 12:11:11.933721 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 12:11:11.933861 (+ 140us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 12:11:11.934436 (+ 575us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0809 12:11:11.934454 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0809 12:11:11.937382 (+ 2928us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.222.62:44607: Server oryx/127.0.222.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":468,"thread_start_us":314,"threads_started":1}
W20260809 12:11:11.937994 10914 negotiation.cc:337] Failed RPC negotiation. Trace:
0809 12:11:11.929523 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.222.1:49399 (local address 127.0.222.62:44607)
0809 12:11:11.929722 (+ 199us) server_negotiation.cc:206] Beginning negotiation
0809 12:11:11.929727 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0809 12:11:11.930109 (+ 382us) server_negotiation.cc:407] Connection header received
0809 12:11:11.930253 (+ 144us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 12:11:11.930256 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 12:11:11.930317 (+ 61us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 12:11:11.930397 (+ 80us) server_negotiation.cc:226] Negotiated authn=SASL
0809 12:11:11.931200 (+ 803us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 12:11:11.931946 (+ 746us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 12:11:11.933159 (+ 1213us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 12:11:11.933579 (+ 420us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 12:11:11.933671 (+ 92us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 12:11:11.937665 (+ 3994us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.222.1:49399: BlockingRecv error: recv got EOF from 127.0.222.1:49399 (error 108)
Metrics: {"server-negotiator.queue_time_us":80}
Aug 09 12:11:12 dist-test-slave-2w3w krb5kdc[9807](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.222.2@KRBTEST.COM for oryx/127.0.222.62@KRBTEST.COM, Server not found in Kerberos database
Aug 09 12:11:12 dist-test-slave-2w3w krb5kdc[9807](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.222.2@KRBTEST.COM for oryx/127.0.222.62@KRBTEST.COM, Server not found in Kerberos database
I20260809 12:11:12.423395 10949 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 12:11:12.414055 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.222.62:44607 (local address 127.0.222.2:44933)
0809 12:11:12.414573 (+ 518us) negotiation.cc:107] Waiting for socket to connect
0809 12:11:12.414589 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0809 12:11:12.414772 (+ 183us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 12:11:12.415164 (+ 392us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 12:11:12.415169 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 12:11:12.415613 (+ 444us) client_negotiation.cc:189] Negotiated authn=SASL
0809 12:11:12.415755 (+ 142us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 12:11:12.415764 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 12:11:12.416818 (+ 1054us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 12:11:12.416825 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 12:11:12.417772 (+ 947us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 12:11:12.417783 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 12:11:12.418530 (+ 747us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 12:11:12.418537 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 12:11:12.418695 (+ 158us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 12:11:12.419436 (+ 741us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0809 12:11:12.419454 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0809 12:11:12.423034 (+ 3580us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.222.62:44607: Server oryx/127.0.222.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":422,"thread_start_us":295,"threads_started":1}
W20260809 12:11:12.423590 10914 negotiation.cc:337] Failed RPC negotiation. Trace:
0809 12:11:12.414127 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.222.2:44933 (local address 127.0.222.62:44607)
0809 12:11:12.414364 (+ 237us) server_negotiation.cc:206] Beginning negotiation
0809 12:11:12.414374 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0809 12:11:12.414688 (+ 314us) server_negotiation.cc:407] Connection header received
0809 12:11:12.414910 (+ 222us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 12:11:12.414917 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 12:11:12.415015 (+ 98us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 12:11:12.415128 (+ 113us) server_negotiation.cc:226] Negotiated authn=SASL
0809 12:11:12.415906 (+ 778us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 12:11:12.416657 (+ 751us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 12:11:12.417928 (+ 1271us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 12:11:12.418355 (+ 427us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 12:11:12.418493 (+ 138us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 12:11:12.423270 (+ 4777us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.222.2:44933: BlockingRecv error: recv got EOF from 127.0.222.2:44933 (error 108)
Metrics: {"server-negotiator.queue_time_us":64}
Aug 09 12:11:12 dist-test-slave-2w3w krb5kdc[9807](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.222.3@KRBTEST.COM for oryx/127.0.222.62@KRBTEST.COM, Server not found in Kerberos database
Aug 09 12:11:12 dist-test-slave-2w3w krb5kdc[9807](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.222.3@KRBTEST.COM for oryx/127.0.222.62@KRBTEST.COM, Server not found in Kerberos database
I20260809 12:11:12.887497 10950 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 12:11:12.878375 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.222.62:44607 (local address 127.0.222.3:37545)
0809 12:11:12.878946 (+ 571us) negotiation.cc:107] Waiting for socket to connect
0809 12:11:12.879034 (+ 88us) client_negotiation.cc:174] Beginning negotiation
0809 12:11:12.879201 (+ 167us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 12:11:12.879576 (+ 375us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 12:11:12.879581 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 12:11:12.880005 (+ 424us) client_negotiation.cc:189] Negotiated authn=SASL
0809 12:11:12.880199 (+ 194us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 12:11:12.880215 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 12:11:12.881526 (+ 1311us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 12:11:12.881532 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 12:11:12.882504 (+ 972us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 12:11:12.882517 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 12:11:12.883346 (+ 829us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 12:11:12.883352 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 12:11:12.883543 (+ 191us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 12:11:12.884225 (+ 682us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0809 12:11:12.884244 (+ 19us) client_negotiation.cc:656] calling sasl_client_start()
0809 12:11:12.887144 (+ 2900us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.222.62:44607: Server oryx/127.0.222.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":451,"thread_start_us":318,"threads_started":1}
W20260809 12:11:12.887651 10914 negotiation.cc:337] Failed RPC negotiation. Trace:
0809 12:11:12.878632 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.222.3:37545 (local address 127.0.222.62:44607)
0809 12:11:12.878888 (+ 256us) server_negotiation.cc:206] Beginning negotiation
0809 12:11:12.878895 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0809 12:11:12.879125 (+ 230us) server_negotiation.cc:407] Connection header received
0809 12:11:12.879334 (+ 209us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 12:11:12.879341 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 12:11:12.879410 (+ 69us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 12:11:12.879524 (+ 114us) server_negotiation.cc:226] Negotiated authn=SASL
0809 12:11:12.880388 (+ 864us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 12:11:12.881369 (+ 981us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 12:11:12.882662 (+ 1293us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 12:11:12.883176 (+ 514us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 12:11:12.883318 (+ 142us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 12:11:12.887360 (+ 4042us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.222.3:37545: BlockingRecv error: recv got EOF from 127.0.222.3:37545 (error 108)
Metrics: {"server-negotiator.queue_time_us":101}
Aug 09 12:11:12 dist-test-slave-2w3w krb5kdc[9807](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.222.1@KRBTEST.COM for oryx/127.0.222.62@KRBTEST.COM, Server not found in Kerberos database
Aug 09 12:11:12 dist-test-slave-2w3w krb5kdc[9807](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.222.1@KRBTEST.COM for oryx/127.0.222.62@KRBTEST.COM, Server not found in Kerberos database
I20260809 12:11:12.948266 10951 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 12:11:12.938854 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.222.62:44607 (local address 127.0.222.1:57577)
0809 12:11:12.939510 (+ 656us) negotiation.cc:107] Waiting for socket to connect
0809 12:11:12.939526 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0809 12:11:12.939689 (+ 163us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 12:11:12.940043 (+ 354us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 12:11:12.940047 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 12:11:12.940643 (+ 596us) client_negotiation.cc:189] Negotiated authn=SASL
0809 12:11:12.940830 (+ 187us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 12:11:12.940839 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 12:11:12.942056 (+ 1217us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 12:11:12.942063 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 12:11:12.943064 (+ 1001us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 12:11:12.943077 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 12:11:12.943817 (+ 740us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 12:11:12.943823 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 12:11:12.944033 (+ 210us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 12:11:12.944717 (+ 684us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0809 12:11:12.944735 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0809 12:11:12.947909 (+ 3174us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.222.62:44607: Server oryx/127.0.222.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":537,"thread_start_us":349,"threads_started":1}
W20260809 12:11:12.948352 10914 negotiation.cc:337] Failed RPC negotiation. Trace:
0809 12:11:12.938925 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.222.1:57577 (local address 127.0.222.62:44607)
0809 12:11:12.939206 (+ 281us) server_negotiation.cc:206] Beginning negotiation
0809 12:11:12.939214 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0809 12:11:12.939613 (+ 399us) server_negotiation.cc:407] Connection header received
0809 12:11:12.939826 (+ 213us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 12:11:12.939833 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 12:11:12.939903 (+ 70us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 12:11:12.940019 (+ 116us) server_negotiation.cc:226] Negotiated authn=SASL
0809 12:11:12.940986 (+ 967us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 12:11:12.941912 (+ 926us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 12:11:12.943205 (+ 1293us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 12:11:12.943636 (+ 431us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 12:11:12.943783 (+ 147us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 12:11:12.948112 (+ 4329us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.222.1:57577: BlockingRecv error: recv got EOF from 127.0.222.1:57577 (error 108)
Metrics: {"server-negotiator.queue_time_us":98}
Aug 09 12:11:13 dist-test-slave-2w3w krb5kdc[9807](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.222.2@KRBTEST.COM for oryx/127.0.222.62@KRBTEST.COM, Server not found in Kerberos database
Aug 09 12:11:13 dist-test-slave-2w3w krb5kdc[9807](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.222.2@KRBTEST.COM for oryx/127.0.222.62@KRBTEST.COM, Server not found in Kerberos database
I20260809 12:11:13.433961 10952 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 12:11:13.424593 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.222.62:44607 (local address 127.0.222.2:46593)
0809 12:11:13.425407 (+ 814us) negotiation.cc:107] Waiting for socket to connect
0809 12:11:13.425432 (+ 25us) client_negotiation.cc:174] Beginning negotiation
0809 12:11:13.425634 (+ 202us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 12:11:13.426225 (+ 591us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 12:11:13.426232 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 12:11:13.426813 (+ 581us) client_negotiation.cc:189] Negotiated authn=SASL
0809 12:11:13.427116 (+ 303us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 12:11:13.427133 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 12:11:13.428425 (+ 1292us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 12:11:13.428429 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 12:11:13.429323 (+ 894us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 12:11:13.429334 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 12:11:13.430018 (+ 684us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 12:11:13.430022 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 12:11:13.430146 (+ 124us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 12:11:13.430827 (+ 681us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0809 12:11:13.430846 (+ 19us) client_negotiation.cc:656] calling sasl_client_start()
0809 12:11:13.433712 (+ 2866us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.222.62:44607: Server oryx/127.0.222.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":674,"thread_start_us":552,"threads_started":1}
W20260809 12:11:13.434155 10914 negotiation.cc:337] Failed RPC negotiation. Trace:
0809 12:11:13.424652 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.222.2:46593 (local address 127.0.222.62:44607)
0809 12:11:13.424926 (+ 274us) server_negotiation.cc:206] Beginning negotiation
0809 12:11:13.424936 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0809 12:11:13.425518 (+ 582us) server_negotiation.cc:407] Connection header received
0809 12:11:13.425786 (+ 268us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 12:11:13.425795 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 12:11:13.425874 (+ 79us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 12:11:13.425946 (+ 72us) server_negotiation.cc:226] Negotiated authn=SASL
0809 12:11:13.427351 (+ 1405us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 12:11:13.428286 (+ 935us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 12:11:13.429453 (+ 1167us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 12:11:13.429872 (+ 419us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 12:11:13.429970 (+ 98us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 12:11:13.433918 (+ 3948us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.222.2:46593: BlockingRecv error: recv got EOF from 127.0.222.2:46593 (error 108)
Metrics: {"server-negotiator.queue_time_us":97}
Aug 09 12:11:13 dist-test-slave-2w3w krb5kdc[9807](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.222.3@KRBTEST.COM for oryx/127.0.222.62@KRBTEST.COM, Server not found in Kerberos database
Aug 09 12:11:13 dist-test-slave-2w3w krb5kdc[9807](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.222.3@KRBTEST.COM for oryx/127.0.222.62@KRBTEST.COM, Server not found in Kerberos database
I20260809 12:11:13.897552 10953 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 12:11:13.888537 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.222.62:44607 (local address 127.0.222.3:37431)
0809 12:11:13.889057 (+ 520us) negotiation.cc:107] Waiting for socket to connect
0809 12:11:13.889078 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0809 12:11:13.889243 (+ 165us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 12:11:13.889600 (+ 357us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 12:11:13.889604 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 12:11:13.890051 (+ 447us) client_negotiation.cc:189] Negotiated authn=SASL
0809 12:11:13.890231 (+ 180us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 12:11:13.890240 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 12:11:13.891347 (+ 1107us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 12:11:13.891356 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 12:11:13.892236 (+ 880us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 12:11:13.892247 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 12:11:13.892984 (+ 737us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 12:11:13.892990 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 12:11:13.893174 (+ 184us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 12:11:13.893785 (+ 611us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0809 12:11:13.893804 (+ 19us) client_negotiation.cc:656] calling sasl_client_start()
0809 12:11:13.897232 (+ 3428us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.222.62:44607: Server oryx/127.0.222.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":414,"thread_start_us":275,"threads_started":1}
W20260809 12:11:13.897717 10914 negotiation.cc:337] Failed RPC negotiation. Trace:
0809 12:11:13.888607 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.222.3:37431 (local address 127.0.222.62:44607)
0809 12:11:13.888844 (+ 237us) server_negotiation.cc:206] Beginning negotiation
0809 12:11:13.888853 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0809 12:11:13.889170 (+ 317us) server_negotiation.cc:407] Connection header received
0809 12:11:13.889386 (+ 216us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 12:11:13.889391 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 12:11:13.889460 (+ 69us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 12:11:13.889568 (+ 108us) server_negotiation.cc:226] Negotiated authn=SASL
0809 12:11:13.890391 (+ 823us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 12:11:13.891204 (+ 813us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 12:11:13.892390 (+ 1186us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 12:11:13.892807 (+ 417us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 12:11:13.892965 (+ 158us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 12:11:13.897454 (+ 4489us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.222.3:37431: BlockingRecv error: recv got EOF from 127.0.222.3:37431 (error 108)
Metrics: {"server-negotiator.queue_time_us":65}
Aug 09 12:11:13 dist-test-slave-2w3w krb5kdc[9807](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.222.1@KRBTEST.COM for oryx/127.0.222.62@KRBTEST.COM, Server not found in Kerberos database
Aug 09 12:11:13 dist-test-slave-2w3w krb5kdc[9807](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.222.1@KRBTEST.COM for oryx/127.0.222.62@KRBTEST.COM, Server not found in Kerberos database
I20260809 12:11:13.957901 10954 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 12:11:13.949193 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.222.62:44607 (local address 127.0.222.1:34727)
0809 12:11:13.949795 (+ 602us) negotiation.cc:107] Waiting for socket to connect
0809 12:11:13.949820 (+ 25us) client_negotiation.cc:174] Beginning negotiation
0809 12:11:13.949937 (+ 117us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 12:11:13.950263 (+ 326us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 12:11:13.950266 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 12:11:13.950721 (+ 455us) client_negotiation.cc:189] Negotiated authn=SASL
0809 12:11:13.950900 (+ 179us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 12:11:13.950910 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 12:11:13.952015 (+ 1105us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 12:11:13.952019 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 12:11:13.952949 (+ 930us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 12:11:13.952961 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 12:11:13.953702 (+ 741us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 12:11:13.953706 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 12:11:13.953838 (+ 132us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 12:11:13.954448 (+ 610us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0809 12:11:13.954465 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0809 12:11:13.957552 (+ 3087us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.222.62:44607: Server oryx/127.0.222.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":490,"thread_start_us":345,"threads_started":1}
W20260809 12:11:13.958058 10914 negotiation.cc:337] Failed RPC negotiation. Trace:
0809 12:11:13.949278 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.222.1:34727 (local address 127.0.222.62:44607)
0809 12:11:13.949484 (+ 206us) server_negotiation.cc:206] Beginning negotiation
0809 12:11:13.949488 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0809 12:11:13.949909 (+ 421us) server_negotiation.cc:407] Connection header received
0809 12:11:13.950049 (+ 140us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 12:11:13.950052 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 12:11:13.950115 (+ 63us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 12:11:13.950206 (+ 91us) server_negotiation.cc:226] Negotiated authn=SASL
0809 12:11:13.951081 (+ 875us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 12:11:13.951861 (+ 780us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 12:11:13.953081 (+ 1220us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 12:11:13.953565 (+ 484us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 12:11:13.953662 (+ 97us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 12:11:13.957772 (+ 4110us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.222.1:34727: BlockingRecv error: recv got EOF from 127.0.222.1:34727 (error 108)
Metrics: {"server-negotiator.queue_time_us":63}
Aug 09 12:11:14 dist-test-slave-2w3w krb5kdc[9807](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.222.2@KRBTEST.COM for oryx/127.0.222.62@KRBTEST.COM, Server not found in Kerberos database
Aug 09 12:11:14 dist-test-slave-2w3w krb5kdc[9807](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.222.2@KRBTEST.COM for oryx/127.0.222.62@KRBTEST.COM, Server not found in Kerberos database
I20260809 12:11:14.443972 10955 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 12:11:14.434906 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.222.62:44607 (local address 127.0.222.2:42251)
0809 12:11:14.435560 (+ 654us) negotiation.cc:107] Waiting for socket to connect
0809 12:11:14.435581 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0809 12:11:14.435710 (+ 129us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 12:11:14.436023 (+ 313us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 12:11:14.436027 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 12:11:14.436411 (+ 384us) client_negotiation.cc:189] Negotiated authn=SASL
0809 12:11:14.436569 (+ 158us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 12:11:14.436577 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 12:11:14.437830 (+ 1253us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 12:11:14.437835 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 12:11:14.438773 (+ 938us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 12:11:14.438785 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 12:11:14.439600 (+ 815us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 12:11:14.439604 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 12:11:14.439739 (+ 135us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 12:11:14.440438 (+ 699us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0809 12:11:14.440457 (+ 19us) client_negotiation.cc:656] calling sasl_client_start()
0809 12:11:14.443604 (+ 3147us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.222.62:44607: Server oryx/127.0.222.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":530,"thread_start_us":391,"threads_started":1}
W20260809 12:11:14.444097 10914 negotiation.cc:337] Failed RPC negotiation. Trace:
0809 12:11:14.435041 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.222.2:42251 (local address 127.0.222.62:44607)
0809 12:11:14.435227 (+ 186us) server_negotiation.cc:206] Beginning negotiation
0809 12:11:14.435232 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0809 12:11:14.435672 (+ 440us) server_negotiation.cc:407] Connection header received
0809 12:11:14.435807 (+ 135us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 12:11:14.435811 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 12:11:14.435880 (+ 69us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 12:11:14.435962 (+ 82us) server_negotiation.cc:226] Negotiated authn=SASL
0809 12:11:14.436691 (+ 729us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 12:11:14.437478 (+ 787us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 12:11:14.438921 (+ 1443us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 12:11:14.439464 (+ 543us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 12:11:14.439567 (+ 103us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 12:11:14.443820 (+ 4253us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.222.2:42251: BlockingRecv error: recv got EOF from 127.0.222.2:42251 (error 108)
Metrics: {"server-negotiator.queue_time_us":61}
Aug 09 12:11:14 dist-test-slave-2w3w krb5kdc[9807](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.222.3@KRBTEST.COM for oryx/127.0.222.62@KRBTEST.COM, Server not found in Kerberos database
Aug 09 12:11:14 dist-test-slave-2w3w krb5kdc[9807](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.222.3@KRBTEST.COM for oryx/127.0.222.62@KRBTEST.COM, Server not found in Kerberos database
I20260809 12:11:14.907337 10956 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 12:11:14.898608 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.222.62:44607 (local address 127.0.222.3:50205)
0809 12:11:14.899153 (+ 545us) negotiation.cc:107] Waiting for socket to connect
0809 12:11:14.899175 (+ 22us) client_negotiation.cc:174] Beginning negotiation
0809 12:11:14.899287 (+ 112us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 12:11:14.899626 (+ 339us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 12:11:14.899629 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 12:11:14.900033 (+ 404us) client_negotiation.cc:189] Negotiated authn=SASL
0809 12:11:14.900189 (+ 156us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 12:11:14.900199 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 12:11:14.901173 (+ 974us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 12:11:14.901177 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 12:11:14.902127 (+ 950us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 12:11:14.902140 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 12:11:14.902885 (+ 745us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 12:11:14.902893 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 12:11:14.903139 (+ 246us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 12:11:14.904014 (+ 875us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0809 12:11:14.904032 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0809 12:11:14.907088 (+ 3056us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.222.62:44607: Server oryx/127.0.222.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":449,"thread_start_us":307,"threads_started":1}
W20260809 12:11:14.907595 10914 negotiation.cc:337] Failed RPC negotiation. Trace:
0809 12:11:14.898682 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.222.3:50205 (local address 127.0.222.62:44607)
0809 12:11:14.898928 (+ 246us) server_negotiation.cc:206] Beginning negotiation
0809 12:11:14.898935 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0809 12:11:14.899252 (+ 317us) server_negotiation.cc:407] Connection header received
0809 12:11:14.899402 (+ 150us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 12:11:14.899405 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 12:11:14.899492 (+ 87us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 12:11:14.899574 (+ 82us) server_negotiation.cc:226] Negotiated authn=SASL
0809 12:11:14.900312 (+ 738us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 12:11:14.901046 (+ 734us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 12:11:14.902247 (+ 1201us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 12:11:14.902716 (+ 469us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 12:11:14.902849 (+ 133us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 12:11:14.907330 (+ 4481us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.222.3:50205: BlockingRecv error: recv got EOF from 127.0.222.3:50205 (error 108)
Metrics: {"server-negotiator.queue_time_us":72}
Aug 09 12:11:14 dist-test-slave-2w3w krb5kdc[9807](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.222.1@KRBTEST.COM for oryx/127.0.222.62@KRBTEST.COM, Server not found in Kerberos database
Aug 09 12:11:14 dist-test-slave-2w3w krb5kdc[9807](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.222.1@KRBTEST.COM for oryx/127.0.222.62@KRBTEST.COM, Server not found in Kerberos database
I20260809 12:11:14.968104 10957 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 12:11:14.959078 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.222.62:44607 (local address 127.0.222.1:59867)
0809 12:11:14.959677 (+ 599us) negotiation.cc:107] Waiting for socket to connect
0809 12:11:14.959694 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0809 12:11:14.959862 (+ 168us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 12:11:14.960204 (+ 342us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 12:11:14.960209 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 12:11:14.960636 (+ 427us) client_negotiation.cc:189] Negotiated authn=SASL
0809 12:11:14.960860 (+ 224us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 12:11:14.960870 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 12:11:14.961968 (+ 1098us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 12:11:14.961984 (+ 16us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 12:11:14.963027 (+ 1043us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 12:11:14.963048 (+ 21us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 12:11:14.963966 (+ 918us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 12:11:14.963972 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 12:11:14.964162 (+ 190us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 12:11:14.964808 (+ 646us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0809 12:11:14.964848 (+ 40us) client_negotiation.cc:656] calling sasl_client_start()
0809 12:11:14.967859 (+ 3011us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.222.62:44607: Server oryx/127.0.222.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":469,"thread_start_us":277,"threads_started":1}
W20260809 12:11:14.968312 10914 negotiation.cc:337] Failed RPC negotiation. Trace:
0809 12:11:14.959344 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.222.1:59867 (local address 127.0.222.62:44607)
0809 12:11:14.959620 (+ 276us) server_negotiation.cc:206] Beginning negotiation
0809 12:11:14.959628 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0809 12:11:14.959781 (+ 153us) server_negotiation.cc:407] Connection header received
0809 12:11:14.959993 (+ 212us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 12:11:14.959999 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 12:11:14.960069 (+ 70us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 12:11:14.960172 (+ 103us) server_negotiation.cc:226] Negotiated authn=SASL
0809 12:11:14.961001 (+ 829us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 12:11:14.961799 (+ 798us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 12:11:14.963178 (+ 1379us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 12:11:14.963773 (+ 595us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 12:11:14.963918 (+ 145us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 12:11:14.968054 (+ 4136us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.222.1:59867: BlockingRecv error: recv got EOF from 127.0.222.1:59867 (error 108)
Metrics: {"server-negotiator.queue_time_us":78}
Aug 09 12:11:15 dist-test-slave-2w3w krb5kdc[9807](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.222.2@KRBTEST.COM for oryx/127.0.222.62@KRBTEST.COM, Server not found in Kerberos database
Aug 09 12:11:15 dist-test-slave-2w3w krb5kdc[9807](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.222.2@KRBTEST.COM for oryx/127.0.222.62@KRBTEST.COM, Server not found in Kerberos database
I20260809 12:11:15.454154 10959 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 12:11:15.445148 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.222.62:44607 (local address 127.0.222.2:45561)
0809 12:11:15.445758 (+ 610us) negotiation.cc:107] Waiting for socket to connect
0809 12:11:15.445775 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0809 12:11:15.445890 (+ 115us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 12:11:15.446248 (+ 358us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 12:11:15.446252 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 12:11:15.446800 (+ 548us) client_negotiation.cc:189] Negotiated authn=SASL
0809 12:11:15.447088 (+ 288us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 12:11:15.447119 (+ 31us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 12:11:15.448539 (+ 1420us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 12:11:15.448544 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 12:11:15.449472 (+ 928us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 12:11:15.449493 (+ 21us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 12:11:15.450251 (+ 758us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 12:11:15.450255 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 12:11:15.450374 (+ 119us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 12:11:15.451054 (+ 680us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0809 12:11:15.451083 (+ 29us) client_negotiation.cc:656] calling sasl_client_start()
0809 12:11:15.453900 (+ 2817us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.222.62:44607: Server oryx/127.0.222.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":489,"thread_start_us":329,"threads_started":1}
W20260809 12:11:15.454314 10914 negotiation.cc:337] Failed RPC negotiation. Trace:
0809 12:11:15.445345 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.222.2:45561 (local address 127.0.222.62:44607)
0809 12:11:15.445548 (+ 203us) server_negotiation.cc:206] Beginning negotiation
0809 12:11:15.445553 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0809 12:11:15.445848 (+ 295us) server_negotiation.cc:407] Connection header received
0809 12:11:15.445992 (+ 144us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 12:11:15.445996 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 12:11:15.446079 (+ 83us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 12:11:15.446164 (+ 85us) server_negotiation.cc:226] Negotiated authn=SASL
0809 12:11:15.447267 (+ 1103us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 12:11:15.448392 (+ 1125us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 12:11:15.449613 (+ 1221us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 12:11:15.450086 (+ 473us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 12:11:15.450184 (+ 98us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 12:11:15.454114 (+ 3930us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.222.2:45561: BlockingRecv error: recv got EOF from 127.0.222.2:45561 (error 108)
Metrics: {"server-negotiator.queue_time_us":70}
Aug 09 12:11:15 dist-test-slave-2w3w krb5kdc[9807](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.222.3@KRBTEST.COM for oryx/127.0.222.62@KRBTEST.COM, Server not found in Kerberos database
Aug 09 12:11:15 dist-test-slave-2w3w krb5kdc[9807](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.222.3@KRBTEST.COM for oryx/127.0.222.62@KRBTEST.COM, Server not found in Kerberos database
I20260809 12:11:15.917253 10960 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 12:11:15.908362 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.222.62:44607 (local address 127.0.222.3:50873)
0809 12:11:15.908909 (+ 547us) negotiation.cc:107] Waiting for socket to connect
0809 12:11:15.908928 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0809 12:11:15.909107 (+ 179us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 12:11:15.909458 (+ 351us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 12:11:15.909463 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 12:11:15.909926 (+ 463us) client_negotiation.cc:189] Negotiated authn=SASL
0809 12:11:15.910104 (+ 178us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 12:11:15.910113 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 12:11:15.911228 (+ 1115us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 12:11:15.911235 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 12:11:15.912326 (+ 1091us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 12:11:15.912339 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 12:11:15.913249 (+ 910us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 12:11:15.913256 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 12:11:15.913465 (+ 209us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 12:11:15.914117 (+ 652us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0809 12:11:15.914135 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0809 12:11:15.917017 (+ 2882us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.222.62:44607: Server oryx/127.0.222.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":451,"thread_start_us":280,"threads_started":1}
W20260809 12:11:15.917444 10914 negotiation.cc:337] Failed RPC negotiation. Trace:
0809 12:11:15.908457 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.222.3:50873 (local address 127.0.222.62:44607)
0809 12:11:15.908728 (+ 271us) server_negotiation.cc:206] Beginning negotiation
0809 12:11:15.908737 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0809 12:11:15.909030 (+ 293us) server_negotiation.cc:407] Connection header received
0809 12:11:15.909236 (+ 206us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 12:11:15.909241 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 12:11:15.909312 (+ 71us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 12:11:15.909418 (+ 106us) server_negotiation.cc:226] Negotiated authn=SASL
0809 12:11:15.910258 (+ 840us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 12:11:15.911086 (+ 828us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 12:11:15.912470 (+ 1384us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 12:11:15.913011 (+ 541us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 12:11:15.913172 (+ 161us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 12:11:15.917228 (+ 4056us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.222.3:50873: BlockingRecv error: recv got EOF from 127.0.222.3:50873 (error 108)
Metrics: {"server-negotiator.queue_time_us":95}
Aug 09 12:11:15 dist-test-slave-2w3w krb5kdc[9807](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.222.1@KRBTEST.COM for oryx/127.0.222.62@KRBTEST.COM, Server not found in Kerberos database
Aug 09 12:11:15 dist-test-slave-2w3w krb5kdc[9807](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.222.1@KRBTEST.COM for oryx/127.0.222.62@KRBTEST.COM, Server not found in Kerberos database
I20260809 12:11:15.978188 10961 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 12:11:15.969165 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.222.62:44607 (local address 127.0.222.1:33165)
0809 12:11:15.969761 (+ 596us) negotiation.cc:107] Waiting for socket to connect
0809 12:11:15.969777 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0809 12:11:15.969949 (+ 172us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 12:11:15.970317 (+ 368us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 12:11:15.970322 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 12:11:15.970748 (+ 426us) client_negotiation.cc:189] Negotiated authn=SASL
0809 12:11:15.970938 (+ 190us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 12:11:15.970947 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 12:11:15.972040 (+ 1093us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 12:11:15.972046 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 12:11:15.973318 (+ 1272us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 12:11:15.973334 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 12:11:15.974303 (+ 969us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 12:11:15.974308 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 12:11:15.974434 (+ 126us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 12:11:15.975059 (+ 625us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0809 12:11:15.975077 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0809 12:11:15.977925 (+ 2848us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.222.62:44607: Server oryx/127.0.222.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":471,"thread_start_us":336,"threads_started":1}
W20260809 12:11:15.978442 10914 negotiation.cc:337] Failed RPC negotiation. Trace:
0809 12:11:15.969336 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.222.1:33165 (local address 127.0.222.62:44607)
0809 12:11:15.969692 (+ 356us) server_negotiation.cc:206] Beginning negotiation
0809 12:11:15.969699 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0809 12:11:15.969865 (+ 166us) server_negotiation.cc:407] Connection header received
0809 12:11:15.970092 (+ 227us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 12:11:15.970098 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 12:11:15.970172 (+ 74us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 12:11:15.970281 (+ 109us) server_negotiation.cc:226] Negotiated authn=SASL
0809 12:11:15.971148 (+ 867us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 12:11:15.971878 (+ 730us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 12:11:15.973506 (+ 1628us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 12:11:15.974153 (+ 647us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 12:11:15.974256 (+ 103us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 12:11:15.978130 (+ 3874us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.222.1:33165: BlockingRecv error: recv got EOF from 127.0.222.1:33165 (error 108)
Metrics: {"server-negotiator.queue_time_us":166}
Aug 09 12:11:16 dist-test-slave-2w3w krb5kdc[9807](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.222.2@KRBTEST.COM for oryx/127.0.222.62@KRBTEST.COM, Server not found in Kerberos database
Aug 09 12:11:16 dist-test-slave-2w3w krb5kdc[9807](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.222.2@KRBTEST.COM for oryx/127.0.222.62@KRBTEST.COM, Server not found in Kerberos database
I20260809 12:11:16.468650 10962 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 12:11:16.455344 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.222.62:44607 (local address 127.0.222.2:55553)
0809 12:11:16.455848 (+ 504us) negotiation.cc:107] Waiting for socket to connect
0809 12:11:16.455864 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0809 12:11:16.456035 (+ 171us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 12:11:16.456413 (+ 378us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 12:11:16.456418 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 12:11:16.456828 (+ 410us) client_negotiation.cc:189] Negotiated authn=SASL
0809 12:11:16.457015 (+ 187us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 12:11:16.457024 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 12:11:16.458108 (+ 1084us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 12:11:16.458121 (+ 13us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 12:11:16.459121 (+ 1000us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 12:11:16.459133 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 12:11:16.459831 (+ 698us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 12:11:16.459837 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 12:11:16.460033 (+ 196us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 12:11:16.465182 (+ 5149us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0809 12:11:16.465209 (+ 27us) client_negotiation.cc:656] calling sasl_client_start()
0809 12:11:16.468262 (+ 3053us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.222.62:44607: Server oryx/127.0.222.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":394,"thread_start_us":257,"threads_started":1}
W20260809 12:11:16.468729 10914 negotiation.cc:337] Failed RPC negotiation. Trace:
0809 12:11:16.455415 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.222.2:55553 (local address 127.0.222.62:44607)
0809 12:11:16.455640 (+ 225us) server_negotiation.cc:206] Beginning negotiation
0809 12:11:16.455649 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0809 12:11:16.455957 (+ 308us) server_negotiation.cc:407] Connection header received
0809 12:11:16.456170 (+ 213us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 12:11:16.456175 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 12:11:16.456265 (+ 90us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 12:11:16.456374 (+ 109us) server_negotiation.cc:226] Negotiated authn=SASL
0809 12:11:16.457194 (+ 820us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 12:11:16.457940 (+ 746us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 12:11:16.459258 (+ 1318us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 12:11:16.459678 (+ 420us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 12:11:16.459815 (+ 137us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 12:11:16.468476 (+ 8661us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.222.2:55553: BlockingRecv error: recv got EOF from 127.0.222.2:55553 (error 108)
Metrics: {"server-negotiator.queue_time_us":58}
Aug 09 12:11:16 dist-test-slave-2w3w krb5kdc[9807](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.222.3@KRBTEST.COM for oryx/127.0.222.62@KRBTEST.COM, Server not found in Kerberos database
Aug 09 12:11:16 dist-test-slave-2w3w krb5kdc[9807](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.222.3@KRBTEST.COM for oryx/127.0.222.62@KRBTEST.COM, Server not found in Kerberos database
I20260809 12:11:16.927418 10963 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 12:11:16.918340 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.222.62:44607 (local address 127.0.222.3:44253)
0809 12:11:16.918878 (+ 538us) negotiation.cc:107] Waiting for socket to connect
0809 12:11:16.918900 (+ 22us) client_negotiation.cc:174] Beginning negotiation
0809 12:11:16.919032 (+ 132us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 12:11:16.919324 (+ 292us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 12:11:16.919327 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 12:11:16.919700 (+ 373us) client_negotiation.cc:189] Negotiated authn=SASL
0809 12:11:16.919903 (+ 203us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 12:11:16.919912 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 12:11:16.921389 (+ 1477us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 12:11:16.921393 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 12:11:16.922615 (+ 1222us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 12:11:16.922630 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 12:11:16.923455 (+ 825us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 12:11:16.923459 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 12:11:16.923575 (+ 116us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 12:11:16.924201 (+ 626us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0809 12:11:16.924218 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0809 12:11:16.927127 (+ 2909us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.222.62:44607: Server oryx/127.0.222.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":416,"thread_start_us":295,"threads_started":1}
W20260809 12:11:16.927598 10914 negotiation.cc:337] Failed RPC negotiation. Trace:
0809 12:11:16.918423 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.222.3:44253 (local address 127.0.222.62:44607)
0809 12:11:16.918598 (+ 175us) server_negotiation.cc:206] Beginning negotiation
0809 12:11:16.918602 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0809 12:11:16.918976 (+ 374us) server_negotiation.cc:407] Connection header received
0809 12:11:16.919129 (+ 153us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 12:11:16.919133 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 12:11:16.919193 (+ 60us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 12:11:16.919275 (+ 82us) server_negotiation.cc:226] Negotiated authn=SASL
0809 12:11:16.920063 (+ 788us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 12:11:16.921174 (+ 1111us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 12:11:16.922780 (+ 1606us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 12:11:16.923326 (+ 546us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 12:11:16.923419 (+ 93us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 12:11:16.927364 (+ 3945us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.222.3:44253: BlockingRecv error: recv got EOF from 127.0.222.3:44253 (error 108)
Metrics: {"server-negotiator.queue_time_us":58}
Aug 09 12:11:16 dist-test-slave-2w3w krb5kdc[9807](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.222.1@KRBTEST.COM for oryx/127.0.222.62@KRBTEST.COM, Server not found in Kerberos database
Aug 09 12:11:16 dist-test-slave-2w3w krb5kdc[9807](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.222.1@KRBTEST.COM for oryx/127.0.222.62@KRBTEST.COM, Server not found in Kerberos database
I20260809 12:11:16.988300 10964 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 12:11:16.979449 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.222.62:44607 (local address 127.0.222.1:42701)
0809 12:11:16.979958 (+ 509us) negotiation.cc:107] Waiting for socket to connect
0809 12:11:16.979976 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0809 12:11:16.980094 (+ 118us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 12:11:16.980407 (+ 313us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 12:11:16.980411 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 12:11:16.980875 (+ 464us) client_negotiation.cc:189] Negotiated authn=SASL
0809 12:11:16.981207 (+ 332us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 12:11:16.981224 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 12:11:16.982425 (+ 1201us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 12:11:16.982431 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 12:11:16.983407 (+ 976us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 12:11:16.983419 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 12:11:16.984195 (+ 776us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 12:11:16.984201 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 12:11:16.984409 (+ 208us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 12:11:16.985010 (+ 601us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0809 12:11:16.985029 (+ 19us) client_negotiation.cc:656] calling sasl_client_start()
0809 12:11:16.987906 (+ 2877us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.222.62:44607: Server oryx/127.0.222.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":400,"thread_start_us":283,"threads_started":1}
W20260809 12:11:16.988397 10914 negotiation.cc:337] Failed RPC negotiation. Trace:
0809 12:11:16.979509 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.222.1:42701 (local address 127.0.222.62:44607)
0809 12:11:16.979782 (+ 273us) server_negotiation.cc:206] Beginning negotiation
0809 12:11:16.979789 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0809 12:11:16.980068 (+ 279us) server_negotiation.cc:407] Connection header received
0809 12:11:16.980208 (+ 140us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 12:11:16.980212 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 12:11:16.980275 (+ 63us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 12:11:16.980359 (+ 84us) server_negotiation.cc:226] Negotiated authn=SASL
0809 12:11:16.981390 (+ 1031us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 12:11:16.982279 (+ 889us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 12:11:16.983566 (+ 1287us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 12:11:16.984022 (+ 456us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 12:11:16.984162 (+ 140us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 12:11:16.988155 (+ 3993us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.222.1:42701: BlockingRecv error: recv got EOF from 127.0.222.1:42701 (error 108)
Metrics: {"server-negotiator.queue_time_us":75}
Aug 09 12:11:17 dist-test-slave-2w3w krb5kdc[9807](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.222.2@KRBTEST.COM for oryx/127.0.222.62@KRBTEST.COM, Server not found in Kerberos database
Aug 09 12:11:17 dist-test-slave-2w3w krb5kdc[9807](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.222.2@KRBTEST.COM for oryx/127.0.222.62@KRBTEST.COM, Server not found in Kerberos database
I20260809 12:11:17.478919 10966 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 12:11:17.469830 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.222.62:44607 (local address 127.0.222.2:34087)
0809 12:11:17.470430 (+ 600us) negotiation.cc:107] Waiting for socket to connect
0809 12:11:17.470447 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0809 12:11:17.470611 (+ 164us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 12:11:17.471027 (+ 416us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 12:11:17.471032 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 12:11:17.471507 (+ 475us) client_negotiation.cc:189] Negotiated authn=SASL
0809 12:11:17.471699 (+ 192us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 12:11:17.471708 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 12:11:17.472862 (+ 1154us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 12:11:17.472870 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 12:11:17.473829 (+ 959us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 12:11:17.473841 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 12:11:17.474547 (+ 706us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 12:11:17.474553 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 12:11:17.474730 (+ 177us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 12:11:17.475378 (+ 648us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0809 12:11:17.475395 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0809 12:11:17.478653 (+ 3258us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.222.62:44607: Server oryx/127.0.222.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":468,"thread_start_us":312,"threads_started":1}
W20260809 12:11:17.479139 10914 negotiation.cc:337] Failed RPC negotiation. Trace:
0809 12:11:17.470000 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.222.2:34087 (local address 127.0.222.62:44607)
0809 12:11:17.470245 (+ 245us) server_negotiation.cc:206] Beginning negotiation
0809 12:11:17.470253 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0809 12:11:17.470543 (+ 290us) server_negotiation.cc:407] Connection header received
0809 12:11:17.470740 (+ 197us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 12:11:17.470745 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 12:11:17.470852 (+ 107us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 12:11:17.470965 (+ 113us) server_negotiation.cc:226] Negotiated authn=SASL
0809 12:11:17.471857 (+ 892us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 12:11:17.472720 (+ 863us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 12:11:17.473975 (+ 1255us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 12:11:17.474400 (+ 425us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 12:11:17.474535 (+ 135us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 12:11:17.478855 (+ 4320us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.222.2:34087: BlockingRecv error: recv got EOF from 127.0.222.2:34087 (error 108)
Metrics: {"server-negotiator.queue_time_us":84}
I20260809 12:11:17.764506 888 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskgH5Pjm/build/asan/bin/kudu pid 10457
I20260809 12:11:17.776974 10968 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 12:11:17.765574 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54850 (local address 127.0.222.1:34291)
0809 12:11:17.766173 (+ 599us) server_negotiation.cc:206] Beginning negotiation
0809 12:11:17.766180 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0809 12:11:17.766207 (+ 27us) server_negotiation.cc:407] Connection header received
0809 12:11:17.766275 (+ 68us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 12:11:17.766281 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 12:11:17.766499 (+ 218us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 12:11:17.766667 (+ 168us) server_negotiation.cc:226] Negotiated authn=SASL
0809 12:11:17.768216 (+ 1549us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 12:11:17.769283 (+ 1067us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 12:11:17.770423 (+ 1140us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 12:11:17.771055 (+ 632us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 12:11:17.771146 (+ 91us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 12:11:17.773871 (+ 2725us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 12:11:17.773880 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 12:11:17.773894 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0809 12:11:17.773930 (+ 36us) server_negotiation.cc:922] calling sasl_server_start()
0809 12:11:17.776473 (+ 2543us) server_negotiation.cc:394] Sending RPC error: FATAL_UNAUTHORIZED: Not authorized: Request ticket server kudu/127.0.222.1@KRBTEST.COM not found in keytab (ticket kvno 2)
0809 12:11:17.776739 (+ 266us) negotiation.cc:327] Negotiation complete: Not authorized: Server connection negotiation failed: server connection from 127.0.0.1:54850: Request ticket server kudu/127.0.222.1@KRBTEST.COM not found in keytab (ticket kvno 2)
Metrics: {"server-negotiator.queue_time_us":443,"thread_start_us":320,"threads_started":1}
W20260809 12:11:17.777105 10968 negotiation.cc:344] Unauthorized connection attempt: Server connection negotiation failed: server connection from 127.0.0.1:54850: Request ticket server kudu/127.0.222.1@KRBTEST.COM not found in keytab (ticket kvno 2)
W20260809 12:11:17.777133 888 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskgH5Pjm/build/asan/bin/kudu pid 10457: Not authorized: Client connection negotiation failed: client connection to 127.0.222.1:34291: FATAL_UNAUTHORIZED: Not authorized: Request ticket server kudu/127.0.222.1@KRBTEST.COM not found in keytab (ticket kvno 2)
I20260809 12:11:17.777391 888 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskgH5Pjm/build/asan/bin/kudu with pid 10457
I20260809 12:11:17.798417 888 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskgH5Pjm/build/asan/bin/kudu pid 10621
I20260809 12:11:17.811108 10969 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 12:11:17.799535 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:32954 (local address 127.0.222.2:39033)
0809 12:11:17.800172 (+ 637us) server_negotiation.cc:206] Beginning negotiation
0809 12:11:17.800179 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0809 12:11:17.800210 (+ 31us) server_negotiation.cc:407] Connection header received
0809 12:11:17.800271 (+ 61us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 12:11:17.800278 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 12:11:17.800490 (+ 212us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 12:11:17.800656 (+ 166us) server_negotiation.cc:226] Negotiated authn=SASL
0809 12:11:17.802043 (+ 1387us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 12:11:17.803351 (+ 1308us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 12:11:17.804613 (+ 1262us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 12:11:17.805206 (+ 593us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 12:11:17.805358 (+ 152us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 12:11:17.807997 (+ 2639us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 12:11:17.808006 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 12:11:17.808020 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0809 12:11:17.808064 (+ 44us) server_negotiation.cc:922] calling sasl_server_start()
0809 12:11:17.810391 (+ 2327us) server_negotiation.cc:394] Sending RPC error: FATAL_UNAUTHORIZED: Not authorized: Request ticket server kudu/127.0.222.2@KRBTEST.COM not found in keytab (ticket kvno 2)
0809 12:11:17.810729 (+ 338us) negotiation.cc:327] Negotiation complete: Not authorized: Server connection negotiation failed: server connection from 127.0.0.1:32954: Request ticket server kudu/127.0.222.2@KRBTEST.COM not found in keytab (ticket kvno 2)
Metrics: {"server-negotiator.queue_time_us":481,"thread_start_us":309,"threads_started":1}
W20260809 12:11:17.811228 10969 negotiation.cc:344] Unauthorized connection attempt: Server connection negotiation failed: server connection from 127.0.0.1:32954: Request ticket server kudu/127.0.222.2@KRBTEST.COM not found in keytab (ticket kvno 2)
W20260809 12:11:17.811197 888 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskgH5Pjm/build/asan/bin/kudu pid 10621: Not authorized: Client connection negotiation failed: client connection to 127.0.222.2:39033: FATAL_UNAUTHORIZED: Not authorized: Request ticket server kudu/127.0.222.2@KRBTEST.COM not found in keytab (ticket kvno 2)
I20260809 12:11:17.811378 888 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskgH5Pjm/build/asan/bin/kudu with pid 10621
I20260809 12:11:17.832386 888 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskgH5Pjm/build/asan/bin/kudu pid 10769
I20260809 12:11:17.844712 10970 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 12:11:17.833502 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34574 (local address 127.0.222.3:35897)
0809 12:11:17.834087 (+ 585us) server_negotiation.cc:206] Beginning negotiation
0809 12:11:17.834095 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0809 12:11:17.834123 (+ 28us) server_negotiation.cc:407] Connection header received
0809 12:11:17.834185 (+ 62us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 12:11:17.834192 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 12:11:17.834426 (+ 234us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 12:11:17.834582 (+ 156us) server_negotiation.cc:226] Negotiated authn=SASL
0809 12:11:17.835995 (+ 1413us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 12:11:17.837069 (+ 1074us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 12:11:17.838213 (+ 1144us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 12:11:17.838765 (+ 552us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 12:11:17.838878 (+ 113us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 12:11:17.841471 (+ 2593us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 12:11:17.841480 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 12:11:17.841494 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0809 12:11:17.841527 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0809 12:11:17.844147 (+ 2620us) server_negotiation.cc:394] Sending RPC error: FATAL_UNAUTHORIZED: Not authorized: Request ticket server kudu/127.0.222.3@KRBTEST.COM not found in keytab (ticket kvno 2)
0809 12:11:17.844449 (+ 302us) negotiation.cc:327] Negotiation complete: Not authorized: Server connection negotiation failed: server connection from 127.0.0.1:34574: Request ticket server kudu/127.0.222.3@KRBTEST.COM not found in keytab (ticket kvno 2)
Metrics: {"server-negotiator.queue_time_us":420,"thread_start_us":299,"threads_started":1}
W20260809 12:11:17.844869 10970 negotiation.cc:344] Unauthorized connection attempt: Server connection negotiation failed: server connection from 127.0.0.1:34574: Request ticket server kudu/127.0.222.3@KRBTEST.COM not found in keytab (ticket kvno 2)
W20260809 12:11:17.844919 888 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskgH5Pjm/build/asan/bin/kudu pid 10769: Not authorized: Client connection negotiation failed: client connection to 127.0.222.3:35897: FATAL_UNAUTHORIZED: Not authorized: Request ticket server kudu/127.0.222.3@KRBTEST.COM not found in keytab (ticket kvno 2)
I20260809 12:11:17.845130 888 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskgH5Pjm/build/asan/bin/kudu with pid 10769
I20260809 12:11:17.865918 888 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskgH5Pjm/build/asan/bin/kudu pid 10377
I20260809 12:11:17.867498 10448 generic_service.cc:196] Checking for leaks (request via RPC)
I20260809 12:11:17.990420 888 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskgH5Pjm/build/asan/bin/kudu with pid 10377
2026-08-09T12:11:18Z chronyd exiting
[ OK ] SecurityITest.TestMismatchingPrincipals (21654 ms)
[ RUN ] SecurityITest.TestRequireAuthenticationInsecureCluster
2026-08-09T12:11:18Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-09T12:11:18Z Disabled control of system clock
I20260809 12:11:18.084658 888 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskgH5Pjm/build/asan/bin/kudu
/tmp/dist-test-taskgH5Pjm/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskgH5Pjm/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786277382073938-888-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskgH5Pjm/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786277382073938-888-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskgH5Pjm/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786277382073938-888-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskgH5Pjm/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786277382073938-888-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.222.62:44495
--webserver_interface=127.0.222.62
--webserver_port=0
--builtin_ntp_servers=127.0.222.20:33419
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.222.62:44495
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {}
W20260809 12:11:18.420852 10975 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260809 12:11:18.421144 10975 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260809 12:11:18.421223 10975 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260809 12:11:18.428635 10975 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260809 12:11:18.428762 10975 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260809 12:11:18.428802 10975 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260809 12:11:18.428869 10975 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260809 12:11:18.428907 10975 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260809 12:11:18.438328 10975 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.222.20:33419
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskgH5Pjm/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786277382073938-888-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskgH5Pjm/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786277382073938-888-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.222.62:44495
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.222.62:44495
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskgH5Pjm/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786277382073938-888-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.222.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskgH5Pjm/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786277382073938-888-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 09 Aug 2026 11:06:41 UTC on ccycloud.quasar-bwoogy.root.comops.site
build id 4575991
ASAN enabled
I20260809 12:11:18.439690 10975 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260809 12:11:18.441421 10975 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260809 12:11:18.454634 10980 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 12:11:18.454634 10981 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 12:11:18.454821 10983 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260809 12:11:18.455526 10975 server_base.cc:1034] running on GCE node
I20260809 12:11:18.456332 10975 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260809 12:11:18.457782 10975 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260809 12:11:18.459072 10975 hybrid_clock.cc:648] HybridClock initialized: now 1786277478459002 us; error 77 us; skew 500 ppm
I20260809 12:11:18.459450 10975 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260809 12:11:18.461865 10975 webserver.cc:460] Webserver started at http://127.0.222.62:46007/ using document root <none> and password file <none>
I20260809 12:11:18.462628 10975 fs_manager.cc:359] Metadata directory not provided
I20260809 12:11:18.462766 10975 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260809 12:11:18.463187 10975 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260809 12:11:18.466171 10975 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskgH5Pjm/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786277382073938-888-0/minicluster-data/master-0/data/instance:
uuid: "34718732490f45d09fc796ee090f1137"
format_stamp: "Formatted at 2026-08-09 12:11:18 on dist-test-slave-2w3w"
server_key: "fba8658eed6562a4d12c561c3b8d0bcd"
server_key_iv: "89db8029511ab04dba11d0de2750a4a5"
server_key_version: "encryptionkey@0"
I20260809 12:11:18.467084 10975 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskgH5Pjm/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786277382073938-888-0/minicluster-data/master-0/wal/instance:
uuid: "34718732490f45d09fc796ee090f1137"
format_stamp: "Formatted at 2026-08-09 12:11:18 on dist-test-slave-2w3w"
server_key: "fba8658eed6562a4d12c561c3b8d0bcd"
server_key_iv: "89db8029511ab04dba11d0de2750a4a5"
server_key_version: "encryptionkey@0"
I20260809 12:11:18.472352 10975 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.001s sys 0.004s
I20260809 12:11:18.475793 10990 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260809 12:11:18.477069 10975 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260809 12:11:18.477269 10975 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskgH5Pjm/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786277382073938-888-0/minicluster-data/master-0/data,/tmp/dist-test-taskgH5Pjm/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786277382073938-888-0/minicluster-data/master-0/wal
uuid: "34718732490f45d09fc796ee090f1137"
format_stamp: "Formatted at 2026-08-09 12:11:18 on dist-test-slave-2w3w"
server_key: "fba8658eed6562a4d12c561c3b8d0bcd"
server_key_iv: "89db8029511ab04dba11d0de2750a4a5"
server_key_version: "encryptionkey@0"
I20260809 12:11:18.477506 10975 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskgH5Pjm/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786277382073938-888-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskgH5Pjm/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786277382073938-888-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskgH5Pjm/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786277382073938-888-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260809 12:11:18.500263 10975 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260809 12:11:18.501174 10975 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260809 12:11:18.501485 10975 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260809 12:11:18.520449 10975 rpc_server.cc:307] RPC server started. Bound to: 127.0.222.62:44495
I20260809 12:11:18.520462 11048 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.222.62:44495 every 8 connection(s)
I20260809 12:11:18.521790 10975 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskgH5Pjm/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786277382073938-888-0/minicluster-data/master-0/data/info.pb
I20260809 12:11:18.525552 11049 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260809 12:11:18.529524 888 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskgH5Pjm/build/asan/bin/kudu as pid 10975
I20260809 12:11:18.529723 888 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskgH5Pjm/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786277382073938-888-0/minicluster-data/master-0/wal/instance
I20260809 12:11:18.530395 888 external_mini_cluster.cc:1428] Setting key d1824fa4c74f488efb067c3611a721e7
I20260809 12:11:18.536811 11049 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 34718732490f45d09fc796ee090f1137: Bootstrap starting.
I20260809 12:11:18.540488 11049 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 34718732490f45d09fc796ee090f1137: Neither blocks nor log segments found. Creating new log.
I20260809 12:11:18.541761 11049 log.cc:824] T 00000000000000000000000000000000 P 34718732490f45d09fc796ee090f1137: Log is configured to *not* fsync() on all Append() calls
I20260809 12:11:18.545205 11049 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 34718732490f45d09fc796ee090f1137: No bootstrap required, opened a new log
I20260809 12:11:18.545384 11052 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 12:11:18.532531 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34146 (local address 127.0.222.62:44495)
0809 12:11:18.533577 (+ 1046us) server_negotiation.cc:206] Beginning negotiation
0809 12:11:18.533588 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0809 12:11:18.533637 (+ 49us) server_negotiation.cc:407] Connection header received
0809 12:11:18.535476 (+ 1839us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 12:11:18.535500 (+ 24us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 12:11:18.535931 (+ 431us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 12:11:18.536245 (+ 314us) server_negotiation.cc:226] Negotiated authn=SASL
0809 12:11:18.537008 (+ 763us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 12:11:18.538308 (+ 1300us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 12:11:18.540285 (+ 1977us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 12:11:18.541083 (+ 798us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 12:11:18.541217 (+ 134us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 12:11:18.542658 (+ 1441us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 12:11:18.542675 (+ 17us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 12:11:18.542699 (+ 24us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0809 12:11:18.543544 (+ 845us) server_negotiation.cc:922] calling sasl_server_start()
0809 12:11:18.543622 (+ 78us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0809 12:11:18.543649 (+ 27us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 12:11:18.543754 (+ 105us) server_negotiation.cc:1035] Waiting for connection context
0809 12:11:18.544022 (+ 268us) server_negotiation.cc:299] Negotiation successful
0809 12:11:18.544377 (+ 355us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":617,"thread_start_us":354,"threads_started":1}
I20260809 12:11:18.553422 11049 raft_consensus.cc:348] T 00000000000000000000000000000000 P 34718732490f45d09fc796ee090f1137 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "34718732490f45d09fc796ee090f1137" member_type: VOTER last_known_addr { host: "127.0.222.62" port: 44495 } }
I20260809 12:11:18.553790 11049 raft_consensus.cc:374] T 00000000000000000000000000000000 P 34718732490f45d09fc796ee090f1137 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260809 12:11:18.553908 11049 raft_consensus.cc:729] T 00000000000000000000000000000000 P 34718732490f45d09fc796ee090f1137 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 34718732490f45d09fc796ee090f1137, State: Initialized, Role: FOLLOWER
I20260809 12:11:18.554441 11049 consensus_queue.cc:260] T 00000000000000000000000000000000 P 34718732490f45d09fc796ee090f1137 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "34718732490f45d09fc796ee090f1137" member_type: VOTER last_known_addr { host: "127.0.222.62" port: 44495 } }
I20260809 12:11:18.554677 11049 raft_consensus.cc:388] T 00000000000000000000000000000000 P 34718732490f45d09fc796ee090f1137 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260809 12:11:18.554845 11049 raft_consensus.cc:482] T 00000000000000000000000000000000 P 34718732490f45d09fc796ee090f1137 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260809 12:11:18.555058 11049 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 34718732490f45d09fc796ee090f1137 [term 0 FOLLOWER]: Advancing to term 1
I20260809 12:11:18.557538 11049 raft_consensus.cc:504] T 00000000000000000000000000000000 P 34718732490f45d09fc796ee090f1137 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "34718732490f45d09fc796ee090f1137" member_type: VOTER last_known_addr { host: "127.0.222.62" port: 44495 } }
I20260809 12:11:18.558060 11049 leader_election.cc:302] T 00000000000000000000000000000000 P 34718732490f45d09fc796ee090f1137 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 34718732490f45d09fc796ee090f1137; no voters:
I20260809 12:11:18.558807 11049 leader_election.cc:288] T 00000000000000000000000000000000 P 34718732490f45d09fc796ee090f1137 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260809 12:11:18.558996 11054 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 34718732490f45d09fc796ee090f1137 [term 1 FOLLOWER]: Leader election won for term 1
I20260809 12:11:18.559806 11054 raft_consensus.cc:686] T 00000000000000000000000000000000 P 34718732490f45d09fc796ee090f1137 [term 1 LEADER]: Becoming Leader. State: Replica: 34718732490f45d09fc796ee090f1137, State: Running, Role: LEADER
I20260809 12:11:18.560436 11054 consensus_queue.cc:237] T 00000000000000000000000000000000 P 34718732490f45d09fc796ee090f1137 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "34718732490f45d09fc796ee090f1137" member_type: VOTER last_known_addr { host: "127.0.222.62" port: 44495 } }
I20260809 12:11:18.561043 11049 sys_catalog.cc:564] T 00000000000000000000000000000000 P 34718732490f45d09fc796ee090f1137 [sys.catalog]: configured and running, proceeding with master startup.
I20260809 12:11:18.563261 11055 sys_catalog.cc:455] T 00000000000000000000000000000000 P 34718732490f45d09fc796ee090f1137 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "34718732490f45d09fc796ee090f1137" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "34718732490f45d09fc796ee090f1137" member_type: VOTER last_known_addr { host: "127.0.222.62" port: 44495 } } }
I20260809 12:11:18.563634 11055 sys_catalog.cc:458] T 00000000000000000000000000000000 P 34718732490f45d09fc796ee090f1137 [sys.catalog]: This master's current role is: LEADER
I20260809 12:11:18.564286 11056 sys_catalog.cc:455] T 00000000000000000000000000000000 P 34718732490f45d09fc796ee090f1137 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 34718732490f45d09fc796ee090f1137. Latest consensus state: current_term: 1 leader_uuid: "34718732490f45d09fc796ee090f1137" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "34718732490f45d09fc796ee090f1137" member_type: VOTER last_known_addr { host: "127.0.222.62" port: 44495 } } }
I20260809 12:11:18.564723 11056 sys_catalog.cc:458] T 00000000000000000000000000000000 P 34718732490f45d09fc796ee090f1137 [sys.catalog]: This master's current role is: LEADER
I20260809 12:11:18.567549 11060 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260809 12:11:18.572434 11060 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260809 12:11:18.581259 11060 catalog_manager.cc:1347] Generated new cluster ID: 0a0a5e6391464e419da7720b0315ef90
I20260809 12:11:18.581480 11060 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260809 12:11:18.602072 11060 catalog_manager.cc:1370] Generated new certificate authority record
I20260809 12:11:18.603672 11060 catalog_manager.cc:1504] Loading token signing keys...
I20260809 12:11:18.609059 11060 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 34718732490f45d09fc796ee090f1137: Generated new TSK 0
I20260809 12:11:18.610074 11060 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260809 12:11:18.615080 888 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskgH5Pjm/build/asan/bin/kudu
/tmp/dist-test-taskgH5Pjm/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskgH5Pjm/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786277382073938-888-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskgH5Pjm/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786277382073938-888-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskgH5Pjm/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786277382073938-888-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskgH5Pjm/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786277382073938-888-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.222.1:0
--local_ip_for_outbound_sockets=127.0.222.1
--webserver_interface=127.0.222.1
--webserver_port=0
--tserver_master_addrs=127.0.222.62:44495
--builtin_ntp_servers=127.0.222.20:33419
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {}
W20260809 12:11:18.958148 11075 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260809 12:11:18.958496 11075 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260809 12:11:18.958664 11075 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260809 12:11:18.965886 11075 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260809 12:11:18.966028 11075 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260809 12:11:18.966215 11075 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.222.1
I20260809 12:11:18.975251 11075 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.222.20:33419
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskgH5Pjm/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786277382073938-888-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskgH5Pjm/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786277382073938-888-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.222.1:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskgH5Pjm/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786277382073938-888-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.222.1
--webserver_port=0
--tserver_master_addrs=127.0.222.62:44495
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.222.1
--log_dir=/tmp/dist-test-taskgH5Pjm/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786277382073938-888-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 09 Aug 2026 11:06:41 UTC on ccycloud.quasar-bwoogy.root.comops.site
build id 4575991
ASAN enabled
I20260809 12:11:18.976615 11075 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260809 12:11:18.978453 11075 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260809 12:11:18.992062 11080 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 12:11:18.992137 11081 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 12:11:18.992115 11083 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260809 12:11:18.992887 11075 server_base.cc:1034] running on GCE node
I20260809 12:11:18.993589 11075 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260809 12:11:18.994585 11075 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260809 12:11:18.995786 11075 hybrid_clock.cc:648] HybridClock initialized: now 1786277478995725 us; error 71 us; skew 500 ppm
I20260809 12:11:18.996127 11075 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260809 12:11:18.998409 11075 webserver.cc:460] Webserver started at http://127.0.222.1:37067/ using document root <none> and password file <none>
I20260809 12:11:18.999197 11075 fs_manager.cc:359] Metadata directory not provided
I20260809 12:11:18.999322 11075 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260809 12:11:18.999694 11075 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260809 12:11:19.002626 11075 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskgH5Pjm/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786277382073938-888-0/minicluster-data/ts-0/data/instance:
uuid: "0e46442ddef24e54bd99b622bb7a5671"
format_stamp: "Formatted at 2026-08-09 12:11:18 on dist-test-slave-2w3w"
server_key: "fcbaeea30de1d35b39c9107c083862f0"
server_key_iv: "a8cb598e750249840e1b35c753453af9"
server_key_version: "encryptionkey@0"
I20260809 12:11:19.003572 11075 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskgH5Pjm/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786277382073938-888-0/minicluster-data/ts-0/wal/instance:
uuid: "0e46442ddef24e54bd99b622bb7a5671"
format_stamp: "Formatted at 2026-08-09 12:11:18 on dist-test-slave-2w3w"
server_key: "fcbaeea30de1d35b39c9107c083862f0"
server_key_iv: "a8cb598e750249840e1b35c753453af9"
server_key_version: "encryptionkey@0"
I20260809 12:11:19.009101 11075 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.001s sys 0.004s
I20260809 12:11:19.012563 11089 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260809 12:11:19.013834 11075 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.000s
I20260809 12:11:19.014026 11075 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskgH5Pjm/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786277382073938-888-0/minicluster-data/ts-0/data,/tmp/dist-test-taskgH5Pjm/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786277382073938-888-0/minicluster-data/ts-0/wal
uuid: "0e46442ddef24e54bd99b622bb7a5671"
format_stamp: "Formatted at 2026-08-09 12:11:18 on dist-test-slave-2w3w"
server_key: "fcbaeea30de1d35b39c9107c083862f0"
server_key_iv: "a8cb598e750249840e1b35c753453af9"
server_key_version: "encryptionkey@0"
I20260809 12:11:19.014263 11075 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskgH5Pjm/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786277382073938-888-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskgH5Pjm/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786277382073938-888-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskgH5Pjm/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786277382073938-888-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260809 12:11:19.073800 11075 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260809 12:11:19.074702 11075 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260809 12:11:19.075075 11075 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260809 12:11:19.076102 11075 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260809 12:11:19.077581 11075 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260809 12:11:19.077709 11075 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260809 12:11:19.077840 11075 ts_tablet_manager.cc:610] Registered 0 tablets
I20260809 12:11:19.077934 11075 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260809 12:11:19.112238 11075 rpc_server.cc:307] RPC server started. Bound to: 127.0.222.1:44249
I20260809 12:11:19.112326 11216 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.222.1:44249 every 8 connection(s)
I20260809 12:11:19.113649 11075 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskgH5Pjm/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786277382073938-888-0/minicluster-data/ts-0/data/info.pb
I20260809 12:11:19.115247 888 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskgH5Pjm/build/asan/bin/kudu as pid 11075
I20260809 12:11:19.115468 888 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskgH5Pjm/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786277382073938-888-0/minicluster-data/ts-0/wal/instance
I20260809 12:11:19.116169 888 external_mini_cluster.cc:1428] Setting key d690c48927cbf97113e33a56221248da
I20260809 12:11:19.119560 888 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskgH5Pjm/build/asan/bin/kudu
/tmp/dist-test-taskgH5Pjm/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskgH5Pjm/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786277382073938-888-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskgH5Pjm/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786277382073938-888-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskgH5Pjm/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786277382073938-888-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskgH5Pjm/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786277382073938-888-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.222.2:0
--local_ip_for_outbound_sockets=127.0.222.2
--webserver_interface=127.0.222.2
--webserver_port=0
--tserver_master_addrs=127.0.222.62:44495
--builtin_ntp_servers=127.0.222.20:33419
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {}
I20260809 12:11:19.127310 11219 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 12:11:19.116541 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.222.1:53823 (local address 127.0.222.62:44495)
0809 12:11:19.117167 (+ 626us) server_negotiation.cc:206] Beginning negotiation
0809 12:11:19.117173 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0809 12:11:19.118626 (+ 1453us) server_negotiation.cc:407] Connection header received
0809 12:11:19.120626 (+ 2000us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 12:11:19.120633 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 12:11:19.120716 (+ 83us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 12:11:19.120851 (+ 135us) server_negotiation.cc:226] Negotiated authn=SASL
0809 12:11:19.121910 (+ 1059us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 12:11:19.122738 (+ 828us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 12:11:19.124367 (+ 1629us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 12:11:19.124796 (+ 429us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 12:11:19.124938 (+ 142us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 12:11:19.126384 (+ 1446us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 12:11:19.126389 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 12:11:19.126392 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0809 12:11:19.126410 (+ 18us) server_negotiation.cc:922] calling sasl_server_start()
0809 12:11:19.126450 (+ 40us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0809 12:11:19.126462 (+ 12us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 12:11:19.126570 (+ 108us) server_negotiation.cc:1035] Waiting for connection context
0809 12:11:19.126795 (+ 225us) server_negotiation.cc:299] Negotiation successful
0809 12:11:19.126898 (+ 103us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":395,"thread_start_us":268,"threads_started":1}
I20260809 12:11:19.128338 11220 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 12:11:19.117324 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.222.62:44495 (local address 127.0.222.1:53823)
0809 12:11:19.118444 (+ 1120us) negotiation.cc:107] Waiting for socket to connect
0809 12:11:19.118531 (+ 87us) client_negotiation.cc:174] Beginning negotiation
0809 12:11:19.120361 (+ 1830us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 12:11:19.121090 (+ 729us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 12:11:19.121098 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 12:11:19.121189 (+ 91us) client_negotiation.cc:189] Negotiated authn=SASL
0809 12:11:19.121737 (+ 548us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 12:11:19.121753 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 12:11:19.122907 (+ 1154us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 12:11:19.122913 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 12:11:19.124209 (+ 1296us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 12:11:19.124221 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 12:11:19.125066 (+ 845us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 12:11:19.125076 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 12:11:19.125292 (+ 216us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 12:11:19.126064 (+ 772us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0809 12:11:19.126067 (+ 3us) client_negotiation.cc:656] calling sasl_client_start()
0809 12:11:19.126105 (+ 38us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0809 12:11:19.126183 (+ 78us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0809 12:11:19.126571 (+ 388us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0809 12:11:19.126585 (+ 14us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0809 12:11:19.126595 (+ 10us) client_negotiation.cc:769] Sending connection context
0809 12:11:19.126794 (+ 199us) client_negotiation.cc:240] Negotiation successful
0809 12:11:19.127208 (+ 414us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":534,"thread_start_us":317,"threads_started":1}
I20260809 12:11:19.129865 11217 heartbeater.cc:347] Connected to a master server at 127.0.222.62:44495
I20260809 12:11:19.130194 11217 heartbeater.cc:464] Registering TS with master...
I20260809 12:11:19.131016 11217 heartbeater.cc:511] Master 127.0.222.62:44495 requested a full tablet report, sending...
I20260809 12:11:19.133324 11008 ts_manager.cc:194] Registered new tserver with Master: 0e46442ddef24e54bd99b622bb7a5671 (127.0.222.1:44249)
I20260809 12:11:19.134940 11008 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.222.1:53823
W20260809 12:11:19.459849 11221 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260809 12:11:19.460198 11221 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260809 12:11:19.460353 11221 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260809 12:11:19.468112 11221 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260809 12:11:19.468251 11221 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260809 12:11:19.468425 11221 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.222.2
I20260809 12:11:19.477495 11221 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.222.20:33419
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskgH5Pjm/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786277382073938-888-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskgH5Pjm/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786277382073938-888-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.222.2:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskgH5Pjm/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786277382073938-888-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.222.2
--webserver_port=0
--tserver_master_addrs=127.0.222.62:44495
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.222.2
--log_dir=/tmp/dist-test-taskgH5Pjm/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786277382073938-888-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 09 Aug 2026 11:06:41 UTC on ccycloud.quasar-bwoogy.root.comops.site
build id 4575991
ASAN enabled
I20260809 12:11:19.478953 11221 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260809 12:11:19.480795 11221 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260809 12:11:19.494750 11226 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 12:11:19.494647 11229 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 12:11:19.494715 11227 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260809 12:11:19.495802 11221 server_base.cc:1034] running on GCE node
I20260809 12:11:19.496443 11221 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260809 12:11:19.497435 11221 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260809 12:11:19.498675 11221 hybrid_clock.cc:648] HybridClock initialized: now 1786277479498618 us; error 71 us; skew 500 ppm
I20260809 12:11:19.499066 11221 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260809 12:11:19.501322 11221 webserver.cc:460] Webserver started at http://127.0.222.2:41681/ using document root <none> and password file <none>
I20260809 12:11:19.502054 11221 fs_manager.cc:359] Metadata directory not provided
I20260809 12:11:19.502195 11221 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260809 12:11:19.502569 11221 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260809 12:11:19.505646 11221 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskgH5Pjm/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786277382073938-888-0/minicluster-data/ts-1/data/instance:
uuid: "c22a3f17ddb141578bac8c4b28338a81"
format_stamp: "Formatted at 2026-08-09 12:11:19 on dist-test-slave-2w3w"
server_key: "1ea1fe4d65a88929c6324053793c2918"
server_key_iv: "54cd679d1324f9cf1e3bc97f81a03381"
server_key_version: "encryptionkey@0"
I20260809 12:11:19.506501 11221 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskgH5Pjm/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786277382073938-888-0/minicluster-data/ts-1/wal/instance:
uuid: "c22a3f17ddb141578bac8c4b28338a81"
format_stamp: "Formatted at 2026-08-09 12:11:19 on dist-test-slave-2w3w"
server_key: "1ea1fe4d65a88929c6324053793c2918"
server_key_iv: "54cd679d1324f9cf1e3bc97f81a03381"
server_key_version: "encryptionkey@0"
I20260809 12:11:19.511831 11221 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.008s sys 0.000s
I20260809 12:11:19.515350 11235 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260809 12:11:19.516673 11221 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.002s
I20260809 12:11:19.516875 11221 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskgH5Pjm/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786277382073938-888-0/minicluster-data/ts-1/data,/tmp/dist-test-taskgH5Pjm/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786277382073938-888-0/minicluster-data/ts-1/wal
uuid: "c22a3f17ddb141578bac8c4b28338a81"
format_stamp: "Formatted at 2026-08-09 12:11:19 on dist-test-slave-2w3w"
server_key: "1ea1fe4d65a88929c6324053793c2918"
server_key_iv: "54cd679d1324f9cf1e3bc97f81a03381"
server_key_version: "encryptionkey@0"
I20260809 12:11:19.517113 11221 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskgH5Pjm/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786277382073938-888-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskgH5Pjm/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786277382073938-888-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskgH5Pjm/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786277382073938-888-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260809 12:11:19.536417 11221 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260809 12:11:19.537286 11221 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260809 12:11:19.537581 11221 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260809 12:11:19.538614 11221 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260809 12:11:19.540323 11221 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260809 12:11:19.540468 11221 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260809 12:11:19.540603 11221 ts_tablet_manager.cc:610] Registered 0 tablets
I20260809 12:11:19.540693 11221 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260809 12:11:19.574405 11221 rpc_server.cc:307] RPC server started. Bound to: 127.0.222.2:41401
I20260809 12:11:19.574443 11357 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.222.2:41401 every 8 connection(s)
I20260809 12:11:19.575865 11221 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskgH5Pjm/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786277382073938-888-0/minicluster-data/ts-1/data/info.pb
I20260809 12:11:19.580870 888 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskgH5Pjm/build/asan/bin/kudu as pid 11221
I20260809 12:11:19.581049 888 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskgH5Pjm/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786277382073938-888-0/minicluster-data/ts-1/wal/instance
I20260809 12:11:19.581645 888 external_mini_cluster.cc:1428] Setting key 348bd4674f82a303ec186a7953160332
I20260809 12:11:19.584964 888 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskgH5Pjm/build/asan/bin/kudu
/tmp/dist-test-taskgH5Pjm/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskgH5Pjm/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786277382073938-888-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskgH5Pjm/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786277382073938-888-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskgH5Pjm/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786277382073938-888-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskgH5Pjm/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786277382073938-888-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.222.3:0
--local_ip_for_outbound_sockets=127.0.222.3
--webserver_interface=127.0.222.3
--webserver_port=0
--tserver_master_addrs=127.0.222.62:44495
--builtin_ntp_servers=127.0.222.20:33419
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {}
I20260809 12:11:19.590443 11219 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 12:11:19.578484 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.222.2:53127 (local address 127.0.222.62:44495)
0809 12:11:19.578664 (+ 180us) server_negotiation.cc:206] Beginning negotiation
0809 12:11:19.578669 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0809 12:11:19.580102 (+ 1433us) server_negotiation.cc:407] Connection header received
0809 12:11:19.582570 (+ 2468us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 12:11:19.582575 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 12:11:19.582647 (+ 72us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 12:11:19.582814 (+ 167us) server_negotiation.cc:226] Negotiated authn=SASL
0809 12:11:19.584231 (+ 1417us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 12:11:19.584983 (+ 752us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 12:11:19.587590 (+ 2607us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 12:11:19.587998 (+ 408us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 12:11:19.588111 (+ 113us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 12:11:19.589746 (+ 1635us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 12:11:19.589750 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 12:11:19.589753 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0809 12:11:19.589772 (+ 19us) server_negotiation.cc:922] calling sasl_server_start()
0809 12:11:19.589804 (+ 32us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0809 12:11:19.589815 (+ 11us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 12:11:19.589881 (+ 66us) server_negotiation.cc:1035] Waiting for connection context
0809 12:11:19.590134 (+ 253us) server_negotiation.cc:299] Negotiation successful
0809 12:11:19.590214 (+ 80us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":64}
I20260809 12:11:19.591948 11361 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 12:11:19.579040 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.222.62:44495 (local address 127.0.222.2:53127)
0809 12:11:19.579933 (+ 893us) negotiation.cc:107] Waiting for socket to connect
0809 12:11:19.579969 (+ 36us) client_negotiation.cc:174] Beginning negotiation
0809 12:11:19.582278 (+ 2309us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 12:11:19.583170 (+ 892us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 12:11:19.583182 (+ 12us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 12:11:19.583294 (+ 112us) client_negotiation.cc:189] Negotiated authn=SASL
0809 12:11:19.584042 (+ 748us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 12:11:19.584068 (+ 26us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 12:11:19.585184 (+ 1116us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 12:11:19.585192 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 12:11:19.587429 (+ 2237us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 12:11:19.587455 (+ 26us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 12:11:19.588239 (+ 784us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 12:11:19.588248 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 12:11:19.588583 (+ 335us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 12:11:19.589381 (+ 798us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0809 12:11:19.589386 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0809 12:11:19.589439 (+ 53us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0809 12:11:19.589540 (+ 101us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0809 12:11:19.589908 (+ 368us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0809 12:11:19.589921 (+ 13us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0809 12:11:19.589931 (+ 10us) client_negotiation.cc:769] Sending connection context
0809 12:11:19.590104 (+ 173us) client_negotiation.cc:240] Negotiation successful
0809 12:11:19.590510 (+ 406us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":471,"thread_start_us":272,"threads_started":1}
I20260809 12:11:19.593364 11358 heartbeater.cc:347] Connected to a master server at 127.0.222.62:44495
I20260809 12:11:19.593685 11358 heartbeater.cc:464] Registering TS with master...
I20260809 12:11:19.594475 11358 heartbeater.cc:511] Master 127.0.222.62:44495 requested a full tablet report, sending...
I20260809 12:11:19.595954 11008 ts_manager.cc:194] Registered new tserver with Master: c22a3f17ddb141578bac8c4b28338a81 (127.0.222.2:41401)
I20260809 12:11:19.596791 11008 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.222.2:53127
W20260809 12:11:19.922700 11362 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260809 12:11:19.923074 11362 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260809 12:11:19.923177 11362 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260809 12:11:19.930584 11362 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260809 12:11:19.930685 11362 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260809 12:11:19.930799 11362 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.222.3
I20260809 12:11:19.940655 11362 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.222.20:33419
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskgH5Pjm/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786277382073938-888-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskgH5Pjm/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786277382073938-888-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.222.3:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskgH5Pjm/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786277382073938-888-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.222.3
--webserver_port=0
--tserver_master_addrs=127.0.222.62:44495
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.222.3
--log_dir=/tmp/dist-test-taskgH5Pjm/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786277382073938-888-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 09 Aug 2026 11:06:41 UTC on ccycloud.quasar-bwoogy.root.comops.site
build id 4575991
ASAN enabled
I20260809 12:11:19.941972 11362 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260809 12:11:19.943886 11362 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260809 12:11:19.957325 11369 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 12:11:19.957288 11368 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 12:11:19.957259 11371 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260809 12:11:19.958160 11362 server_base.cc:1034] running on GCE node
I20260809 12:11:19.958841 11362 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260809 12:11:19.959930 11362 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260809 12:11:19.961185 11362 hybrid_clock.cc:648] HybridClock initialized: now 1786277479961139 us; error 58 us; skew 500 ppm
I20260809 12:11:19.961521 11362 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260809 12:11:19.964090 11362 webserver.cc:460] Webserver started at http://127.0.222.3:39221/ using document root <none> and password file <none>
I20260809 12:11:19.964845 11362 fs_manager.cc:359] Metadata directory not provided
I20260809 12:11:19.964983 11362 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260809 12:11:19.965324 11362 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260809 12:11:19.968441 11362 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskgH5Pjm/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786277382073938-888-0/minicluster-data/ts-2/data/instance:
uuid: "9797d032215b4bfc987c7c55f0f803f3"
format_stamp: "Formatted at 2026-08-09 12:11:19 on dist-test-slave-2w3w"
server_key: "1b4ff5519a0715a04b0264c6480d7776"
server_key_iv: "be1c7ac34d16280a376ff2e944862ef6"
server_key_version: "encryptionkey@0"
I20260809 12:11:19.969257 11362 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskgH5Pjm/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786277382073938-888-0/minicluster-data/ts-2/wal/instance:
uuid: "9797d032215b4bfc987c7c55f0f803f3"
format_stamp: "Formatted at 2026-08-09 12:11:19 on dist-test-slave-2w3w"
server_key: "1b4ff5519a0715a04b0264c6480d7776"
server_key_iv: "be1c7ac34d16280a376ff2e944862ef6"
server_key_version: "encryptionkey@0"
I20260809 12:11:19.974399 11362 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.001s
I20260809 12:11:19.977793 11377 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260809 12:11:19.979188 11362 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260809 12:11:19.979364 11362 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskgH5Pjm/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786277382073938-888-0/minicluster-data/ts-2/data,/tmp/dist-test-taskgH5Pjm/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786277382073938-888-0/minicluster-data/ts-2/wal
uuid: "9797d032215b4bfc987c7c55f0f803f3"
format_stamp: "Formatted at 2026-08-09 12:11:19 on dist-test-slave-2w3w"
server_key: "1b4ff5519a0715a04b0264c6480d7776"
server_key_iv: "be1c7ac34d16280a376ff2e944862ef6"
server_key_version: "encryptionkey@0"
I20260809 12:11:19.979585 11362 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskgH5Pjm/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786277382073938-888-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskgH5Pjm/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786277382073938-888-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskgH5Pjm/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786277382073938-888-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260809 12:11:19.996450 11362 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260809 12:11:19.997292 11362 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260809 12:11:19.997613 11362 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260809 12:11:19.998612 11362 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260809 12:11:20.000222 11362 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260809 12:11:20.000352 11362 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260809 12:11:20.000490 11362 ts_tablet_manager.cc:610] Registered 0 tablets
I20260809 12:11:20.000581 11362 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260809 12:11:20.035064 11362 rpc_server.cc:307] RPC server started. Bound to: 127.0.222.3:34143
I20260809 12:11:20.035163 11498 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.222.3:34143 every 8 connection(s)
I20260809 12:11:20.036532 11362 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskgH5Pjm/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786277382073938-888-0/minicluster-data/ts-2/data/info.pb
I20260809 12:11:20.045004 888 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskgH5Pjm/build/asan/bin/kudu as pid 11362
I20260809 12:11:20.045184 888 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskgH5Pjm/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786277382073938-888-0/minicluster-data/ts-2/wal/instance
I20260809 12:11:20.045660 888 external_mini_cluster.cc:1428] Setting key 3165df7bb02d3f8a61284eec62275d5c
I20260809 12:11:20.049400 11219 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 12:11:20.039049 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.222.3:58127 (local address 127.0.222.62:44495)
0809 12:11:20.039238 (+ 189us) server_negotiation.cc:206] Beginning negotiation
0809 12:11:20.039243 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0809 12:11:20.040645 (+ 1402us) server_negotiation.cc:407] Connection header received
0809 12:11:20.042393 (+ 1748us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 12:11:20.042399 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 12:11:20.042490 (+ 91us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 12:11:20.042607 (+ 117us) server_negotiation.cc:226] Negotiated authn=SASL
0809 12:11:20.043664 (+ 1057us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 12:11:20.044438 (+ 774us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 12:11:20.046157 (+ 1719us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 12:11:20.046718 (+ 561us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 12:11:20.046874 (+ 156us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 12:11:20.048512 (+ 1638us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 12:11:20.048517 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 12:11:20.048520 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0809 12:11:20.048536 (+ 16us) server_negotiation.cc:922] calling sasl_server_start()
0809 12:11:20.048567 (+ 31us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0809 12:11:20.048579 (+ 12us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 12:11:20.048649 (+ 70us) server_negotiation.cc:1035] Waiting for connection context
0809 12:11:20.048985 (+ 336us) server_negotiation.cc:299] Negotiation successful
0809 12:11:20.049090 (+ 105us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":70}
I20260809 12:11:20.050273 11501 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 12:11:20.039584 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.222.62:44495 (local address 127.0.222.3:58127)
0809 12:11:20.040478 (+ 894us) negotiation.cc:107] Waiting for socket to connect
0809 12:11:20.040520 (+ 42us) client_negotiation.cc:174] Beginning negotiation
0809 12:11:20.042118 (+ 1598us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 12:11:20.042830 (+ 712us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 12:11:20.042838 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 12:11:20.042922 (+ 84us) client_negotiation.cc:189] Negotiated authn=SASL
0809 12:11:20.043491 (+ 569us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 12:11:20.043507 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 12:11:20.044623 (+ 1116us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 12:11:20.044627 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 12:11:20.046013 (+ 1386us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 12:11:20.046026 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 12:11:20.047124 (+ 1098us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 12:11:20.047135 (+ 11us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 12:11:20.047362 (+ 227us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 12:11:20.048176 (+ 814us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0809 12:11:20.048180 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0809 12:11:20.048230 (+ 50us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0809 12:11:20.048322 (+ 92us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0809 12:11:20.048682 (+ 360us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0809 12:11:20.048695 (+ 13us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0809 12:11:20.048706 (+ 11us) client_negotiation.cc:769] Sending connection context
0809 12:11:20.048949 (+ 243us) client_negotiation.cc:240] Negotiation successful
0809 12:11:20.049263 (+ 314us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":475,"thread_start_us":292,"threads_started":1}
I20260809 12:11:20.051594 11499 heartbeater.cc:347] Connected to a master server at 127.0.222.62:44495
I20260809 12:11:20.051872 11499 heartbeater.cc:464] Registering TS with master...
I20260809 12:11:20.052702 11499 heartbeater.cc:511] Master 127.0.222.62:44495 requested a full tablet report, sending...
I20260809 12:11:20.054005 11008 ts_manager.cc:194] Registered new tserver with Master: 9797d032215b4bfc987c7c55f0f803f3 (127.0.222.3:34143)
I20260809 12:11:20.054756 11008 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.222.3:58127
I20260809 12:11:20.061161 888 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
W20260809 12:11:20.067495 11219 negotiation.cc:337] Failed RPC negotiation. Trace:
0809 12:11:20.065717 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34148 (local address 127.0.222.62:44495)
0809 12:11:20.065954 (+ 237us) server_negotiation.cc:206] Beginning negotiation
0809 12:11:20.065962 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0809 12:11:20.066457 (+ 495us) server_negotiation.cc:407] Connection header received
0809 12:11:20.066562 (+ 105us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 12:11:20.066565 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 12:11:20.066638 (+ 73us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 12:11:20.066757 (+ 119us) server_negotiation.cc:226] Negotiated authn=SASL
0809 12:11:20.067256 (+ 499us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:34148: BlockingRecv error: recv got EOF from 127.0.0.1:34148 (error 108)
Metrics: {"server-negotiator.queue_time_us":62}
I20260809 12:11:20.067839 888 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskgH5Pjm/build/asan/bin/kudu pid 11075
I20260809 12:11:20.075464 11511 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 12:11:20.068914 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49240 (local address 127.0.222.1:44249)
0809 12:11:20.069501 (+ 587us) server_negotiation.cc:206] Beginning negotiation
0809 12:11:20.069510 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0809 12:11:20.069545 (+ 35us) server_negotiation.cc:407] Connection header received
0809 12:11:20.069599 (+ 54us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 12:11:20.069605 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 12:11:20.069808 (+ 203us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 12:11:20.069943 (+ 135us) server_negotiation.cc:226] Negotiated authn=SASL
0809 12:11:20.070274 (+ 331us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 12:11:20.071348 (+ 1074us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 12:11:20.072570 (+ 1222us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 12:11:20.073108 (+ 538us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 12:11:20.073202 (+ 94us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 12:11:20.074151 (+ 949us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 12:11:20.074159 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 12:11:20.074171 (+ 12us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0809 12:11:20.074731 (+ 560us) server_negotiation.cc:922] calling sasl_server_start()
0809 12:11:20.074789 (+ 58us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0809 12:11:20.074809 (+ 20us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 12:11:20.074903 (+ 94us) server_negotiation.cc:1035] Waiting for connection context
0809 12:11:20.075066 (+ 163us) server_negotiation.cc:299] Negotiation successful
0809 12:11:20.075169 (+ 103us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":439,"thread_start_us":314,"threads_started":1}
I20260809 12:11:20.076610 11210 generic_service.cc:196] Checking for leaks (request via RPC)
I20260809 12:11:20.157809 11217 heartbeater.cc:503] Master 127.0.222.62:44495 was elected leader, sending a full tablet report...
I20260809 12:11:20.161257 888 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskgH5Pjm/build/asan/bin/kudu with pid 11075
I20260809 12:11:20.180208 888 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskgH5Pjm/build/asan/bin/kudu pid 11221
I20260809 12:11:20.188457 11513 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 12:11:20.181165 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51844 (local address 127.0.222.2:41401)
0809 12:11:20.181727 (+ 562us) server_negotiation.cc:206] Beginning negotiation
0809 12:11:20.181743 (+ 16us) server_negotiation.cc:399] Waiting for connection header
0809 12:11:20.181772 (+ 29us) server_negotiation.cc:407] Connection header received
0809 12:11:20.181834 (+ 62us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 12:11:20.181841 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 12:11:20.182076 (+ 235us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 12:11:20.182209 (+ 133us) server_negotiation.cc:226] Negotiated authn=SASL
0809 12:11:20.182611 (+ 402us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 12:11:20.183764 (+ 1153us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 12:11:20.185039 (+ 1275us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 12:11:20.185608 (+ 569us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 12:11:20.185711 (+ 103us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 12:11:20.186728 (+ 1017us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 12:11:20.186735 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 12:11:20.186749 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0809 12:11:20.187546 (+ 797us) server_negotiation.cc:922] calling sasl_server_start()
0809 12:11:20.187592 (+ 46us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0809 12:11:20.187613 (+ 21us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 12:11:20.187715 (+ 102us) server_negotiation.cc:1035] Waiting for connection context
0809 12:11:20.187906 (+ 191us) server_negotiation.cc:299] Negotiation successful
0809 12:11:20.188079 (+ 173us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":386,"thread_start_us":297,"threads_started":1}
I20260809 12:11:20.189788 11353 generic_service.cc:196] Checking for leaks (request via RPC)
I20260809 12:11:20.271767 888 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskgH5Pjm/build/asan/bin/kudu with pid 11221
I20260809 12:11:20.292174 888 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskgH5Pjm/build/asan/bin/kudu pid 11362
I20260809 12:11:20.300837 11515 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 12:11:20.293309 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39854 (local address 127.0.222.3:34143)
0809 12:11:20.293964 (+ 655us) server_negotiation.cc:206] Beginning negotiation
0809 12:11:20.293971 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0809 12:11:20.294013 (+ 42us) server_negotiation.cc:407] Connection header received
0809 12:11:20.294079 (+ 66us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 12:11:20.294085 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 12:11:20.294296 (+ 211us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 12:11:20.294488 (+ 192us) server_negotiation.cc:226] Negotiated authn=SASL
0809 12:11:20.295019 (+ 531us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 12:11:20.296096 (+ 1077us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 12:11:20.297379 (+ 1283us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 12:11:20.297963 (+ 584us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 12:11:20.298122 (+ 159us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 12:11:20.299328 (+ 1206us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 12:11:20.299337 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 12:11:20.299350 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0809 12:11:20.299948 (+ 598us) server_negotiation.cc:922] calling sasl_server_start()
0809 12:11:20.300011 (+ 63us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0809 12:11:20.300031 (+ 20us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 12:11:20.300162 (+ 131us) server_negotiation.cc:1035] Waiting for connection context
0809 12:11:20.300320 (+ 158us) server_negotiation.cc:299] Negotiation successful
0809 12:11:20.300474 (+ 154us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":513,"thread_start_us":349,"threads_started":1}
I20260809 12:11:20.302084 11491 generic_service.cc:196] Checking for leaks (request via RPC)
I20260809 12:11:20.385867 888 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskgH5Pjm/build/asan/bin/kudu with pid 11362
I20260809 12:11:20.405184 888 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskgH5Pjm/build/asan/bin/kudu pid 10975
I20260809 12:11:20.406508 11044 generic_service.cc:196] Checking for leaks (request via RPC)
I20260809 12:11:20.515162 888 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskgH5Pjm/build/asan/bin/kudu with pid 10975
2026-08-09T12:11:20Z chronyd exiting
[ OK ] SecurityITest.TestRequireAuthenticationInsecureCluster (2517 ms)
[ RUN ] SecurityITest.TestRequireEncryptionInsecureCluster
2026-08-09T12:11:20Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-09T12:11:20Z Disabled control of system clock
I20260809 12:11:20.605716 888 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskgH5Pjm/build/asan/bin/kudu
/tmp/dist-test-taskgH5Pjm/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskgH5Pjm/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786277382073938-888-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskgH5Pjm/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786277382073938-888-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskgH5Pjm/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786277382073938-888-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskgH5Pjm/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786277382073938-888-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.222.62:38275
--webserver_interface=127.0.222.62
--webserver_port=0
--builtin_ntp_servers=127.0.222.20:39751
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.222.62:38275
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--rpc_encryption=disabled
--rpc_authentication=disabled with env {}
W20260809 12:11:20.942245 11521 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260809 12:11:20.942539 11521 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260809 12:11:20.942629 11521 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260809 12:11:20.950088 11521 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260809 12:11:20.950217 11521 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260809 12:11:20.950294 11521 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260809 12:11:20.950405 11521 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260809 12:11:20.950485 11521 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260809 12:11:20.959754 11521 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.222.20:39751
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskgH5Pjm/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786277382073938-888-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskgH5Pjm/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786277382073938-888-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.222.62:38275
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.222.62:38275
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskgH5Pjm/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786277382073938-888-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.222.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskgH5Pjm/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786277382073938-888-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 09 Aug 2026 11:06:41 UTC on ccycloud.quasar-bwoogy.root.comops.site
build id 4575991
ASAN enabled
I20260809 12:11:20.961058 11521 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260809 12:11:20.962915 11521 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260809 12:11:20.975193 11526 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 12:11:20.975273 11529 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 12:11:20.975306 11527 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260809 12:11:20.976042 11521 server_base.cc:1034] running on GCE node
I20260809 12:11:20.976836 11521 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260809 12:11:20.978204 11521 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260809 12:11:20.979436 11521 hybrid_clock.cc:648] HybridClock initialized: now 1786277480979382 us; error 43 us; skew 500 ppm
I20260809 12:11:20.979789 11521 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260809 12:11:20.982137 11521 webserver.cc:460] Webserver started at http://127.0.222.62:45773/ using document root <none> and password file <none>
I20260809 12:11:20.982811 11521 fs_manager.cc:359] Metadata directory not provided
I20260809 12:11:20.982945 11521 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260809 12:11:20.983346 11521 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260809 12:11:20.986208 11521 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskgH5Pjm/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786277382073938-888-0/minicluster-data/master-0/data/instance:
uuid: "2bef08e820e647dbaa6240e53f820ce1"
format_stamp: "Formatted at 2026-08-09 12:11:20 on dist-test-slave-2w3w"
server_key: "06040247a5327701bd5fdaaf8ddcf16e"
server_key_iv: "ff72d36e8a2594e38f4e6d12f0aaae48"
server_key_version: "encryptionkey@0"
I20260809 12:11:20.987071 11521 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskgH5Pjm/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786277382073938-888-0/minicluster-data/master-0/wal/instance:
uuid: "2bef08e820e647dbaa6240e53f820ce1"
format_stamp: "Formatted at 2026-08-09 12:11:20 on dist-test-slave-2w3w"
server_key: "06040247a5327701bd5fdaaf8ddcf16e"
server_key_iv: "ff72d36e8a2594e38f4e6d12f0aaae48"
server_key_version: "encryptionkey@0"
I20260809 12:11:20.992430 11521 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.000s sys 0.004s
I20260809 12:11:20.996268 11535 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260809 12:11:20.997601 11521 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.003s
I20260809 12:11:20.997814 11521 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskgH5Pjm/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786277382073938-888-0/minicluster-data/master-0/data,/tmp/dist-test-taskgH5Pjm/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786277382073938-888-0/minicluster-data/master-0/wal
uuid: "2bef08e820e647dbaa6240e53f820ce1"
format_stamp: "Formatted at 2026-08-09 12:11:20 on dist-test-slave-2w3w"
server_key: "06040247a5327701bd5fdaaf8ddcf16e"
server_key_iv: "ff72d36e8a2594e38f4e6d12f0aaae48"
server_key_version: "encryptionkey@0"
I20260809 12:11:20.998059 11521 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskgH5Pjm/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786277382073938-888-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskgH5Pjm/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786277382073938-888-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskgH5Pjm/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786277382073938-888-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260809 12:11:21.004153 11521 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260809 12:11:21.004940 11521 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260809 12:11:21.005261 11521 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260809 12:11:21.024204 11521 rpc_server.cc:307] RPC server started. Bound to: 127.0.222.62:38275
I20260809 12:11:21.024237 11586 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.222.62:38275 every 8 connection(s)
I20260809 12:11:21.025708 11521 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskgH5Pjm/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786277382073938-888-0/minicluster-data/master-0/data/info.pb
I20260809 12:11:21.029486 11588 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260809 12:11:21.031296 888 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskgH5Pjm/build/asan/bin/kudu as pid 11521
I20260809 12:11:21.031534 888 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskgH5Pjm/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786277382073938-888-0/minicluster-data/master-0/wal/instance
I20260809 12:11:21.032405 888 external_mini_cluster.cc:1428] Setting key 2c2e286d8f185d2b9775f085a7f6db44
I20260809 12:11:21.040969 11591 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 12:11:21.034561 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57002 (local address 127.0.222.62:38275)
0809 12:11:21.035732 (+ 1171us) server_negotiation.cc:206] Beginning negotiation
0809 12:11:21.035743 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0809 12:11:21.035791 (+ 48us) server_negotiation.cc:407] Connection header received
0809 12:11:21.037493 (+ 1702us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 12:11:21.037516 (+ 23us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 12:11:21.037906 (+ 390us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 12:11:21.038215 (+ 309us) server_negotiation.cc:226] Negotiated authn=SASL
0809 12:11:21.039385 (+ 1170us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 12:11:21.039391 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 12:11:21.039402 (+ 11us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0809 12:11:21.039416 (+ 14us) server_negotiation.cc:922] calling sasl_server_start()
0809 12:11:21.039477 (+ 61us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0809 12:11:21.039495 (+ 18us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 12:11:21.039673 (+ 178us) server_negotiation.cc:1035] Waiting for connection context
0809 12:11:21.039776 (+ 103us) server_negotiation.cc:299] Negotiation successful
0809 12:11:21.040062 (+ 286us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":747,"thread_start_us":483,"threads_started":1}
I20260809 12:11:21.042732 11588 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 2bef08e820e647dbaa6240e53f820ce1: Bootstrap starting.
I20260809 12:11:21.045887 11588 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 2bef08e820e647dbaa6240e53f820ce1: Neither blocks nor log segments found. Creating new log.
I20260809 12:11:21.047454 11588 log.cc:824] T 00000000000000000000000000000000 P 2bef08e820e647dbaa6240e53f820ce1: Log is configured to *not* fsync() on all Append() calls
I20260809 12:11:21.051860 11588 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 2bef08e820e647dbaa6240e53f820ce1: No bootstrap required, opened a new log
I20260809 12:11:21.060191 11588 raft_consensus.cc:348] T 00000000000000000000000000000000 P 2bef08e820e647dbaa6240e53f820ce1 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2bef08e820e647dbaa6240e53f820ce1" member_type: VOTER last_known_addr { host: "127.0.222.62" port: 38275 } }
I20260809 12:11:21.060555 11588 raft_consensus.cc:374] T 00000000000000000000000000000000 P 2bef08e820e647dbaa6240e53f820ce1 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260809 12:11:21.060667 11588 raft_consensus.cc:729] T 00000000000000000000000000000000 P 2bef08e820e647dbaa6240e53f820ce1 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2bef08e820e647dbaa6240e53f820ce1, State: Initialized, Role: FOLLOWER
I20260809 12:11:21.061344 11588 consensus_queue.cc:260] T 00000000000000000000000000000000 P 2bef08e820e647dbaa6240e53f820ce1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2bef08e820e647dbaa6240e53f820ce1" member_type: VOTER last_known_addr { host: "127.0.222.62" port: 38275 } }
I20260809 12:11:21.061789 11588 raft_consensus.cc:388] T 00000000000000000000000000000000 P 2bef08e820e647dbaa6240e53f820ce1 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260809 12:11:21.061946 11588 raft_consensus.cc:482] T 00000000000000000000000000000000 P 2bef08e820e647dbaa6240e53f820ce1 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260809 12:11:21.062142 11588 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 2bef08e820e647dbaa6240e53f820ce1 [term 0 FOLLOWER]: Advancing to term 1
I20260809 12:11:21.064350 11588 raft_consensus.cc:504] T 00000000000000000000000000000000 P 2bef08e820e647dbaa6240e53f820ce1 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2bef08e820e647dbaa6240e53f820ce1" member_type: VOTER last_known_addr { host: "127.0.222.62" port: 38275 } }
I20260809 12:11:21.064788 11588 leader_election.cc:302] T 00000000000000000000000000000000 P 2bef08e820e647dbaa6240e53f820ce1 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 2bef08e820e647dbaa6240e53f820ce1; no voters:
I20260809 12:11:21.065343 11588 leader_election.cc:288] T 00000000000000000000000000000000 P 2bef08e820e647dbaa6240e53f820ce1 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260809 12:11:21.065500 11593 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 2bef08e820e647dbaa6240e53f820ce1 [term 1 FOLLOWER]: Leader election won for term 1
I20260809 12:11:21.066280 11593 raft_consensus.cc:686] T 00000000000000000000000000000000 P 2bef08e820e647dbaa6240e53f820ce1 [term 1 LEADER]: Becoming Leader. State: Replica: 2bef08e820e647dbaa6240e53f820ce1, State: Running, Role: LEADER
I20260809 12:11:21.066839 11593 consensus_queue.cc:237] T 00000000000000000000000000000000 P 2bef08e820e647dbaa6240e53f820ce1 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2bef08e820e647dbaa6240e53f820ce1" member_type: VOTER last_known_addr { host: "127.0.222.62" port: 38275 } }
I20260809 12:11:21.067617 11588 sys_catalog.cc:564] T 00000000000000000000000000000000 P 2bef08e820e647dbaa6240e53f820ce1 [sys.catalog]: configured and running, proceeding with master startup.
I20260809 12:11:21.069588 11595 sys_catalog.cc:455] T 00000000000000000000000000000000 P 2bef08e820e647dbaa6240e53f820ce1 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 2bef08e820e647dbaa6240e53f820ce1. Latest consensus state: current_term: 1 leader_uuid: "2bef08e820e647dbaa6240e53f820ce1" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2bef08e820e647dbaa6240e53f820ce1" member_type: VOTER last_known_addr { host: "127.0.222.62" port: 38275 } } }
I20260809 12:11:21.069988 11595 sys_catalog.cc:458] T 00000000000000000000000000000000 P 2bef08e820e647dbaa6240e53f820ce1 [sys.catalog]: This master's current role is: LEADER
I20260809 12:11:21.070428 11594 sys_catalog.cc:455] T 00000000000000000000000000000000 P 2bef08e820e647dbaa6240e53f820ce1 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "2bef08e820e647dbaa6240e53f820ce1" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2bef08e820e647dbaa6240e53f820ce1" member_type: VOTER last_known_addr { host: "127.0.222.62" port: 38275 } } }
I20260809 12:11:21.070762 11594 sys_catalog.cc:458] T 00000000000000000000000000000000 P 2bef08e820e647dbaa6240e53f820ce1 [sys.catalog]: This master's current role is: LEADER
I20260809 12:11:21.072317 11601 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260809 12:11:21.077306 11601 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260809 12:11:21.087044 11601 catalog_manager.cc:1347] Generated new cluster ID: 239a9804aaa046b8b7f31dd58ae1c49f
I20260809 12:11:21.087152 11601 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260809 12:11:21.099407 11601 catalog_manager.cc:1370] Generated new certificate authority record
I20260809 12:11:21.099867 11601 catalog_manager.cc:1504] Loading token signing keys...
I20260809 12:11:21.125066 11601 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 2bef08e820e647dbaa6240e53f820ce1: Generated new TSK 0
I20260809 12:11:21.125965 11601 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260809 12:11:21.136931 888 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskgH5Pjm/build/asan/bin/kudu
/tmp/dist-test-taskgH5Pjm/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskgH5Pjm/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786277382073938-888-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskgH5Pjm/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786277382073938-888-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskgH5Pjm/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786277382073938-888-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskgH5Pjm/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786277382073938-888-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.222.1:0
--local_ip_for_outbound_sockets=127.0.222.1
--webserver_interface=127.0.222.1
--webserver_port=0
--tserver_master_addrs=127.0.222.62:38275
--builtin_ntp_servers=127.0.222.20:39751
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_encryption=disabled
--rpc_authentication=disabled with env {}
W20260809 12:11:21.480412 11615 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260809 12:11:21.480679 11615 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260809 12:11:21.480842 11615 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260809 12:11:21.488143 11615 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260809 12:11:21.488238 11615 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260809 12:11:21.488413 11615 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.222.1
I20260809 12:11:21.497617 11615 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.222.20:39751
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskgH5Pjm/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786277382073938-888-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskgH5Pjm/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786277382073938-888-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.222.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskgH5Pjm/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786277382073938-888-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.222.1
--webserver_port=0
--tserver_master_addrs=127.0.222.62:38275
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.222.1
--log_dir=/tmp/dist-test-taskgH5Pjm/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786277382073938-888-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 09 Aug 2026 11:06:41 UTC on ccycloud.quasar-bwoogy.root.comops.site
build id 4575991
ASAN enabled
I20260809 12:11:21.499007 11615 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260809 12:11:21.500829 11615 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260809 12:11:21.514592 11620 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 12:11:21.514626 11623 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 12:11:21.514669 11621 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260809 12:11:21.515375 11615 server_base.cc:1034] running on GCE node
I20260809 12:11:21.516218 11615 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260809 12:11:21.517174 11615 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260809 12:11:21.518419 11615 hybrid_clock.cc:648] HybridClock initialized: now 1786277481518379 us; error 48 us; skew 500 ppm
I20260809 12:11:21.518736 11615 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260809 12:11:21.524994 11615 webserver.cc:460] Webserver started at http://127.0.222.1:44655/ using document root <none> and password file <none>
I20260809 12:11:21.525712 11615 fs_manager.cc:359] Metadata directory not provided
I20260809 12:11:21.525859 11615 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260809 12:11:21.526191 11615 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260809 12:11:21.529160 11615 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskgH5Pjm/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786277382073938-888-0/minicluster-data/ts-0/data/instance:
uuid: "3b2a8776a8944d48a6ede57098b23d1c"
format_stamp: "Formatted at 2026-08-09 12:11:21 on dist-test-slave-2w3w"
server_key: "0e6657f5a8ee61d9faa5b812e7c6fb87"
server_key_iv: "c4581f4c968113eec3e578a98ea9b638"
server_key_version: "encryptionkey@0"
I20260809 12:11:21.529973 11615 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskgH5Pjm/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786277382073938-888-0/minicluster-data/ts-0/wal/instance:
uuid: "3b2a8776a8944d48a6ede57098b23d1c"
format_stamp: "Formatted at 2026-08-09 12:11:21 on dist-test-slave-2w3w"
server_key: "0e6657f5a8ee61d9faa5b812e7c6fb87"
server_key_iv: "c4581f4c968113eec3e578a98ea9b638"
server_key_version: "encryptionkey@0"
I20260809 12:11:21.535492 11615 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.002s sys 0.005s
I20260809 12:11:21.539112 11630 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260809 12:11:21.540495 11615 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260809 12:11:21.540688 11615 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskgH5Pjm/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786277382073938-888-0/minicluster-data/ts-0/data,/tmp/dist-test-taskgH5Pjm/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786277382073938-888-0/minicluster-data/ts-0/wal
uuid: "3b2a8776a8944d48a6ede57098b23d1c"
format_stamp: "Formatted at 2026-08-09 12:11:21 on dist-test-slave-2w3w"
server_key: "0e6657f5a8ee61d9faa5b812e7c6fb87"
server_key_iv: "c4581f4c968113eec3e578a98ea9b638"
server_key_version: "encryptionkey@0"
I20260809 12:11:21.540966 11615 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskgH5Pjm/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786277382073938-888-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskgH5Pjm/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786277382073938-888-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskgH5Pjm/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786277382073938-888-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260809 12:11:21.547104 11615 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260809 12:11:21.547930 11615 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260809 12:11:21.548208 11615 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260809 12:11:21.549293 11615 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260809 12:11:21.551021 11615 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260809 12:11:21.551147 11615 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260809 12:11:21.551261 11615 ts_tablet_manager.cc:610] Registered 0 tablets
I20260809 12:11:21.551337 11615 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260809 12:11:21.585048 11615 rpc_server.cc:307] RPC server started. Bound to: 127.0.222.1:38517
I20260809 12:11:21.585139 11751 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.222.1:38517 every 8 connection(s)
I20260809 12:11:21.586386 11615 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskgH5Pjm/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786277382073938-888-0/minicluster-data/ts-0/data/info.pb
I20260809 12:11:21.587546 888 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskgH5Pjm/build/asan/bin/kudu as pid 11615
I20260809 12:11:21.587774 888 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskgH5Pjm/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786277382073938-888-0/minicluster-data/ts-0/wal/instance
I20260809 12:11:21.588603 888 external_mini_cluster.cc:1428] Setting key 244c7ddf82c44bf3d08f9238cdecd1ad
I20260809 12:11:21.591876 888 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskgH5Pjm/build/asan/bin/kudu
/tmp/dist-test-taskgH5Pjm/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskgH5Pjm/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786277382073938-888-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskgH5Pjm/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786277382073938-888-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskgH5Pjm/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786277382073938-888-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskgH5Pjm/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786277382073938-888-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.222.2:0
--local_ip_for_outbound_sockets=127.0.222.2
--webserver_interface=127.0.222.2
--webserver_port=0
--tserver_master_addrs=127.0.222.62:38275
--builtin_ntp_servers=127.0.222.20:39751
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_encryption=disabled
--rpc_authentication=disabled with env {}
I20260809 12:11:21.595360 11754 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 12:11:21.589348 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.222.1:51151 (local address 127.0.222.62:38275)
0809 12:11:21.589809 (+ 461us) server_negotiation.cc:206] Beginning negotiation
0809 12:11:21.589819 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0809 12:11:21.591202 (+ 1383us) server_negotiation.cc:407] Connection header received
0809 12:11:21.593004 (+ 1802us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 12:11:21.593010 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 12:11:21.593075 (+ 65us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 12:11:21.593213 (+ 138us) server_negotiation.cc:226] Negotiated authn=SASL
0809 12:11:21.594544 (+ 1331us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 12:11:21.594550 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 12:11:21.594555 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0809 12:11:21.594581 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0809 12:11:21.594625 (+ 44us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0809 12:11:21.594637 (+ 12us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 12:11:21.594784 (+ 147us) server_negotiation.cc:1035] Waiting for connection context
0809 12:11:21.595003 (+ 219us) server_negotiation.cc:299] Negotiation successful
0809 12:11:21.595075 (+ 72us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":343,"thread_start_us":245,"threads_started":1}
I20260809 12:11:21.596261 11756 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 12:11:21.589926 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.222.62:38275 (local address 127.0.222.1:51151)
0809 12:11:21.591026 (+ 1100us) negotiation.cc:107] Waiting for socket to connect
0809 12:11:21.591091 (+ 65us) client_negotiation.cc:174] Beginning negotiation
0809 12:11:21.592743 (+ 1652us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 12:11:21.593444 (+ 701us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 12:11:21.593452 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 12:11:21.593540 (+ 88us) client_negotiation.cc:189] Negotiated authn=SASL
0809 12:11:21.594243 (+ 703us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0809 12:11:21.594245 (+ 2us) client_negotiation.cc:656] calling sasl_client_start()
0809 12:11:21.594283 (+ 38us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0809 12:11:21.594367 (+ 84us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0809 12:11:21.594747 (+ 380us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0809 12:11:21.594761 (+ 14us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0809 12:11:21.594770 (+ 9us) client_negotiation.cc:769] Sending connection context
0809 12:11:21.595087 (+ 317us) client_negotiation.cc:240] Negotiation successful
0809 12:11:21.595328 (+ 241us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":499,"thread_start_us":302,"threads_started":1}
I20260809 12:11:21.597735 11752 heartbeater.cc:347] Connected to a master server at 127.0.222.62:38275
I20260809 12:11:21.598039 11752 heartbeater.cc:464] Registering TS with master...
I20260809 12:11:21.598609 11752 heartbeater.cc:511] Master 127.0.222.62:38275 requested a full tablet report, sending...
I20260809 12:11:21.600953 11552 ts_manager.cc:194] Registered new tserver with Master: 3b2a8776a8944d48a6ede57098b23d1c (127.0.222.1:38517)
W20260809 12:11:21.930477 11757 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260809 12:11:21.930811 11757 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260809 12:11:21.931015 11757 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260809 12:11:21.938529 11757 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260809 12:11:21.938666 11757 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260809 12:11:21.938849 11757 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.222.2
I20260809 12:11:21.948431 11757 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.222.20:39751
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskgH5Pjm/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786277382073938-888-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskgH5Pjm/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786277382073938-888-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.222.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskgH5Pjm/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786277382073938-888-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.222.2
--webserver_port=0
--tserver_master_addrs=127.0.222.62:38275
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.222.2
--log_dir=/tmp/dist-test-taskgH5Pjm/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786277382073938-888-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 09 Aug 2026 11:06:41 UTC on ccycloud.quasar-bwoogy.root.comops.site
build id 4575991
ASAN enabled
I20260809 12:11:21.949757 11757 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260809 12:11:21.951637 11757 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260809 12:11:21.965008 11762 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 12:11:21.964955 11763 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 12:11:21.964921 11765 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260809 12:11:21.965934 11757 server_base.cc:1034] running on GCE node
I20260809 12:11:21.966569 11757 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260809 12:11:21.967641 11757 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260809 12:11:21.968878 11757 hybrid_clock.cc:648] HybridClock initialized: now 1786277481968831 us; error 43 us; skew 500 ppm
I20260809 12:11:21.969244 11757 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260809 12:11:21.971860 11757 webserver.cc:460] Webserver started at http://127.0.222.2:45081/ using document root <none> and password file <none>
I20260809 12:11:21.972649 11757 fs_manager.cc:359] Metadata directory not provided
I20260809 12:11:21.972793 11757 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260809 12:11:21.973140 11757 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260809 12:11:21.976183 11757 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskgH5Pjm/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786277382073938-888-0/minicluster-data/ts-1/data/instance:
uuid: "c29096b4e74649fe90d4d0e6efcd9d0d"
format_stamp: "Formatted at 2026-08-09 12:11:21 on dist-test-slave-2w3w"
server_key: "5d890e6a325e507ebcc045a11a7f1a7a"
server_key_iv: "bd2463deb029e6425cbe984b1a727e52"
server_key_version: "encryptionkey@0"
I20260809 12:11:21.976989 11757 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskgH5Pjm/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786277382073938-888-0/minicluster-data/ts-1/wal/instance:
uuid: "c29096b4e74649fe90d4d0e6efcd9d0d"
format_stamp: "Formatted at 2026-08-09 12:11:21 on dist-test-slave-2w3w"
server_key: "5d890e6a325e507ebcc045a11a7f1a7a"
server_key_iv: "bd2463deb029e6425cbe984b1a727e52"
server_key_version: "encryptionkey@0"
I20260809 12:11:21.982146 11757 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.000s
I20260809 12:11:21.985461 11772 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260809 12:11:21.986725 11757 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.001s
I20260809 12:11:21.986896 11757 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskgH5Pjm/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786277382073938-888-0/minicluster-data/ts-1/data,/tmp/dist-test-taskgH5Pjm/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786277382073938-888-0/minicluster-data/ts-1/wal
uuid: "c29096b4e74649fe90d4d0e6efcd9d0d"
format_stamp: "Formatted at 2026-08-09 12:11:21 on dist-test-slave-2w3w"
server_key: "5d890e6a325e507ebcc045a11a7f1a7a"
server_key_iv: "bd2463deb029e6425cbe984b1a727e52"
server_key_version: "encryptionkey@0"
I20260809 12:11:21.987174 11757 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskgH5Pjm/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786277382073938-888-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskgH5Pjm/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786277382073938-888-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskgH5Pjm/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786277382073938-888-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260809 12:11:21.992902 11757 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260809 12:11:21.993682 11757 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260809 12:11:21.993963 11757 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260809 12:11:21.994869 11757 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260809 12:11:21.996220 11757 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260809 12:11:21.996351 11757 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260809 12:11:21.996487 11757 ts_tablet_manager.cc:610] Registered 0 tablets
I20260809 12:11:21.996578 11757 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260809 12:11:22.029709 11757 rpc_server.cc:307] RPC server started. Bound to: 127.0.222.2:42671
I20260809 12:11:22.029726 11894 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.222.2:42671 every 8 connection(s)
I20260809 12:11:22.031315 11757 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskgH5Pjm/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786277382073938-888-0/minicluster-data/ts-1/data/info.pb
I20260809 12:11:22.031816 888 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskgH5Pjm/build/asan/bin/kudu as pid 11757
I20260809 12:11:22.032032 888 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskgH5Pjm/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786277382073938-888-0/minicluster-data/ts-1/wal/instance
I20260809 12:11:22.032603 888 external_mini_cluster.cc:1428] Setting key 77a3244018747a5496ea6f8b30553050
I20260809 12:11:22.036298 888 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskgH5Pjm/build/asan/bin/kudu
/tmp/dist-test-taskgH5Pjm/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskgH5Pjm/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786277382073938-888-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskgH5Pjm/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786277382073938-888-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskgH5Pjm/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786277382073938-888-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskgH5Pjm/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786277382073938-888-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.222.3:0
--local_ip_for_outbound_sockets=127.0.222.3
--webserver_interface=127.0.222.3
--webserver_port=0
--tserver_master_addrs=127.0.222.62:38275
--builtin_ntp_servers=127.0.222.20:39751
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_encryption=disabled
--rpc_authentication=disabled with env {}
I20260809 12:11:22.041427 11754 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 12:11:22.034412 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.222.2:38603 (local address 127.0.222.62:38275)
0809 12:11:22.034634 (+ 222us) server_negotiation.cc:206] Beginning negotiation
0809 12:11:22.034641 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0809 12:11:22.036404 (+ 1763us) server_negotiation.cc:407] Connection header received
0809 12:11:22.038847 (+ 2443us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 12:11:22.038857 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 12:11:22.038948 (+ 91us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 12:11:22.039092 (+ 144us) server_negotiation.cc:226] Negotiated authn=SASL
0809 12:11:22.040577 (+ 1485us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 12:11:22.040585 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 12:11:22.040592 (+ 7us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0809 12:11:22.040616 (+ 24us) server_negotiation.cc:922] calling sasl_server_start()
0809 12:11:22.040661 (+ 45us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0809 12:11:22.040679 (+ 18us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 12:11:22.040818 (+ 139us) server_negotiation.cc:1035] Waiting for connection context
0809 12:11:22.041017 (+ 199us) server_negotiation.cc:299] Negotiation successful
0809 12:11:22.041088 (+ 71us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":77}
I20260809 12:11:22.042568 11897 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 12:11:22.035101 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.222.62:38275 (local address 127.0.222.2:38603)
0809 12:11:22.036231 (+ 1130us) negotiation.cc:107] Waiting for socket to connect
0809 12:11:22.036282 (+ 51us) client_negotiation.cc:174] Beginning negotiation
0809 12:11:22.038544 (+ 2262us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 12:11:22.039310 (+ 766us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 12:11:22.039321 (+ 11us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 12:11:22.039437 (+ 116us) client_negotiation.cc:189] Negotiated authn=SASL
0809 12:11:22.040211 (+ 774us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0809 12:11:22.040214 (+ 3us) client_negotiation.cc:656] calling sasl_client_start()
0809 12:11:22.040266 (+ 52us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0809 12:11:22.040381 (+ 115us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0809 12:11:22.040785 (+ 404us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0809 12:11:22.040794 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0809 12:11:22.040802 (+ 8us) client_negotiation.cc:769] Sending connection context
0809 12:11:22.041083 (+ 281us) client_negotiation.cc:240] Negotiation successful
0809 12:11:22.041315 (+ 232us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":558,"thread_start_us":346,"threads_started":1}
I20260809 12:11:22.044070 11895 heartbeater.cc:347] Connected to a master server at 127.0.222.62:38275
I20260809 12:11:22.044395 11895 heartbeater.cc:464] Registering TS with master...
I20260809 12:11:22.045269 11895 heartbeater.cc:511] Master 127.0.222.62:38275 requested a full tablet report, sending...
I20260809 12:11:22.046934 11552 ts_manager.cc:194] Registered new tserver with Master: c29096b4e74649fe90d4d0e6efcd9d0d (127.0.222.2:42671)
W20260809 12:11:22.370532 11898 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260809 12:11:22.370798 11898 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260809 12:11:22.371057 11898 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260809 12:11:22.378433 11898 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260809 12:11:22.378573 11898 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260809 12:11:22.378753 11898 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.222.3
I20260809 12:11:22.387744 11898 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.222.20:39751
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskgH5Pjm/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786277382073938-888-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskgH5Pjm/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786277382073938-888-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.222.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskgH5Pjm/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786277382073938-888-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.222.3
--webserver_port=0
--tserver_master_addrs=127.0.222.62:38275
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.222.3
--log_dir=/tmp/dist-test-taskgH5Pjm/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786277382073938-888-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 09 Aug 2026 11:06:41 UTC on ccycloud.quasar-bwoogy.root.comops.site
build id 4575991
ASAN enabled
I20260809 12:11:22.388993 11898 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260809 12:11:22.390856 11898 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260809 12:11:22.404207 11905 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 12:11:22.404742 11904 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 12:11:22.404150 11907 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260809 12:11:22.405220 11898 server_base.cc:1034] running on GCE node
I20260809 12:11:22.405830 11898 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260809 12:11:22.406767 11898 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260809 12:11:22.407981 11898 hybrid_clock.cc:648] HybridClock initialized: now 1786277482407924 us; error 50 us; skew 500 ppm
I20260809 12:11:22.408339 11898 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260809 12:11:22.410681 11898 webserver.cc:460] Webserver started at http://127.0.222.3:36185/ using document root <none> and password file <none>
I20260809 12:11:22.411453 11898 fs_manager.cc:359] Metadata directory not provided
I20260809 12:11:22.411576 11898 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260809 12:11:22.411909 11898 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260809 12:11:22.414865 11898 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskgH5Pjm/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786277382073938-888-0/minicluster-data/ts-2/data/instance:
uuid: "285a00ac17674e0b80cada98a8258698"
format_stamp: "Formatted at 2026-08-09 12:11:22 on dist-test-slave-2w3w"
server_key: "35da5566372dff02e79d0a37af525e2f"
server_key_iv: "099ee9bd312b1b7481af28d05a1482c7"
server_key_version: "encryptionkey@0"
I20260809 12:11:22.415745 11898 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskgH5Pjm/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786277382073938-888-0/minicluster-data/ts-2/wal/instance:
uuid: "285a00ac17674e0b80cada98a8258698"
format_stamp: "Formatted at 2026-08-09 12:11:22 on dist-test-slave-2w3w"
server_key: "35da5566372dff02e79d0a37af525e2f"
server_key_iv: "099ee9bd312b1b7481af28d05a1482c7"
server_key_version: "encryptionkey@0"
I20260809 12:11:22.420838 11898 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.000s
I20260809 12:11:22.424299 11913 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260809 12:11:22.425601 11898 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260809 12:11:22.425786 11898 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskgH5Pjm/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786277382073938-888-0/minicluster-data/ts-2/data,/tmp/dist-test-taskgH5Pjm/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786277382073938-888-0/minicluster-data/ts-2/wal
uuid: "285a00ac17674e0b80cada98a8258698"
format_stamp: "Formatted at 2026-08-09 12:11:22 on dist-test-slave-2w3w"
server_key: "35da5566372dff02e79d0a37af525e2f"
server_key_iv: "099ee9bd312b1b7481af28d05a1482c7"
server_key_version: "encryptionkey@0"
I20260809 12:11:22.426033 11898 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskgH5Pjm/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786277382073938-888-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskgH5Pjm/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786277382073938-888-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskgH5Pjm/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786277382073938-888-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260809 12:11:22.431751 11898 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260809 12:11:22.432554 11898 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260809 12:11:22.432824 11898 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260809 12:11:22.433714 11898 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260809 12:11:22.435421 11898 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260809 12:11:22.435546 11898 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260809 12:11:22.435693 11898 ts_tablet_manager.cc:610] Registered 0 tablets
I20260809 12:11:22.435762 11898 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260809 12:11:22.470928 11898 rpc_server.cc:307] RPC server started. Bound to: 127.0.222.3:35691
I20260809 12:11:22.471042 12038 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.222.3:35691 every 8 connection(s)
I20260809 12:11:22.472350 11898 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskgH5Pjm/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786277382073938-888-0/minicluster-data/ts-2/data/info.pb
I20260809 12:11:22.476552 888 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskgH5Pjm/build/asan/bin/kudu as pid 11898
I20260809 12:11:22.476818 888 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskgH5Pjm/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786277382073938-888-0/minicluster-data/ts-2/wal/instance
I20260809 12:11:22.477317 888 external_mini_cluster.cc:1428] Setting key 1ff07f4c1d07d528cdb7201d85787405
I20260809 12:11:22.481097 11754 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 12:11:22.475050 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.222.3:36763 (local address 127.0.222.62:38275)
0809 12:11:22.475228 (+ 178us) server_negotiation.cc:206] Beginning negotiation
0809 12:11:22.475232 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0809 12:11:22.476776 (+ 1544us) server_negotiation.cc:407] Connection header received
0809 12:11:22.478802 (+ 2026us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 12:11:22.478808 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 12:11:22.478892 (+ 84us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 12:11:22.479048 (+ 156us) server_negotiation.cc:226] Negotiated authn=SASL
0809 12:11:22.480358 (+ 1310us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 12:11:22.480366 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 12:11:22.480374 (+ 8us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0809 12:11:22.480390 (+ 16us) server_negotiation.cc:922] calling sasl_server_start()
0809 12:11:22.480422 (+ 32us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0809 12:11:22.480434 (+ 12us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 12:11:22.480565 (+ 131us) server_negotiation.cc:1035] Waiting for connection context
0809 12:11:22.480760 (+ 195us) server_negotiation.cc:299] Negotiation successful
0809 12:11:22.480828 (+ 68us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":59}
I20260809 12:11:22.482046 12041 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 12:11:22.475506 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.222.62:38275 (local address 127.0.222.3:36763)
0809 12:11:22.476602 (+ 1096us) negotiation.cc:107] Waiting for socket to connect
0809 12:11:22.476658 (+ 56us) client_negotiation.cc:174] Beginning negotiation
0809 12:11:22.478510 (+ 1852us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 12:11:22.479278 (+ 768us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 12:11:22.479286 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 12:11:22.479377 (+ 91us) client_negotiation.cc:189] Negotiated authn=SASL
0809 12:11:22.480058 (+ 681us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0809 12:11:22.480061 (+ 3us) client_negotiation.cc:656] calling sasl_client_start()
0809 12:11:22.480097 (+ 36us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0809 12:11:22.480183 (+ 86us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0809 12:11:22.480541 (+ 358us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0809 12:11:22.480551 (+ 10us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0809 12:11:22.480559 (+ 8us) client_negotiation.cc:769] Sending connection context
0809 12:11:22.480880 (+ 321us) client_negotiation.cc:240] Negotiation successful
0809 12:11:22.481129 (+ 249us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":580,"thread_start_us":377,"threads_started":1}
I20260809 12:11:22.483351 12039 heartbeater.cc:347] Connected to a master server at 127.0.222.62:38275
I20260809 12:11:22.483659 12039 heartbeater.cc:464] Registering TS with master...
I20260809 12:11:22.484221 12039 heartbeater.cc:511] Master 127.0.222.62:38275 requested a full tablet report, sending...
I20260809 12:11:22.485644 11552 ts_manager.cc:194] Registered new tserver with Master: 285a00ac17674e0b80cada98a8258698 (127.0.222.3:35691)
I20260809 12:11:22.492740 888 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
W20260809 12:11:22.498862 11754 negotiation.cc:337] Failed RPC negotiation. Trace:
0809 12:11:22.497265 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57006 (local address 127.0.222.62:38275)
0809 12:11:22.497450 (+ 185us) server_negotiation.cc:206] Beginning negotiation
0809 12:11:22.497458 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0809 12:11:22.497923 (+ 465us) server_negotiation.cc:407] Connection header received
0809 12:11:22.498123 (+ 200us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 12:11:22.498129 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 12:11:22.498186 (+ 57us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 12:11:22.498285 (+ 99us) server_negotiation.cc:226] Negotiated authn=SASL
0809 12:11:22.498665 (+ 380us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:57006: BlockingRecv error: recv got EOF from 127.0.0.1:57006 (error 108)
Metrics: {"server-negotiator.queue_time_us":59}
I20260809 12:11:22.499783 888 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskgH5Pjm/build/asan/bin/kudu pid 11615
I20260809 12:11:22.503484 12048 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 12:11:22.500784 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48924 (local address 127.0.222.1:38517)
0809 12:11:22.501337 (+ 553us) server_negotiation.cc:206] Beginning negotiation
0809 12:11:22.501344 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0809 12:11:22.501369 (+ 25us) server_negotiation.cc:407] Connection header received
0809 12:11:22.501426 (+ 57us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 12:11:22.501432 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 12:11:22.501697 (+ 265us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 12:11:22.501836 (+ 139us) server_negotiation.cc:226] Negotiated authn=SASL
0809 12:11:22.502841 (+ 1005us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 12:11:22.502848 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 12:11:22.502858 (+ 10us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0809 12:11:22.502873 (+ 15us) server_negotiation.cc:922] calling sasl_server_start()
0809 12:11:22.502920 (+ 47us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0809 12:11:22.502936 (+ 16us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 12:11:22.503104 (+ 168us) server_negotiation.cc:1035] Waiting for connection context
0809 12:11:22.503178 (+ 74us) server_negotiation.cc:299] Negotiation successful
0809 12:11:22.503254 (+ 76us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":401,"thread_start_us":279,"threads_started":1}
I20260809 12:11:22.504618 11747 generic_service.cc:196] Checking for leaks (request via RPC)
I20260809 12:11:22.587167 888 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskgH5Pjm/build/asan/bin/kudu with pid 11615
I20260809 12:11:22.608168 888 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskgH5Pjm/build/asan/bin/kudu pid 11757
I20260809 12:11:22.611716 12052 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 12:11:22.609100 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54296 (local address 127.0.222.2:42671)
0809 12:11:22.609646 (+ 546us) server_negotiation.cc:206] Beginning negotiation
0809 12:11:22.609653 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0809 12:11:22.609682 (+ 29us) server_negotiation.cc:407] Connection header received
0809 12:11:22.609749 (+ 67us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 12:11:22.609755 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 12:11:22.609983 (+ 228us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 12:11:22.610125 (+ 142us) server_negotiation.cc:226] Negotiated authn=SASL
0809 12:11:22.611094 (+ 969us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 12:11:22.611100 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 12:11:22.611111 (+ 11us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0809 12:11:22.611124 (+ 13us) server_negotiation.cc:922] calling sasl_server_start()
0809 12:11:22.611170 (+ 46us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0809 12:11:22.611187 (+ 17us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 12:11:22.611296 (+ 109us) server_negotiation.cc:1035] Waiting for connection context
0809 12:11:22.611391 (+ 95us) server_negotiation.cc:299] Negotiation successful
0809 12:11:22.611482 (+ 91us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":389,"thread_start_us":258,"threads_started":1}
I20260809 12:11:22.612953 11889 generic_service.cc:196] Checking for leaks (request via RPC)
I20260809 12:11:22.695501 888 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskgH5Pjm/build/asan/bin/kudu with pid 11757
I20260809 12:11:22.715175 888 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskgH5Pjm/build/asan/bin/kudu pid 11898
I20260809 12:11:22.719527 12054 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 12:11:22.716232 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55756 (local address 127.0.222.3:35691)
0809 12:11:22.716958 (+ 726us) server_negotiation.cc:206] Beginning negotiation
0809 12:11:22.716965 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0809 12:11:22.716996 (+ 31us) server_negotiation.cc:407] Connection header received
0809 12:11:22.717059 (+ 63us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 12:11:22.717066 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 12:11:22.717283 (+ 217us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 12:11:22.717462 (+ 179us) server_negotiation.cc:226] Negotiated authn=SASL
0809 12:11:22.718680 (+ 1218us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 12:11:22.718686 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 12:11:22.718698 (+ 12us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0809 12:11:22.718714 (+ 16us) server_negotiation.cc:922] calling sasl_server_start()
0809 12:11:22.718753 (+ 39us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0809 12:11:22.718767 (+ 14us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 12:11:22.718962 (+ 195us) server_negotiation.cc:1035] Waiting for connection context
0809 12:11:22.719098 (+ 136us) server_negotiation.cc:299] Negotiation successful
0809 12:11:22.719205 (+ 107us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":573,"thread_start_us":373,"threads_started":1}
I20260809 12:11:22.720808 12034 generic_service.cc:196] Checking for leaks (request via RPC)
I20260809 12:11:22.804302 888 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskgH5Pjm/build/asan/bin/kudu with pid 11898
I20260809 12:11:22.822822 888 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskgH5Pjm/build/asan/bin/kudu pid 11521
I20260809 12:11:22.824124 11582 generic_service.cc:196] Checking for leaks (request via RPC)
I20260809 12:11:22.933434 888 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskgH5Pjm/build/asan/bin/kudu with pid 11521
2026-08-09T12:11:22Z chronyd exiting
[ OK ] SecurityITest.TestRequireEncryptionInsecureCluster (2417 ms)
[ RUN ] SecurityITest.TestRequireAuthenticationSecureCluster
Loading random data
Initializing database '/tmp/dist-test-taskgH5Pjm/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786277382073938-888-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 09 12:11:23 dist-test-slave-2w3w krb5kdc[12060](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 09 12:11:23 dist-test-slave-2w3w krb5kdc[12060](info): set up 2 sockets
Aug 09 12:11:23 dist-test-slave-2w3w krb5kdc[12060](info): commencing operation
krb5kdc: starting...
W20260809 12:11:24.104549 888 mini_kdc.cc:121] Time spent starting KDC: real 1.125s user 0.000s sys 0.040s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskgH5Pjm/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786277382073938-888-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskgH5Pjm/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786277382073938-888-0/krb5kdc/test-user.keytab.
Aug 09 12:11:24 dist-test-slave-2w3w krb5kdc[12060](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786277484, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-09T12:11:24Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-09T12:11:24Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.222.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.222.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.222.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskgH5Pjm/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786277382073938-888-0/krb5kdc/kudu_127.0.222.62.keytab.
Entry for principal kudu/127.0.222.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskgH5Pjm/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786277382073938-888-0/krb5kdc/kudu_127.0.222.62.keytab.
I20260809 12:11:24.399621 888 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskgH5Pjm/build/asan/bin/kudu
/tmp/dist-test-taskgH5Pjm/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskgH5Pjm/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786277382073938-888-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskgH5Pjm/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786277382073938-888-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskgH5Pjm/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786277382073938-888-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskgH5Pjm/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786277382073938-888-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.222.62:46055
--webserver_interface=127.0.222.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskgH5Pjm/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786277382073938-888-0/krb5kdc/kudu_127.0.222.62.keytab
--principal=kudu/127.0.222.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.222.20:41235
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.222.62:46055
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskgH5Pjm/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786277382073938-888-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskgH5Pjm/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786277382073938-888-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskgH5Pjm/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786277382073938-888-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260809 12:11:24.743312 12073 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260809 12:11:24.743603 12073 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260809 12:11:24.743695 12073 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260809 12:11:24.750754 12073 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260809 12:11:24.750859 12073 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260809 12:11:24.750905 12073 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260809 12:11:24.751010 12073 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260809 12:11:24.751080 12073 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260809 12:11:24.761266 12073 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.222.20:41235
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskgH5Pjm/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786277382073938-888-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskgH5Pjm/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786277382073938-888-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.222.62:46055
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskgH5Pjm/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786277382073938-888-0/krb5kdc/kudu_127.0.222.62.keytab
--principal=kudu/127.0.222.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.222.62:46055
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskgH5Pjm/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786277382073938-888-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.222.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskgH5Pjm/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786277382073938-888-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 09 Aug 2026 11:06:41 UTC on ccycloud.quasar-bwoogy.root.comops.site
build id 4575991
ASAN enabled
I20260809 12:11:24.762562 12073 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260809 12:11:24.763866 12073 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260809 12:11:24.776062 12078 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 12:11:24.776072 12079 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 12:11:24.775969 12082 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260809 12:11:24.777151 12073 server_base.cc:1034] running on GCE node
I20260809 12:11:24.778013 12073 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260809 12:11:24.779537 12073 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260809 12:11:24.780776 12073 hybrid_clock.cc:648] HybridClock initialized: now 1786277484780732 us; error 56 us; skew 500 ppm
I20260809 12:11:24.781157 12073 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 09 12:11:24 dist-test-slave-2w3w krb5kdc[12060](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786277484, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.222.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260809 12:11:24.784319 12073 init.cc:377] Logged in from keytab as kudu/127.0.222.62@KRBTEST.COM (short username kudu)
I20260809 12:11:24.785861 12073 webserver.cc:460] Webserver started at http://127.0.222.62:41091/ using document root <none> and password file <none>
I20260809 12:11:24.786571 12073 fs_manager.cc:359] Metadata directory not provided
I20260809 12:11:24.786724 12073 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260809 12:11:24.787114 12073 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260809 12:11:24.790166 12073 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskgH5Pjm/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786277382073938-888-0/minicluster-data/master-0/data/instance:
uuid: "d2968055ebf2465e9f45ae5dcd3d4250"
format_stamp: "Formatted at 2026-08-09 12:11:24 on dist-test-slave-2w3w"
server_key: "88cfcd192f02846679dadc10f8c21014"
server_key_iv: "f807ea64472f7a370fa5ea590f5d206f"
server_key_version: "encryptionkey@0"
I20260809 12:11:24.791046 12073 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskgH5Pjm/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786277382073938-888-0/minicluster-data/master-0/wal/instance:
uuid: "d2968055ebf2465e9f45ae5dcd3d4250"
format_stamp: "Formatted at 2026-08-09 12:11:24 on dist-test-slave-2w3w"
server_key: "88cfcd192f02846679dadc10f8c21014"
server_key_iv: "f807ea64472f7a370fa5ea590f5d206f"
server_key_version: "encryptionkey@0"
I20260809 12:11:24.796334 12073 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.000s sys 0.004s
I20260809 12:11:24.799927 12089 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260809 12:11:24.801298 12073 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.002s
I20260809 12:11:24.801493 12073 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskgH5Pjm/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786277382073938-888-0/minicluster-data/master-0/data,/tmp/dist-test-taskgH5Pjm/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786277382073938-888-0/minicluster-data/master-0/wal
uuid: "d2968055ebf2465e9f45ae5dcd3d4250"
format_stamp: "Formatted at 2026-08-09 12:11:24 on dist-test-slave-2w3w"
server_key: "88cfcd192f02846679dadc10f8c21014"
server_key_iv: "f807ea64472f7a370fa5ea590f5d206f"
server_key_version: "encryptionkey@0"
I20260809 12:11:24.801699 12073 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskgH5Pjm/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786277382073938-888-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskgH5Pjm/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786277382073938-888-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskgH5Pjm/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786277382073938-888-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260809 12:11:24.820072 12073 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260809 12:11:24.824162 12073 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260809 12:11:24.824453 12073 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260809 12:11:24.843914 12073 rpc_server.cc:307] RPC server started. Bound to: 127.0.222.62:46055
I20260809 12:11:24.843955 12146 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.222.62:46055 every 8 connection(s)
I20260809 12:11:24.845321 12073 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskgH5Pjm/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786277382073938-888-0/minicluster-data/master-0/data/info.pb
I20260809 12:11:24.848807 888 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskgH5Pjm/build/asan/bin/kudu as pid 12073
I20260809 12:11:24.849074 888 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskgH5Pjm/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786277382073938-888-0/minicluster-data/master-0/wal/instance
I20260809 12:11:24.849486 12147 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260809 12:11:24.849915 888 external_mini_cluster.cc:1428] Setting key a2e5e7330528ae4c53f0f63ad2e83a3e
I20260809 12:11:24.865461 12147 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P d2968055ebf2465e9f45ae5dcd3d4250: Bootstrap starting.
Aug 09 12:11:24 dist-test-slave-2w3w krb5kdc[12060](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786277484, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.222.62@KRBTEST.COM
I20260809 12:11:24.869619 12147 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P d2968055ebf2465e9f45ae5dcd3d4250: Neither blocks nor log segments found. Creating new log.
I20260809 12:11:24.871074 12147 log.cc:824] T 00000000000000000000000000000000 P d2968055ebf2465e9f45ae5dcd3d4250: Log is configured to *not* fsync() on all Append() calls
I20260809 12:11:24.874766 12147 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P d2968055ebf2465e9f45ae5dcd3d4250: No bootstrap required, opened a new log
I20260809 12:11:24.875873 12151 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 12:11:24.852074 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41320 (local address 127.0.222.62:46055)
0809 12:11:24.852930 (+ 856us) server_negotiation.cc:206] Beginning negotiation
0809 12:11:24.852944 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0809 12:11:24.852986 (+ 42us) server_negotiation.cc:407] Connection header received
0809 12:11:24.855127 (+ 2141us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 12:11:24.855150 (+ 23us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 12:11:24.855545 (+ 395us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 12:11:24.855880 (+ 335us) server_negotiation.cc:226] Negotiated authn=SASL
0809 12:11:24.857770 (+ 1890us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 12:11:24.859171 (+ 1401us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 12:11:24.861031 (+ 1860us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 12:11:24.861750 (+ 719us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 12:11:24.861893 (+ 143us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 12:11:24.866743 (+ 4850us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 12:11:24.866756 (+ 13us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 12:11:24.866775 (+ 19us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0809 12:11:24.866831 (+ 56us) server_negotiation.cc:922] calling sasl_server_start()
0809 12:11:24.871012 (+ 4181us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 12:11:24.871859 (+ 847us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 12:11:24.871871 (+ 12us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 12:11:24.871879 (+ 8us) server_negotiation.cc:967] calling sasl_server_step()
0809 12:11:24.872063 (+ 184us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 12:11:24.872604 (+ 541us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 12:11:24.872610 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 12:11:24.872613 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0809 12:11:24.873388 (+ 775us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 12:11:24.873719 (+ 331us) server_negotiation.cc:1035] Waiting for connection context
0809 12:11:24.874025 (+ 306us) server_negotiation.cc:299] Negotiation successful
0809 12:11:24.874468 (+ 443us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":641,"thread_start_us":413,"threads_started":1}
I20260809 12:11:24.883136 12147 raft_consensus.cc:348] T 00000000000000000000000000000000 P d2968055ebf2465e9f45ae5dcd3d4250 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d2968055ebf2465e9f45ae5dcd3d4250" member_type: VOTER last_known_addr { host: "127.0.222.62" port: 46055 } }
I20260809 12:11:24.883466 12147 raft_consensus.cc:374] T 00000000000000000000000000000000 P d2968055ebf2465e9f45ae5dcd3d4250 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260809 12:11:24.883538 12147 raft_consensus.cc:729] T 00000000000000000000000000000000 P d2968055ebf2465e9f45ae5dcd3d4250 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d2968055ebf2465e9f45ae5dcd3d4250, State: Initialized, Role: FOLLOWER
I20260809 12:11:24.883996 12147 consensus_queue.cc:260] T 00000000000000000000000000000000 P d2968055ebf2465e9f45ae5dcd3d4250 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d2968055ebf2465e9f45ae5dcd3d4250" member_type: VOTER last_known_addr { host: "127.0.222.62" port: 46055 } }
I20260809 12:11:24.884220 12147 raft_consensus.cc:388] T 00000000000000000000000000000000 P d2968055ebf2465e9f45ae5dcd3d4250 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260809 12:11:24.884369 12147 raft_consensus.cc:482] T 00000000000000000000000000000000 P d2968055ebf2465e9f45ae5dcd3d4250 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260809 12:11:24.884543 12147 raft_consensus.cc:3037] T 00000000000000000000000000000000 P d2968055ebf2465e9f45ae5dcd3d4250 [term 0 FOLLOWER]: Advancing to term 1
I20260809 12:11:24.886891 12147 raft_consensus.cc:504] T 00000000000000000000000000000000 P d2968055ebf2465e9f45ae5dcd3d4250 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d2968055ebf2465e9f45ae5dcd3d4250" member_type: VOTER last_known_addr { host: "127.0.222.62" port: 46055 } }
I20260809 12:11:24.887413 12147 leader_election.cc:302] T 00000000000000000000000000000000 P d2968055ebf2465e9f45ae5dcd3d4250 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: d2968055ebf2465e9f45ae5dcd3d4250; no voters:
I20260809 12:11:24.887995 12147 leader_election.cc:288] T 00000000000000000000000000000000 P d2968055ebf2465e9f45ae5dcd3d4250 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260809 12:11:24.888164 12153 raft_consensus.cc:2781] T 00000000000000000000000000000000 P d2968055ebf2465e9f45ae5dcd3d4250 [term 1 FOLLOWER]: Leader election won for term 1
I20260809 12:11:24.889425 12153 raft_consensus.cc:686] T 00000000000000000000000000000000 P d2968055ebf2465e9f45ae5dcd3d4250 [term 1 LEADER]: Becoming Leader. State: Replica: d2968055ebf2465e9f45ae5dcd3d4250, State: Running, Role: LEADER
I20260809 12:11:24.890017 12153 consensus_queue.cc:237] T 00000000000000000000000000000000 P d2968055ebf2465e9f45ae5dcd3d4250 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d2968055ebf2465e9f45ae5dcd3d4250" member_type: VOTER last_known_addr { host: "127.0.222.62" port: 46055 } }
I20260809 12:11:24.890333 12147 sys_catalog.cc:564] T 00000000000000000000000000000000 P d2968055ebf2465e9f45ae5dcd3d4250 [sys.catalog]: configured and running, proceeding with master startup.
I20260809 12:11:24.893570 12154 sys_catalog.cc:455] T 00000000000000000000000000000000 P d2968055ebf2465e9f45ae5dcd3d4250 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "d2968055ebf2465e9f45ae5dcd3d4250" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d2968055ebf2465e9f45ae5dcd3d4250" member_type: VOTER last_known_addr { host: "127.0.222.62" port: 46055 } } }
I20260809 12:11:24.893560 12155 sys_catalog.cc:455] T 00000000000000000000000000000000 P d2968055ebf2465e9f45ae5dcd3d4250 [sys.catalog]: SysCatalogTable state changed. Reason: New leader d2968055ebf2465e9f45ae5dcd3d4250. Latest consensus state: current_term: 1 leader_uuid: "d2968055ebf2465e9f45ae5dcd3d4250" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d2968055ebf2465e9f45ae5dcd3d4250" member_type: VOTER last_known_addr { host: "127.0.222.62" port: 46055 } } }
I20260809 12:11:24.893981 12154 sys_catalog.cc:458] T 00000000000000000000000000000000 P d2968055ebf2465e9f45ae5dcd3d4250 [sys.catalog]: This master's current role is: LEADER
I20260809 12:11:24.894025 12155 sys_catalog.cc:458] T 00000000000000000000000000000000 P d2968055ebf2465e9f45ae5dcd3d4250 [sys.catalog]: This master's current role is: LEADER
W20260809 12:11:24.904450 12170 catalog_manager.cc:1558] T 00000000000000000000000000000000 P d2968055ebf2465e9f45ae5dcd3d4250: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260809 12:11:24.904584 12170 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260809 12:11:24.904716 12162 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260809 12:11:24.906896 12162 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260809 12:11:24.915720 12162 catalog_manager.cc:1347] Generated new cluster ID: 490d9eb12d334751903bfac46d2c3713
I20260809 12:11:24.915853 12162 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260809 12:11:24.927063 12162 catalog_manager.cc:1370] Generated new certificate authority record
I20260809 12:11:24.928512 12162 catalog_manager.cc:1504] Loading token signing keys...
I20260809 12:11:24.940975 12162 catalog_manager.cc:5868] T 00000000000000000000000000000000 P d2968055ebf2465e9f45ae5dcd3d4250: Generated new TSK 0
I20260809 12:11:24.941906 12162 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.0.222.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.222.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.222.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskgH5Pjm/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786277382073938-888-0/krb5kdc/kudu_127.0.222.1.keytab.
Entry for principal kudu/127.0.222.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskgH5Pjm/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786277382073938-888-0/krb5kdc/kudu_127.0.222.1.keytab.
I20260809 12:11:25.026353 888 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskgH5Pjm/build/asan/bin/kudu
/tmp/dist-test-taskgH5Pjm/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskgH5Pjm/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786277382073938-888-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskgH5Pjm/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786277382073938-888-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskgH5Pjm/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786277382073938-888-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskgH5Pjm/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786277382073938-888-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.222.1:0
--local_ip_for_outbound_sockets=127.0.222.1
--webserver_interface=127.0.222.1
--webserver_port=0
--tserver_master_addrs=127.0.222.62:46055
--keytab_file=/tmp/dist-test-taskgH5Pjm/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786277382073938-888-0/krb5kdc/kudu_127.0.222.1.keytab
--principal=kudu/127.0.222.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.222.20:41235
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskgH5Pjm/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786277382073938-888-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskgH5Pjm/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786277382073938-888-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskgH5Pjm/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786277382073938-888-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260809 12:11:25.370515 12175 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260809 12:11:25.370832 12175 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260809 12:11:25.371074 12175 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260809 12:11:25.378454 12175 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260809 12:11:25.378610 12175 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260809 12:11:25.378795 12175 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.222.1
I20260809 12:11:25.388597 12175 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.222.20:41235
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskgH5Pjm/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786277382073938-888-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskgH5Pjm/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786277382073938-888-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskgH5Pjm/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786277382073938-888-0/krb5kdc/kudu_127.0.222.1.keytab
--principal=kudu/127.0.222.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.222.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskgH5Pjm/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786277382073938-888-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.222.1
--webserver_port=0
--tserver_master_addrs=127.0.222.62:46055
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.222.1
--log_dir=/tmp/dist-test-taskgH5Pjm/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786277382073938-888-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 09 Aug 2026 11:06:41 UTC on ccycloud.quasar-bwoogy.root.comops.site
build id 4575991
ASAN enabled
I20260809 12:11:25.389946 12175 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260809 12:11:25.391489 12175 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260809 12:11:25.405351 12181 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 12:11:25.405498 12180 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 12:11:25.405375 12183 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260809 12:11:25.406322 12175 server_base.cc:1034] running on GCE node
I20260809 12:11:25.407092 12175 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260809 12:11:25.408059 12175 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260809 12:11:25.409312 12175 hybrid_clock.cc:648] HybridClock initialized: now 1786277485409259 us; error 62 us; skew 500 ppm
I20260809 12:11:25.409648 12175 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 09 12:11:25 dist-test-slave-2w3w krb5kdc[12060](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786277485, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.222.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260809 12:11:25.413143 12175 init.cc:377] Logged in from keytab as kudu/127.0.222.1@KRBTEST.COM (short username kudu)
I20260809 12:11:25.414592 12175 webserver.cc:460] Webserver started at http://127.0.222.1:36619/ using document root <none> and password file <none>
I20260809 12:11:25.415423 12175 fs_manager.cc:359] Metadata directory not provided
I20260809 12:11:25.415575 12175 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260809 12:11:25.415889 12175 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260809 12:11:25.418867 12175 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskgH5Pjm/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786277382073938-888-0/minicluster-data/ts-0/data/instance:
uuid: "b993f1c012d64e0d9f721bd47eb14283"
format_stamp: "Formatted at 2026-08-09 12:11:25 on dist-test-slave-2w3w"
server_key: "f195fdfbac70dac5c9d37635cc77077d"
server_key_iv: "b2a6a3f2092a47bb16ab2c2363378f2d"
server_key_version: "encryptionkey@0"
I20260809 12:11:25.419715 12175 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskgH5Pjm/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786277382073938-888-0/minicluster-data/ts-0/wal/instance:
uuid: "b993f1c012d64e0d9f721bd47eb14283"
format_stamp: "Formatted at 2026-08-09 12:11:25 on dist-test-slave-2w3w"
server_key: "f195fdfbac70dac5c9d37635cc77077d"
server_key_iv: "b2a6a3f2092a47bb16ab2c2363378f2d"
server_key_version: "encryptionkey@0"
I20260809 12:11:25.425024 12175 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.000s
I20260809 12:11:25.428508 12191 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260809 12:11:25.429785 12175 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.000s
I20260809 12:11:25.430002 12175 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskgH5Pjm/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786277382073938-888-0/minicluster-data/ts-0/data,/tmp/dist-test-taskgH5Pjm/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786277382073938-888-0/minicluster-data/ts-0/wal
uuid: "b993f1c012d64e0d9f721bd47eb14283"
format_stamp: "Formatted at 2026-08-09 12:11:25 on dist-test-slave-2w3w"
server_key: "f195fdfbac70dac5c9d37635cc77077d"
server_key_iv: "b2a6a3f2092a47bb16ab2c2363378f2d"
server_key_version: "encryptionkey@0"
I20260809 12:11:25.430222 12175 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskgH5Pjm/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786277382073938-888-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskgH5Pjm/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786277382073938-888-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskgH5Pjm/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786277382073938-888-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260809 12:11:25.444597 12175 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260809 12:11:25.448697 12175 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260809 12:11:25.449007 12175 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260809 12:11:25.449952 12175 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260809 12:11:25.451328 12175 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260809 12:11:25.451431 12175 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260809 12:11:25.451581 12175 ts_tablet_manager.cc:610] Registered 0 tablets
I20260809 12:11:25.451660 12175 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260809 12:11:25.486132 12175 rpc_server.cc:307] RPC server started. Bound to: 127.0.222.1:36605
I20260809 12:11:25.486182 12311 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.222.1:36605 every 8 connection(s)
I20260809 12:11:25.487551 12175 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskgH5Pjm/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786277382073938-888-0/minicluster-data/ts-0/data/info.pb
I20260809 12:11:25.488078 888 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskgH5Pjm/build/asan/bin/kudu as pid 12175
I20260809 12:11:25.488346 888 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskgH5Pjm/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786277382073938-888-0/minicluster-data/ts-0/wal/instance
I20260809 12:11:25.489197 888 external_mini_cluster.cc:1428] Setting key dbbfd7d1865af0efe3f95c1fe65d2d57
Aug 09 12:11:25 dist-test-slave-2w3w krb5kdc[12060](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786277485, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.222.1@KRBTEST.COM for kudu/127.0.222.62@KRBTEST.COM
I20260809 12:11:25.513005 12314 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 12:11:25.490560 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.222.1:53193 (local address 127.0.222.62:46055)
0809 12:11:25.491090 (+ 530us) server_negotiation.cc:206] Beginning negotiation
0809 12:11:25.491096 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0809 12:11:25.492404 (+ 1308us) server_negotiation.cc:407] Connection header received
0809 12:11:25.495439 (+ 3035us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 12:11:25.495443 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 12:11:25.495511 (+ 68us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 12:11:25.495603 (+ 92us) server_negotiation.cc:226] Negotiated authn=SASL
0809 12:11:25.497754 (+ 2151us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 12:11:25.498623 (+ 869us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 12:11:25.501164 (+ 2541us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 12:11:25.501577 (+ 413us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 12:11:25.501675 (+ 98us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 12:11:25.506080 (+ 4405us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 12:11:25.506085 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 12:11:25.506088 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0809 12:11:25.506121 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0809 12:11:25.509534 (+ 3413us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 12:11:25.510265 (+ 731us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 12:11:25.510281 (+ 16us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 12:11:25.510287 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0809 12:11:25.510438 (+ 151us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 12:11:25.510856 (+ 418us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 12:11:25.510861 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 12:11:25.510866 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0809 12:11:25.511330 (+ 464us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 12:11:25.511488 (+ 158us) server_negotiation.cc:1035] Waiting for connection context
0809 12:11:25.512217 (+ 729us) server_negotiation.cc:299] Negotiation successful
0809 12:11:25.512489 (+ 272us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":381,"thread_start_us":270,"threads_started":1}
I20260809 12:11:25.513587 12315 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 12:11:25.491222 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.222.62:46055 (local address 127.0.222.1:53193)
0809 12:11:25.492204 (+ 982us) negotiation.cc:107] Waiting for socket to connect
0809 12:11:25.492269 (+ 65us) client_negotiation.cc:174] Beginning negotiation
0809 12:11:25.495111 (+ 2842us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 12:11:25.495894 (+ 783us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 12:11:25.495908 (+ 14us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 12:11:25.496714 (+ 806us) client_negotiation.cc:189] Negotiated authn=SASL
0809 12:11:25.497568 (+ 854us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 12:11:25.497597 (+ 29us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 12:11:25.498829 (+ 1232us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 12:11:25.498837 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 12:11:25.501010 (+ 2173us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 12:11:25.501028 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 12:11:25.501825 (+ 797us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 12:11:25.501834 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 12:11:25.502179 (+ 345us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 12:11:25.502952 (+ 773us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0809 12:11:25.503014 (+ 62us) client_negotiation.cc:656] calling sasl_client_start()
0809 12:11:25.505850 (+ 2836us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0809 12:11:25.509689 (+ 3839us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0809 12:11:25.509696 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0809 12:11:25.509708 (+ 12us) client_negotiation.cc:760] calling sasl_client_step()
0809 12:11:25.510124 (+ 416us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0809 12:11:25.510546 (+ 422us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0809 12:11:25.510549 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0809 12:11:25.510551 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0809 12:11:25.510737 (+ 186us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0809 12:11:25.511483 (+ 746us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0809 12:11:25.511489 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0809 12:11:25.511815 (+ 326us) client_negotiation.cc:769] Sending connection context
0809 12:11:25.512044 (+ 229us) client_negotiation.cc:240] Negotiation successful
0809 12:11:25.512438 (+ 394us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":628,"thread_start_us":354,"threads_started":1}
I20260809 12:11:25.515372 12312 heartbeater.cc:347] Connected to a master server at 127.0.222.62:46055
I20260809 12:11:25.515682 12312 heartbeater.cc:464] Registering TS with master...
I20260809 12:11:25.516419 12312 heartbeater.cc:511] Master 127.0.222.62:46055 requested a full tablet report, sending...
I20260809 12:11:25.518591 12106 ts_manager.cc:194] Registered new tserver with Master: b993f1c012d64e0d9f721bd47eb14283 (127.0.222.1:36605)
I20260809 12:11:25.520119 12106 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.222.1@KRBTEST.COM'} at 127.0.222.1:53193
WARNING: no policy specified for kudu/127.0.222.2@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.222.2@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.222.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskgH5Pjm/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786277382073938-888-0/krb5kdc/kudu_127.0.222.2.keytab.
Entry for principal kudu/127.0.222.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskgH5Pjm/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786277382073938-888-0/krb5kdc/kudu_127.0.222.2.keytab.
I20260809 12:11:25.559515 888 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskgH5Pjm/build/asan/bin/kudu
/tmp/dist-test-taskgH5Pjm/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskgH5Pjm/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786277382073938-888-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskgH5Pjm/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786277382073938-888-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskgH5Pjm/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786277382073938-888-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskgH5Pjm/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786277382073938-888-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.222.2:0
--local_ip_for_outbound_sockets=127.0.222.2
--webserver_interface=127.0.222.2
--webserver_port=0
--tserver_master_addrs=127.0.222.62:46055
--keytab_file=/tmp/dist-test-taskgH5Pjm/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786277382073938-888-0/krb5kdc/kudu_127.0.222.2.keytab
--principal=kudu/127.0.222.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.222.20:41235
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskgH5Pjm/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786277382073938-888-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskgH5Pjm/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786277382073938-888-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskgH5Pjm/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786277382073938-888-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260809 12:11:25.908738 12319 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260809 12:11:25.909061 12319 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260809 12:11:25.909235 12319 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260809 12:11:25.916631 12319 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260809 12:11:25.916796 12319 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260809 12:11:25.916975 12319 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.222.2
I20260809 12:11:25.926810 12319 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.222.20:41235
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskgH5Pjm/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786277382073938-888-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskgH5Pjm/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786277382073938-888-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskgH5Pjm/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786277382073938-888-0/krb5kdc/kudu_127.0.222.2.keytab
--principal=kudu/127.0.222.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.222.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskgH5Pjm/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786277382073938-888-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.222.2
--webserver_port=0
--tserver_master_addrs=127.0.222.62:46055
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.222.2
--log_dir=/tmp/dist-test-taskgH5Pjm/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786277382073938-888-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 09 Aug 2026 11:06:41 UTC on ccycloud.quasar-bwoogy.root.comops.site
build id 4575991
ASAN enabled
I20260809 12:11:25.928227 12319 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260809 12:11:25.929636 12319 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260809 12:11:25.943440 12326 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 12:11:25.943396 12328 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 12:11:25.943490 12325 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260809 12:11:25.944553 12319 server_base.cc:1034] running on GCE node
I20260809 12:11:25.945163 12319 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260809 12:11:25.946111 12319 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260809 12:11:25.947355 12319 hybrid_clock.cc:648] HybridClock initialized: now 1786277485947307 us; error 53 us; skew 500 ppm
I20260809 12:11:25.947690 12319 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 09 12:11:25 dist-test-slave-2w3w krb5kdc[12060](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786277485, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.222.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260809 12:11:25.951303 12319 init.cc:377] Logged in from keytab as kudu/127.0.222.2@KRBTEST.COM (short username kudu)
I20260809 12:11:25.952932 12319 webserver.cc:460] Webserver started at http://127.0.222.2:37767/ using document root <none> and password file <none>
I20260809 12:11:25.953595 12319 fs_manager.cc:359] Metadata directory not provided
I20260809 12:11:25.953698 12319 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260809 12:11:25.953976 12319 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260809 12:11:25.957314 12319 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskgH5Pjm/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786277382073938-888-0/minicluster-data/ts-1/data/instance:
uuid: "a53ba76a284142b59b14269e0c4f5b79"
format_stamp: "Formatted at 2026-08-09 12:11:25 on dist-test-slave-2w3w"
server_key: "a703f4e34a5ada8741e105e648e7665d"
server_key_iv: "fd7db03410681962ec53bb371dafb83b"
server_key_version: "encryptionkey@0"
I20260809 12:11:25.958323 12319 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskgH5Pjm/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786277382073938-888-0/minicluster-data/ts-1/wal/instance:
uuid: "a53ba76a284142b59b14269e0c4f5b79"
format_stamp: "Formatted at 2026-08-09 12:11:25 on dist-test-slave-2w3w"
server_key: "a703f4e34a5ada8741e105e648e7665d"
server_key_iv: "fd7db03410681962ec53bb371dafb83b"
server_key_version: "encryptionkey@0"
I20260809 12:11:25.963585 12319 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.007s sys 0.000s
I20260809 12:11:25.966943 12336 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260809 12:11:25.968640 12319 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260809 12:11:25.968863 12319 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskgH5Pjm/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786277382073938-888-0/minicluster-data/ts-1/data,/tmp/dist-test-taskgH5Pjm/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786277382073938-888-0/minicluster-data/ts-1/wal
uuid: "a53ba76a284142b59b14269e0c4f5b79"
format_stamp: "Formatted at 2026-08-09 12:11:25 on dist-test-slave-2w3w"
server_key: "a703f4e34a5ada8741e105e648e7665d"
server_key_iv: "fd7db03410681962ec53bb371dafb83b"
server_key_version: "encryptionkey@0"
I20260809 12:11:25.969081 12319 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskgH5Pjm/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786277382073938-888-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskgH5Pjm/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786277382073938-888-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskgH5Pjm/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786277382073938-888-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260809 12:11:25.985843 12319 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260809 12:11:25.989995 12319 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260809 12:11:25.990291 12319 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260809 12:11:25.991336 12319 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260809 12:11:25.992722 12319 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260809 12:11:25.992803 12319 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260809 12:11:25.992960 12319 ts_tablet_manager.cc:610] Registered 0 tablets
I20260809 12:11:25.993012 12319 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260809 12:11:26.027402 12319 rpc_server.cc:307] RPC server started. Bound to: 127.0.222.2:44389
I20260809 12:11:26.027453 12455 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.222.2:44389 every 8 connection(s)
I20260809 12:11:26.028678 12319 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskgH5Pjm/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786277382073938-888-0/minicluster-data/ts-1/data/info.pb
I20260809 12:11:26.029398 888 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskgH5Pjm/build/asan/bin/kudu as pid 12319
I20260809 12:11:26.029613 888 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskgH5Pjm/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786277382073938-888-0/minicluster-data/ts-1/wal/instance
I20260809 12:11:26.030216 888 external_mini_cluster.cc:1428] Setting key 8d29dec96070f0ad6bcb2fcc62cd4c77
Aug 09 12:11:26 dist-test-slave-2w3w krb5kdc[12060](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786277485, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.222.2@KRBTEST.COM for kudu/127.0.222.62@KRBTEST.COM
I20260809 12:11:26.051573 12458 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 12:11:26.031558 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.222.2:50881 (local address 127.0.222.62:46055)
0809 12:11:26.032126 (+ 568us) server_negotiation.cc:206] Beginning negotiation
0809 12:11:26.032133 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0809 12:11:26.033266 (+ 1133us) server_negotiation.cc:407] Connection header received
0809 12:11:26.035251 (+ 1985us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 12:11:26.035257 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 12:11:26.035349 (+ 92us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 12:11:26.035464 (+ 115us) server_negotiation.cc:226] Negotiated authn=SASL
0809 12:11:26.037000 (+ 1536us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 12:11:26.037720 (+ 720us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 12:11:26.039342 (+ 1622us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 12:11:26.039774 (+ 432us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 12:11:26.039910 (+ 136us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 12:11:26.043931 (+ 4021us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 12:11:26.043938 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 12:11:26.043941 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0809 12:11:26.043976 (+ 35us) server_negotiation.cc:922] calling sasl_server_start()
0809 12:11:26.047982 (+ 4006us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 12:11:26.048709 (+ 727us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 12:11:26.048718 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 12:11:26.048724 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0809 12:11:26.048900 (+ 176us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 12:11:26.049352 (+ 452us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 12:11:26.049361 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 12:11:26.049368 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0809 12:11:26.049821 (+ 453us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 12:11:26.050018 (+ 197us) server_negotiation.cc:1035] Waiting for connection context
0809 12:11:26.050654 (+ 636us) server_negotiation.cc:299] Negotiation successful
0809 12:11:26.051010 (+ 356us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":445,"thread_start_us":318,"threads_started":1}
I20260809 12:11:26.052050 12459 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 12:11:26.032284 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.222.62:46055 (local address 127.0.222.2:50881)
0809 12:11:26.033118 (+ 834us) negotiation.cc:107] Waiting for socket to connect
0809 12:11:26.033156 (+ 38us) client_negotiation.cc:174] Beginning negotiation
0809 12:11:26.034912 (+ 1756us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 12:11:26.035702 (+ 790us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 12:11:26.035710 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 12:11:26.036201 (+ 491us) client_negotiation.cc:189] Negotiated authn=SASL
0809 12:11:26.036828 (+ 627us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 12:11:26.036857 (+ 29us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 12:11:26.037896 (+ 1039us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 12:11:26.037902 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 12:11:26.039191 (+ 1289us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 12:11:26.039204 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 12:11:26.040023 (+ 819us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 12:11:26.040030 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 12:11:26.040257 (+ 227us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 12:11:26.040930 (+ 673us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0809 12:11:26.040957 (+ 27us) client_negotiation.cc:656] calling sasl_client_start()
0809 12:11:26.043675 (+ 2718us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0809 12:11:26.048128 (+ 4453us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0809 12:11:26.048135 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0809 12:11:26.048147 (+ 12us) client_negotiation.cc:760] calling sasl_client_step()
0809 12:11:26.048560 (+ 413us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0809 12:11:26.049023 (+ 463us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0809 12:11:26.049026 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0809 12:11:26.049028 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0809 12:11:26.049208 (+ 180us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0809 12:11:26.049946 (+ 738us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0809 12:11:26.049951 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0809 12:11:26.050264 (+ 313us) client_negotiation.cc:769] Sending connection context
0809 12:11:26.050488 (+ 224us) client_negotiation.cc:240] Negotiation successful
0809 12:11:26.050866 (+ 378us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":598,"thread_start_us":353,"threads_started":1}
I20260809 12:11:26.053640 12456 heartbeater.cc:347] Connected to a master server at 127.0.222.62:46055
I20260809 12:11:26.053936 12456 heartbeater.cc:464] Registering TS with master...
I20260809 12:11:26.054657 12456 heartbeater.cc:511] Master 127.0.222.62:46055 requested a full tablet report, sending...
I20260809 12:11:26.056257 12106 ts_manager.cc:194] Registered new tserver with Master: a53ba76a284142b59b14269e0c4f5b79 (127.0.222.2:44389)
I20260809 12:11:26.057137 12106 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.222.2@KRBTEST.COM'} at 127.0.222.2:50881
WARNING: no policy specified for kudu/127.0.222.3@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.222.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.222.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskgH5Pjm/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786277382073938-888-0/krb5kdc/kudu_127.0.222.3.keytab.
Entry for principal kudu/127.0.222.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskgH5Pjm/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786277382073938-888-0/krb5kdc/kudu_127.0.222.3.keytab.
I20260809 12:11:26.100184 888 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskgH5Pjm/build/asan/bin/kudu
/tmp/dist-test-taskgH5Pjm/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskgH5Pjm/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786277382073938-888-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskgH5Pjm/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786277382073938-888-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskgH5Pjm/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786277382073938-888-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskgH5Pjm/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786277382073938-888-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.222.3:0
--local_ip_for_outbound_sockets=127.0.222.3
--webserver_interface=127.0.222.3
--webserver_port=0
--tserver_master_addrs=127.0.222.62:46055
--keytab_file=/tmp/dist-test-taskgH5Pjm/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786277382073938-888-0/krb5kdc/kudu_127.0.222.3.keytab
--principal=kudu/127.0.222.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.222.20:41235
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskgH5Pjm/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786277382073938-888-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskgH5Pjm/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786277382073938-888-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskgH5Pjm/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786277382073938-888-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260809 12:11:26.434999 12462 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260809 12:11:26.435288 12462 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260809 12:11:26.435482 12462 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260809 12:11:26.442837 12462 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260809 12:11:26.442947 12462 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260809 12:11:26.443148 12462 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.222.3
I20260809 12:11:26.453157 12462 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.222.20:41235
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskgH5Pjm/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786277382073938-888-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskgH5Pjm/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786277382073938-888-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskgH5Pjm/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786277382073938-888-0/krb5kdc/kudu_127.0.222.3.keytab
--principal=kudu/127.0.222.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.222.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskgH5Pjm/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786277382073938-888-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.222.3
--webserver_port=0
--tserver_master_addrs=127.0.222.62:46055
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.222.3
--log_dir=/tmp/dist-test-taskgH5Pjm/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786277382073938-888-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 09 Aug 2026 11:06:41 UTC on ccycloud.quasar-bwoogy.root.comops.site
build id 4575991
ASAN enabled
I20260809 12:11:26.454507 12462 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260809 12:11:26.456033 12462 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260809 12:11:26.469290 12467 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 12:11:26.469319 12468 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 12:11:26.469345 12470 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260809 12:11:26.470345 12462 server_base.cc:1034] running on GCE node
I20260809 12:11:26.470950 12462 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260809 12:11:26.471977 12462 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260809 12:11:26.473287 12462 hybrid_clock.cc:648] HybridClock initialized: now 1786277486473212 us; error 53 us; skew 500 ppm
I20260809 12:11:26.473626 12462 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 09 12:11:26 dist-test-slave-2w3w krb5kdc[12060](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786277486, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.222.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260809 12:11:26.477483 12462 init.cc:377] Logged in from keytab as kudu/127.0.222.3@KRBTEST.COM (short username kudu)
I20260809 12:11:26.479072 12462 webserver.cc:460] Webserver started at http://127.0.222.3:41749/ using document root <none> and password file <none>
I20260809 12:11:26.479805 12462 fs_manager.cc:359] Metadata directory not provided
I20260809 12:11:26.479951 12462 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260809 12:11:26.480259 12462 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260809 12:11:26.483276 12462 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskgH5Pjm/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786277382073938-888-0/minicluster-data/ts-2/data/instance:
uuid: "4c1c66a977e240bd9c4ba8dee8a073b9"
format_stamp: "Formatted at 2026-08-09 12:11:26 on dist-test-slave-2w3w"
server_key: "b97de120befa59f429d0902ad44bea60"
server_key_iv: "108ce94d5840a5f27e65dde32f5ab66c"
server_key_version: "encryptionkey@0"
I20260809 12:11:26.484053 12462 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskgH5Pjm/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786277382073938-888-0/minicluster-data/ts-2/wal/instance:
uuid: "4c1c66a977e240bd9c4ba8dee8a073b9"
format_stamp: "Formatted at 2026-08-09 12:11:26 on dist-test-slave-2w3w"
server_key: "b97de120befa59f429d0902ad44bea60"
server_key_iv: "108ce94d5840a5f27e65dde32f5ab66c"
server_key_version: "encryptionkey@0"
I20260809 12:11:26.489562 12462 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.000s
I20260809 12:11:26.493669 12477 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260809 12:11:26.495118 12462 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260809 12:11:26.495309 12462 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskgH5Pjm/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786277382073938-888-0/minicluster-data/ts-2/data,/tmp/dist-test-taskgH5Pjm/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786277382073938-888-0/minicluster-data/ts-2/wal
uuid: "4c1c66a977e240bd9c4ba8dee8a073b9"
format_stamp: "Formatted at 2026-08-09 12:11:26 on dist-test-slave-2w3w"
server_key: "b97de120befa59f429d0902ad44bea60"
server_key_iv: "108ce94d5840a5f27e65dde32f5ab66c"
server_key_version: "encryptionkey@0"
I20260809 12:11:26.495520 12462 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskgH5Pjm/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786277382073938-888-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskgH5Pjm/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786277382073938-888-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskgH5Pjm/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786277382073938-888-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260809 12:11:26.520036 12462 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260809 12:11:26.522943 12312 heartbeater.cc:503] Master 127.0.222.62:46055 was elected leader, sending a full tablet report...
I20260809 12:11:26.524636 12462 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260809 12:11:26.525022 12462 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260809 12:11:26.526134 12462 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260809 12:11:26.527549 12462 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260809 12:11:26.527627 12462 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260809 12:11:26.527796 12462 ts_tablet_manager.cc:610] Registered 0 tablets
I20260809 12:11:26.527861 12462 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260809 12:11:26.559377 12462 rpc_server.cc:307] RPC server started. Bound to: 127.0.222.3:36479
I20260809 12:11:26.559444 12596 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.222.3:36479 every 8 connection(s)
I20260809 12:11:26.560845 12462 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskgH5Pjm/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786277382073938-888-0/minicluster-data/ts-2/data/info.pb
I20260809 12:11:26.569965 888 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskgH5Pjm/build/asan/bin/kudu as pid 12462
I20260809 12:11:26.570168 888 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskgH5Pjm/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786277382073938-888-0/minicluster-data/ts-2/wal/instance
I20260809 12:11:26.570624 888 external_mini_cluster.cc:1428] Setting key 9357cb0a94d073de03faba00fe61c04a
Aug 09 12:11:26 dist-test-slave-2w3w krb5kdc[12060](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786277486, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.222.3@KRBTEST.COM for kudu/127.0.222.62@KRBTEST.COM
I20260809 12:11:26.584429 12599 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 12:11:26.563524 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.222.3:58035 (local address 127.0.222.62:46055)
0809 12:11:26.563993 (+ 469us) server_negotiation.cc:206] Beginning negotiation
0809 12:11:26.563997 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0809 12:11:26.564939 (+ 942us) server_negotiation.cc:407] Connection header received
0809 12:11:26.566781 (+ 1842us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 12:11:26.566787 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 12:11:26.566882 (+ 95us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 12:11:26.567024 (+ 142us) server_negotiation.cc:226] Negotiated authn=SASL
0809 12:11:26.568502 (+ 1478us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 12:11:26.569306 (+ 804us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 12:11:26.571058 (+ 1752us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 12:11:26.571635 (+ 577us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 12:11:26.571774 (+ 139us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 12:11:26.578503 (+ 6729us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 12:11:26.578508 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 12:11:26.578511 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0809 12:11:26.578554 (+ 43us) server_negotiation.cc:922] calling sasl_server_start()
0809 12:11:26.581273 (+ 2719us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 12:11:26.582037 (+ 764us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 12:11:26.582041 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 12:11:26.582043 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0809 12:11:26.582183 (+ 140us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 12:11:26.582621 (+ 438us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 12:11:26.582625 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 12:11:26.582628 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0809 12:11:26.583029 (+ 401us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 12:11:26.583157 (+ 128us) server_negotiation.cc:1035] Waiting for connection context
0809 12:11:26.583897 (+ 740us) server_negotiation.cc:299] Negotiation successful
0809 12:11:26.584083 (+ 186us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":358,"thread_start_us":264,"threads_started":1}
I20260809 12:11:26.585639 12600 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 12:11:26.564034 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.222.62:46055 (local address 127.0.222.3:58035)
0809 12:11:26.564732 (+ 698us) negotiation.cc:107] Waiting for socket to connect
0809 12:11:26.564811 (+ 79us) client_negotiation.cc:174] Beginning negotiation
0809 12:11:26.566514 (+ 1703us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 12:11:26.567263 (+ 749us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 12:11:26.567274 (+ 11us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 12:11:26.567777 (+ 503us) client_negotiation.cc:189] Negotiated authn=SASL
0809 12:11:26.568328 (+ 551us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 12:11:26.568346 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 12:11:26.569468 (+ 1122us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 12:11:26.569472 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 12:11:26.570873 (+ 1401us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 12:11:26.570886 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 12:11:26.571877 (+ 991us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 12:11:26.571885 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 12:11:26.572118 (+ 233us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 12:11:26.575887 (+ 3769us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0809 12:11:26.575911 (+ 24us) client_negotiation.cc:656] calling sasl_client_start()
0809 12:11:26.578276 (+ 2365us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0809 12:11:26.581419 (+ 3143us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0809 12:11:26.581426 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0809 12:11:26.581441 (+ 15us) client_negotiation.cc:760] calling sasl_client_step()
0809 12:11:26.581905 (+ 464us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0809 12:11:26.582277 (+ 372us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0809 12:11:26.582281 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0809 12:11:26.582283 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0809 12:11:26.582470 (+ 187us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0809 12:11:26.583164 (+ 694us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0809 12:11:26.583171 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0809 12:11:26.583516 (+ 345us) client_negotiation.cc:769] Sending connection context
0809 12:11:26.583752 (+ 236us) client_negotiation.cc:240] Negotiation successful
0809 12:11:26.584282 (+ 530us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":475,"thread_start_us":274,"threads_started":1}
I20260809 12:11:26.586939 12597 heartbeater.cc:347] Connected to a master server at 127.0.222.62:46055
I20260809 12:11:26.587301 12597 heartbeater.cc:464] Registering TS with master...
I20260809 12:11:26.588037 12597 heartbeater.cc:511] Master 127.0.222.62:46055 requested a full tablet report, sending...
I20260809 12:11:26.589442 12106 ts_manager.cc:194] Registered new tserver with Master: 4c1c66a977e240bd9c4ba8dee8a073b9 (127.0.222.3:36479)
I20260809 12:11:26.590288 12106 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.222.3@KRBTEST.COM'} at 127.0.222.3:58035
I20260809 12:11:26.598569 888 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260809 12:11:26.615566 12599 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 12:11:26.603371 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41324 (local address 127.0.222.62:46055)
0809 12:11:26.603541 (+ 170us) server_negotiation.cc:206] Beginning negotiation
0809 12:11:26.603546 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0809 12:11:26.604079 (+ 533us) server_negotiation.cc:407] Connection header received
0809 12:11:26.604216 (+ 137us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 12:11:26.604219 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 12:11:26.604282 (+ 63us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 12:11:26.604366 (+ 84us) server_negotiation.cc:226] Negotiated authn=SASL
0809 12:11:26.605880 (+ 1514us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 12:11:26.606665 (+ 785us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 12:11:26.607919 (+ 1254us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 12:11:26.608320 (+ 401us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 12:11:26.608413 (+ 93us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 12:11:26.610569 (+ 2156us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 12:11:26.610574 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 12:11:26.610577 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0809 12:11:26.610610 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0809 12:11:26.613266 (+ 2656us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 12:11:26.613859 (+ 593us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 12:11:26.613863 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 12:11:26.613865 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0809 12:11:26.613997 (+ 132us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 12:11:26.614305 (+ 308us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 12:11:26.614308 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 12:11:26.614310 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0809 12:11:26.614615 (+ 305us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 12:11:26.614805 (+ 190us) server_negotiation.cc:1035] Waiting for connection context
0809 12:11:26.615138 (+ 333us) server_negotiation.cc:299] Negotiation successful
0809 12:11:26.615307 (+ 169us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":56}
I20260809 12:11:26.619704 12106 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:41324:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260809 12:11:26.626744 12106 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260809 12:11:26.664913 12616 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 12:11:26.639408 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38552 (local address 127.0.222.2:44389)
0809 12:11:26.656105 (+ 16697us) server_negotiation.cc:206] Beginning negotiation
0809 12:11:26.656113 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0809 12:11:26.656141 (+ 28us) server_negotiation.cc:407] Connection header received
0809 12:11:26.656242 (+ 101us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 12:11:26.656250 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 12:11:26.656466 (+ 216us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 12:11:26.656669 (+ 203us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0809 12:11:26.657676 (+ 1007us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 12:11:26.659429 (+ 1753us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 12:11:26.662001 (+ 2572us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 12:11:26.663562 (+ 1561us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 12:11:26.663765 (+ 203us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 12:11:26.663996 (+ 231us) server_negotiation.cc:1035] Waiting for connection context
0809 12:11:26.664385 (+ 389us) server_negotiation.cc:299] Negotiation successful
0809 12:11:26.664553 (+ 168us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":16541,"thread_start_us":16429,"threads_started":1}
I20260809 12:11:26.665068 12612 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 12:11:26.638865 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.222.2:44389 (local address 127.0.0.1:38552)
0809 12:11:26.654690 (+ 15825us) negotiation.cc:107] Waiting for socket to connect
0809 12:11:26.654719 (+ 29us) client_negotiation.cc:174] Beginning negotiation
0809 12:11:26.654881 (+ 162us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 12:11:26.656948 (+ 2067us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 12:11:26.656953 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 12:11:26.656976 (+ 23us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0809 12:11:26.657453 (+ 477us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 12:11:26.657465 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 12:11:26.659587 (+ 2122us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 12:11:26.659594 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 12:11:26.661785 (+ 2191us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 12:11:26.661803 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 12:11:26.663863 (+ 2060us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 12:11:26.663871 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 12:11:26.664127 (+ 256us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 12:11:26.664184 (+ 57us) client_negotiation.cc:769] Sending connection context
0809 12:11:26.664304 (+ 120us) client_negotiation.cc:240] Negotiation successful
0809 12:11:26.664541 (+ 237us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":15681,"mutex_wait_us":96,"thread_start_us":19090,"threads_started":1}
I20260809 12:11:26.665093 12615 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 12:11:26.638729 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.222.1:36605 (local address 127.0.0.1:46992)
0809 12:11:26.648767 (+ 10038us) negotiation.cc:107] Waiting for socket to connect
0809 12:11:26.648849 (+ 82us) client_negotiation.cc:174] Beginning negotiation
0809 12:11:26.649132 (+ 283us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 12:11:26.656546 (+ 7414us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 12:11:26.656556 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 12:11:26.656605 (+ 49us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0809 12:11:26.657096 (+ 491us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 12:11:26.657110 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 12:11:26.659593 (+ 2483us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 12:11:26.659600 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 12:11:26.662022 (+ 2422us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 12:11:26.662042 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 12:11:26.663957 (+ 1915us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 12:11:26.663964 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 12:11:26.664132 (+ 168us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 12:11:26.664176 (+ 44us) client_negotiation.cc:769] Sending connection context
0809 12:11:26.664264 (+ 88us) client_negotiation.cc:240] Negotiation successful
0809 12:11:26.664541 (+ 277us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":9653,"mutex_wait_us":7,"thread_start_us":15628,"threads_started":1}
I20260809 12:11:26.666296 12613 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 12:11:26.639339 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.222.3:36479 (local address 127.0.0.1:41852)
0809 12:11:26.658224 (+ 18885us) negotiation.cc:107] Waiting for socket to connect
0809 12:11:26.658244 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0809 12:11:26.658391 (+ 147us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 12:11:26.658892 (+ 501us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 12:11:26.658897 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 12:11:26.658923 (+ 26us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0809 12:11:26.659339 (+ 416us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 12:11:26.659349 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 12:11:26.661414 (+ 2065us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 12:11:26.661420 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 12:11:26.663291 (+ 1871us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 12:11:26.663307 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 12:11:26.665500 (+ 2193us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 12:11:26.665509 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 12:11:26.665718 (+ 209us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 12:11:26.665753 (+ 35us) client_negotiation.cc:769] Sending connection context
0809 12:11:26.665844 (+ 91us) client_negotiation.cc:240] Negotiation successful
0809 12:11:26.665970 (+ 126us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":18765,"thread_start_us":8965,"threads_started":1}
I20260809 12:11:26.666452 12617 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 12:11:26.639935 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41852 (local address 127.0.222.3:36479)
0809 12:11:26.655207 (+ 15272us) server_negotiation.cc:206] Beginning negotiation
0809 12:11:26.655220 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0809 12:11:26.658364 (+ 3144us) server_negotiation.cc:407] Connection header received
0809 12:11:26.658534 (+ 170us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 12:11:26.658540 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 12:11:26.658709 (+ 169us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 12:11:26.658901 (+ 192us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0809 12:11:26.659516 (+ 615us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 12:11:26.661237 (+ 1721us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 12:11:26.663496 (+ 2259us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 12:11:26.664954 (+ 1458us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 12:11:26.665099 (+ 145us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 12:11:26.665337 (+ 238us) server_negotiation.cc:1035] Waiting for connection context
0809 12:11:26.665919 (+ 582us) server_negotiation.cc:299] Negotiation successful
0809 12:11:26.666102 (+ 183us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":15034,"thread_start_us":14825,"threads_started":1}
I20260809 12:11:26.667315 12614 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 12:11:26.638938 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46992 (local address 127.0.222.1:36605)
0809 12:11:26.655784 (+ 16846us) server_negotiation.cc:206] Beginning negotiation
0809 12:11:26.655795 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0809 12:11:26.655828 (+ 33us) server_negotiation.cc:407] Connection header received
0809 12:11:26.655941 (+ 113us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 12:11:26.655951 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 12:11:26.656266 (+ 315us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 12:11:26.657344 (+ 1078us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0809 12:11:26.657594 (+ 250us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 12:11:26.659387 (+ 1793us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 12:11:26.662203 (+ 2816us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 12:11:26.663693 (+ 1490us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 12:11:26.665185 (+ 1492us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 12:11:26.665557 (+ 372us) server_negotiation.cc:1035] Waiting for connection context
0809 12:11:26.665636 (+ 79us) server_negotiation.cc:299] Negotiation successful
0809 12:11:26.667077 (+ 1441us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":16672,"thread_start_us":16490,"threads_started":1}
I20260809 12:11:26.669994 12529 tablet_service.cc:1467] Processing CreateTablet for tablet c706ff8c265d4ba5beb487f92dacff59 (DEFAULT_TABLE table=test-table [id=cfd81d5575e34e9ea46bae3f602f6c89]), partition=RANGE (key) PARTITION UNBOUNDED
I20260809 12:11:26.670058 12386 tablet_service.cc:1467] Processing CreateTablet for tablet c706ff8c265d4ba5beb487f92dacff59 (DEFAULT_TABLE table=test-table [id=cfd81d5575e34e9ea46bae3f602f6c89]), partition=RANGE (key) PARTITION UNBOUNDED
I20260809 12:11:26.671413 12529 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c706ff8c265d4ba5beb487f92dacff59. 1 dirs total, 0 dirs full, 0 dirs failed
I20260809 12:11:26.670909 12241 tablet_service.cc:1467] Processing CreateTablet for tablet c706ff8c265d4ba5beb487f92dacff59 (DEFAULT_TABLE table=test-table [id=cfd81d5575e34e9ea46bae3f602f6c89]), partition=RANGE (key) PARTITION UNBOUNDED
I20260809 12:11:26.671602 12386 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c706ff8c265d4ba5beb487f92dacff59. 1 dirs total, 0 dirs full, 0 dirs failed
I20260809 12:11:26.672462 12241 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c706ff8c265d4ba5beb487f92dacff59. 1 dirs total, 0 dirs full, 0 dirs failed
I20260809 12:11:26.683274 12618 tablet_bootstrap.cc:492] T c706ff8c265d4ba5beb487f92dacff59 P 4c1c66a977e240bd9c4ba8dee8a073b9: Bootstrap starting.
I20260809 12:11:26.686539 12619 tablet_bootstrap.cc:492] T c706ff8c265d4ba5beb487f92dacff59 P a53ba76a284142b59b14269e0c4f5b79: Bootstrap starting.
I20260809 12:11:26.687162 12618 tablet_bootstrap.cc:654] T c706ff8c265d4ba5beb487f92dacff59 P 4c1c66a977e240bd9c4ba8dee8a073b9: Neither blocks nor log segments found. Creating new log.
I20260809 12:11:26.689323 12618 log.cc:824] T c706ff8c265d4ba5beb487f92dacff59 P 4c1c66a977e240bd9c4ba8dee8a073b9: Log is configured to *not* fsync() on all Append() calls
I20260809 12:11:26.690548 12619 tablet_bootstrap.cc:654] T c706ff8c265d4ba5beb487f92dacff59 P a53ba76a284142b59b14269e0c4f5b79: Neither blocks nor log segments found. Creating new log.
I20260809 12:11:26.691910 12619 log.cc:824] T c706ff8c265d4ba5beb487f92dacff59 P a53ba76a284142b59b14269e0c4f5b79: Log is configured to *not* fsync() on all Append() calls
I20260809 12:11:26.692786 12620 tablet_bootstrap.cc:492] T c706ff8c265d4ba5beb487f92dacff59 P b993f1c012d64e0d9f721bd47eb14283: Bootstrap starting.
I20260809 12:11:26.693210 12618 tablet_bootstrap.cc:492] T c706ff8c265d4ba5beb487f92dacff59 P 4c1c66a977e240bd9c4ba8dee8a073b9: No bootstrap required, opened a new log
I20260809 12:11:26.693499 12618 ts_tablet_manager.cc:1397] T c706ff8c265d4ba5beb487f92dacff59 P 4c1c66a977e240bd9c4ba8dee8a073b9: Time spent bootstrapping tablet: real 0.010s user 0.000s sys 0.008s
I20260809 12:11:26.694550 12619 tablet_bootstrap.cc:492] T c706ff8c265d4ba5beb487f92dacff59 P a53ba76a284142b59b14269e0c4f5b79: No bootstrap required, opened a new log
I20260809 12:11:26.694823 12619 ts_tablet_manager.cc:1397] T c706ff8c265d4ba5beb487f92dacff59 P a53ba76a284142b59b14269e0c4f5b79: Time spent bootstrapping tablet: real 0.009s user 0.003s sys 0.003s
I20260809 12:11:26.696388 12620 tablet_bootstrap.cc:654] T c706ff8c265d4ba5beb487f92dacff59 P b993f1c012d64e0d9f721bd47eb14283: Neither blocks nor log segments found. Creating new log.
I20260809 12:11:26.697665 12620 log.cc:824] T c706ff8c265d4ba5beb487f92dacff59 P b993f1c012d64e0d9f721bd47eb14283: Log is configured to *not* fsync() on all Append() calls
I20260809 12:11:26.703034 12619 raft_consensus.cc:348] T c706ff8c265d4ba5beb487f92dacff59 P a53ba76a284142b59b14269e0c4f5b79 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b993f1c012d64e0d9f721bd47eb14283" member_type: VOTER last_known_addr { host: "127.0.222.1" port: 36605 } } peers { permanent_uuid: "a53ba76a284142b59b14269e0c4f5b79" member_type: VOTER last_known_addr { host: "127.0.222.2" port: 44389 } } peers { permanent_uuid: "4c1c66a977e240bd9c4ba8dee8a073b9" member_type: VOTER last_known_addr { host: "127.0.222.3" port: 36479 } }
I20260809 12:11:26.703593 12619 raft_consensus.cc:374] T c706ff8c265d4ba5beb487f92dacff59 P a53ba76a284142b59b14269e0c4f5b79 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260809 12:11:26.703714 12619 raft_consensus.cc:729] T c706ff8c265d4ba5beb487f92dacff59 P a53ba76a284142b59b14269e0c4f5b79 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a53ba76a284142b59b14269e0c4f5b79, State: Initialized, Role: FOLLOWER
I20260809 12:11:26.704401 12619 consensus_queue.cc:260] T c706ff8c265d4ba5beb487f92dacff59 P a53ba76a284142b59b14269e0c4f5b79 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b993f1c012d64e0d9f721bd47eb14283" member_type: VOTER last_known_addr { host: "127.0.222.1" port: 36605 } } peers { permanent_uuid: "a53ba76a284142b59b14269e0c4f5b79" member_type: VOTER last_known_addr { host: "127.0.222.2" port: 44389 } } peers { permanent_uuid: "4c1c66a977e240bd9c4ba8dee8a073b9" member_type: VOTER last_known_addr { host: "127.0.222.3" port: 36479 } }
I20260809 12:11:26.705574 12456 heartbeater.cc:503] Master 127.0.222.62:46055 was elected leader, sending a full tablet report...
I20260809 12:11:26.706172 12620 tablet_bootstrap.cc:492] T c706ff8c265d4ba5beb487f92dacff59 P b993f1c012d64e0d9f721bd47eb14283: No bootstrap required, opened a new log
I20260809 12:11:26.706481 12620 ts_tablet_manager.cc:1397] T c706ff8c265d4ba5beb487f92dacff59 P b993f1c012d64e0d9f721bd47eb14283: Time spent bootstrapping tablet: real 0.016s user 0.008s sys 0.003s
I20260809 12:11:26.706575 12619 ts_tablet_manager.cc:1428] T c706ff8c265d4ba5beb487f92dacff59 P a53ba76a284142b59b14269e0c4f5b79: Time spent starting tablet: real 0.012s user 0.009s sys 0.005s
I20260809 12:11:26.706687 12618 raft_consensus.cc:348] T c706ff8c265d4ba5beb487f92dacff59 P 4c1c66a977e240bd9c4ba8dee8a073b9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b993f1c012d64e0d9f721bd47eb14283" member_type: VOTER last_known_addr { host: "127.0.222.1" port: 36605 } } peers { permanent_uuid: "a53ba76a284142b59b14269e0c4f5b79" member_type: VOTER last_known_addr { host: "127.0.222.2" port: 44389 } } peers { permanent_uuid: "4c1c66a977e240bd9c4ba8dee8a073b9" member_type: VOTER last_known_addr { host: "127.0.222.3" port: 36479 } }
I20260809 12:11:26.707321 12618 raft_consensus.cc:374] T c706ff8c265d4ba5beb487f92dacff59 P 4c1c66a977e240bd9c4ba8dee8a073b9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260809 12:11:26.707455 12618 raft_consensus.cc:729] T c706ff8c265d4ba5beb487f92dacff59 P 4c1c66a977e240bd9c4ba8dee8a073b9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4c1c66a977e240bd9c4ba8dee8a073b9, State: Initialized, Role: FOLLOWER
I20260809 12:11:26.708092 12618 consensus_queue.cc:260] T c706ff8c265d4ba5beb487f92dacff59 P 4c1c66a977e240bd9c4ba8dee8a073b9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b993f1c012d64e0d9f721bd47eb14283" member_type: VOTER last_known_addr { host: "127.0.222.1" port: 36605 } } peers { permanent_uuid: "a53ba76a284142b59b14269e0c4f5b79" member_type: VOTER last_known_addr { host: "127.0.222.2" port: 44389 } } peers { permanent_uuid: "4c1c66a977e240bd9c4ba8dee8a073b9" member_type: VOTER last_known_addr { host: "127.0.222.3" port: 36479 } }
I20260809 12:11:26.709308 12597 heartbeater.cc:503] Master 127.0.222.62:46055 was elected leader, sending a full tablet report...
I20260809 12:11:26.711261 12618 ts_tablet_manager.cc:1428] T c706ff8c265d4ba5beb487f92dacff59 P 4c1c66a977e240bd9c4ba8dee8a073b9: Time spent starting tablet: real 0.018s user 0.016s sys 0.000s
I20260809 12:11:26.716106 12620 raft_consensus.cc:348] T c706ff8c265d4ba5beb487f92dacff59 P b993f1c012d64e0d9f721bd47eb14283 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b993f1c012d64e0d9f721bd47eb14283" member_type: VOTER last_known_addr { host: "127.0.222.1" port: 36605 } } peers { permanent_uuid: "a53ba76a284142b59b14269e0c4f5b79" member_type: VOTER last_known_addr { host: "127.0.222.2" port: 44389 } } peers { permanent_uuid: "4c1c66a977e240bd9c4ba8dee8a073b9" member_type: VOTER last_known_addr { host: "127.0.222.3" port: 36479 } }
I20260809 12:11:26.716502 12620 raft_consensus.cc:374] T c706ff8c265d4ba5beb487f92dacff59 P b993f1c012d64e0d9f721bd47eb14283 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260809 12:11:26.716611 12620 raft_consensus.cc:729] T c706ff8c265d4ba5beb487f92dacff59 P b993f1c012d64e0d9f721bd47eb14283 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b993f1c012d64e0d9f721bd47eb14283, State: Initialized, Role: FOLLOWER
I20260809 12:11:26.717147 12620 consensus_queue.cc:260] T c706ff8c265d4ba5beb487f92dacff59 P b993f1c012d64e0d9f721bd47eb14283 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b993f1c012d64e0d9f721bd47eb14283" member_type: VOTER last_known_addr { host: "127.0.222.1" port: 36605 } } peers { permanent_uuid: "a53ba76a284142b59b14269e0c4f5b79" member_type: VOTER last_known_addr { host: "127.0.222.2" port: 44389 } } peers { permanent_uuid: "4c1c66a977e240bd9c4ba8dee8a073b9" member_type: VOTER last_known_addr { host: "127.0.222.3" port: 36479 } }
I20260809 12:11:26.718518 12620 ts_tablet_manager.cc:1428] T c706ff8c265d4ba5beb487f92dacff59 P b993f1c012d64e0d9f721bd47eb14283: Time spent starting tablet: real 0.012s user 0.013s sys 0.001s
W20260809 12:11:26.740083 12313 tablet.cc:2369] T c706ff8c265d4ba5beb487f92dacff59 P b993f1c012d64e0d9f721bd47eb14283: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260809 12:11:26.780954 12457 tablet.cc:2369] T c706ff8c265d4ba5beb487f92dacff59 P a53ba76a284142b59b14269e0c4f5b79: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260809 12:11:26.812592 12598 tablet.cc:2369] T c706ff8c265d4ba5beb487f92dacff59 P 4c1c66a977e240bd9c4ba8dee8a073b9: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260809 12:11:26.946259 12626 raft_consensus.cc:482] T c706ff8c265d4ba5beb487f92dacff59 P b993f1c012d64e0d9f721bd47eb14283 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260809 12:11:26.946594 12626 raft_consensus.cc:504] T c706ff8c265d4ba5beb487f92dacff59 P b993f1c012d64e0d9f721bd47eb14283 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b993f1c012d64e0d9f721bd47eb14283" member_type: VOTER last_known_addr { host: "127.0.222.1" port: 36605 } } peers { permanent_uuid: "a53ba76a284142b59b14269e0c4f5b79" member_type: VOTER last_known_addr { host: "127.0.222.2" port: 44389 } } peers { permanent_uuid: "4c1c66a977e240bd9c4ba8dee8a073b9" member_type: VOTER last_known_addr { host: "127.0.222.3" port: 36479 } }
I20260809 12:11:26.948429 12626 leader_election.cc:288] T c706ff8c265d4ba5beb487f92dacff59 P b993f1c012d64e0d9f721bd47eb14283 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers a53ba76a284142b59b14269e0c4f5b79 (127.0.222.2:44389), 4c1c66a977e240bd9c4ba8dee8a073b9 (127.0.222.3:36479)
I20260809 12:11:26.955382 12617 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 12:11:26.949136 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.222.1:45179 (local address 127.0.222.3:36479)
0809 12:11:26.949362 (+ 226us) server_negotiation.cc:206] Beginning negotiation
0809 12:11:26.949369 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0809 12:11:26.949945 (+ 576us) server_negotiation.cc:407] Connection header received
0809 12:11:26.950149 (+ 204us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 12:11:26.950153 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 12:11:26.950210 (+ 57us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 12:11:26.950304 (+ 94us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0809 12:11:26.950906 (+ 602us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 12:11:26.951692 (+ 786us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 12:11:26.953630 (+ 1938us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 12:11:26.954393 (+ 763us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 12:11:26.954535 (+ 142us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 12:11:26.954642 (+ 107us) server_negotiation.cc:1035] Waiting for connection context
0809 12:11:26.954996 (+ 354us) server_negotiation.cc:299] Negotiation successful
0809 12:11:26.955124 (+ 128us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":72}
I20260809 12:11:26.955497 12616 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 12:11:26.948856 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.222.1:51921 (local address 127.0.222.2:44389)
0809 12:11:26.949074 (+ 218us) server_negotiation.cc:206] Beginning negotiation
0809 12:11:26.949082 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0809 12:11:26.949919 (+ 837us) server_negotiation.cc:407] Connection header received
0809 12:11:26.950139 (+ 220us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 12:11:26.950143 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 12:11:26.950205 (+ 62us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 12:11:26.950299 (+ 94us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0809 12:11:26.951009 (+ 710us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 12:11:26.951725 (+ 716us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 12:11:26.953672 (+ 1947us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 12:11:26.954523 (+ 851us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 12:11:26.954677 (+ 154us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 12:11:26.954791 (+ 114us) server_negotiation.cc:1035] Waiting for connection context
0809 12:11:26.955108 (+ 317us) server_negotiation.cc:299] Negotiation successful
0809 12:11:26.955235 (+ 127us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":73}
I20260809 12:11:26.955567 12628 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 12:11:26.948909 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.222.3:36479 (local address 127.0.222.1:45179)
0809 12:11:26.949834 (+ 925us) negotiation.cc:107] Waiting for socket to connect
0809 12:11:26.949850 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0809 12:11:26.950003 (+ 153us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 12:11:26.950326 (+ 323us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 12:11:26.950331 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 12:11:26.950365 (+ 34us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0809 12:11:26.950746 (+ 381us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 12:11:26.950758 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 12:11:26.951826 (+ 1068us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 12:11:26.951833 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 12:11:26.953478 (+ 1645us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 12:11:26.953490 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 12:11:26.954604 (+ 1114us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 12:11:26.954609 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 12:11:26.954818 (+ 209us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 12:11:26.954843 (+ 25us) client_negotiation.cc:769] Sending connection context
0809 12:11:26.954950 (+ 107us) client_negotiation.cc:240] Negotiation successful
0809 12:11:26.955112 (+ 162us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":822,"mutex_wait_us":92,"thread_start_us":531,"threads_started":1}
I20260809 12:11:26.955667 12627 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 12:11:26.948600 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.222.2:44389 (local address 127.0.222.1:51921)
0809 12:11:26.949790 (+ 1190us) negotiation.cc:107] Waiting for socket to connect
0809 12:11:26.949810 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0809 12:11:26.949978 (+ 168us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 12:11:26.950319 (+ 341us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 12:11:26.950323 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 12:11:26.950401 (+ 78us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0809 12:11:26.950816 (+ 415us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 12:11:26.950826 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 12:11:26.951865 (+ 1039us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 12:11:26.951871 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 12:11:26.953502 (+ 1631us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 12:11:26.953515 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 12:11:26.954763 (+ 1248us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 12:11:26.954769 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 12:11:26.954931 (+ 162us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 12:11:26.954969 (+ 38us) client_negotiation.cc:769] Sending connection context
0809 12:11:26.955080 (+ 111us) client_negotiation.cc:240] Negotiation successful
0809 12:11:26.955201 (+ 121us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1043,"mutex_wait_us":60,"spinlock_wait_cycles":22784,"thread_start_us":688,"threads_started":1}
I20260809 12:11:26.956691 12549 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c706ff8c265d4ba5beb487f92dacff59" candidate_uuid: "b993f1c012d64e0d9f721bd47eb14283" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4c1c66a977e240bd9c4ba8dee8a073b9" is_pre_election: true
I20260809 12:11:26.956799 12406 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c706ff8c265d4ba5beb487f92dacff59" candidate_uuid: "b993f1c012d64e0d9f721bd47eb14283" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a53ba76a284142b59b14269e0c4f5b79" is_pre_election: true
I20260809 12:11:26.957257 12549 raft_consensus.cc:2454] T c706ff8c265d4ba5beb487f92dacff59 P 4c1c66a977e240bd9c4ba8dee8a073b9 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b993f1c012d64e0d9f721bd47eb14283 in term 0.
I20260809 12:11:26.957314 12406 raft_consensus.cc:2454] T c706ff8c265d4ba5beb487f92dacff59 P a53ba76a284142b59b14269e0c4f5b79 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b993f1c012d64e0d9f721bd47eb14283 in term 0.
I20260809 12:11:26.958000 12192 leader_election.cc:302] T c706ff8c265d4ba5beb487f92dacff59 P b993f1c012d64e0d9f721bd47eb14283 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 3 yes votes; 0 no votes. yes voters: 4c1c66a977e240bd9c4ba8dee8a073b9, a53ba76a284142b59b14269e0c4f5b79, b993f1c012d64e0d9f721bd47eb14283; no voters:
I20260809 12:11:26.958441 12626 raft_consensus.cc:2781] T c706ff8c265d4ba5beb487f92dacff59 P b993f1c012d64e0d9f721bd47eb14283 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260809 12:11:26.958588 12626 raft_consensus.cc:482] T c706ff8c265d4ba5beb487f92dacff59 P b993f1c012d64e0d9f721bd47eb14283 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260809 12:11:26.958715 12626 raft_consensus.cc:3037] T c706ff8c265d4ba5beb487f92dacff59 P b993f1c012d64e0d9f721bd47eb14283 [term 0 FOLLOWER]: Advancing to term 1
I20260809 12:11:26.961004 12626 raft_consensus.cc:504] T c706ff8c265d4ba5beb487f92dacff59 P b993f1c012d64e0d9f721bd47eb14283 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b993f1c012d64e0d9f721bd47eb14283" member_type: VOTER last_known_addr { host: "127.0.222.1" port: 36605 } } peers { permanent_uuid: "a53ba76a284142b59b14269e0c4f5b79" member_type: VOTER last_known_addr { host: "127.0.222.2" port: 44389 } } peers { permanent_uuid: "4c1c66a977e240bd9c4ba8dee8a073b9" member_type: VOTER last_known_addr { host: "127.0.222.3" port: 36479 } }
I20260809 12:11:26.961582 12626 leader_election.cc:288] T c706ff8c265d4ba5beb487f92dacff59 P b993f1c012d64e0d9f721bd47eb14283 [CANDIDATE]: Term 1 election: Requested vote from peers a53ba76a284142b59b14269e0c4f5b79 (127.0.222.2:44389), 4c1c66a977e240bd9c4ba8dee8a073b9 (127.0.222.3:36479)
I20260809 12:11:26.962025 12406 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c706ff8c265d4ba5beb487f92dacff59" candidate_uuid: "b993f1c012d64e0d9f721bd47eb14283" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a53ba76a284142b59b14269e0c4f5b79"
I20260809 12:11:26.962076 12549 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c706ff8c265d4ba5beb487f92dacff59" candidate_uuid: "b993f1c012d64e0d9f721bd47eb14283" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4c1c66a977e240bd9c4ba8dee8a073b9"
I20260809 12:11:26.962220 12406 raft_consensus.cc:3037] T c706ff8c265d4ba5beb487f92dacff59 P a53ba76a284142b59b14269e0c4f5b79 [term 0 FOLLOWER]: Advancing to term 1
I20260809 12:11:26.962276 12549 raft_consensus.cc:3037] T c706ff8c265d4ba5beb487f92dacff59 P 4c1c66a977e240bd9c4ba8dee8a073b9 [term 0 FOLLOWER]: Advancing to term 1
I20260809 12:11:26.964557 12549 raft_consensus.cc:2454] T c706ff8c265d4ba5beb487f92dacff59 P 4c1c66a977e240bd9c4ba8dee8a073b9 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b993f1c012d64e0d9f721bd47eb14283 in term 1.
I20260809 12:11:26.964581 12406 raft_consensus.cc:2454] T c706ff8c265d4ba5beb487f92dacff59 P a53ba76a284142b59b14269e0c4f5b79 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b993f1c012d64e0d9f721bd47eb14283 in term 1.
I20260809 12:11:26.965004 12192 leader_election.cc:302] T c706ff8c265d4ba5beb487f92dacff59 P b993f1c012d64e0d9f721bd47eb14283 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4c1c66a977e240bd9c4ba8dee8a073b9, b993f1c012d64e0d9f721bd47eb14283; no voters:
I20260809 12:11:26.965291 12626 raft_consensus.cc:2781] T c706ff8c265d4ba5beb487f92dacff59 P b993f1c012d64e0d9f721bd47eb14283 [term 1 FOLLOWER]: Leader election won for term 1
I20260809 12:11:26.965816 12626 raft_consensus.cc:686] T c706ff8c265d4ba5beb487f92dacff59 P b993f1c012d64e0d9f721bd47eb14283 [term 1 LEADER]: Becoming Leader. State: Replica: b993f1c012d64e0d9f721bd47eb14283, State: Running, Role: LEADER
I20260809 12:11:26.966368 12626 consensus_queue.cc:237] T c706ff8c265d4ba5beb487f92dacff59 P b993f1c012d64e0d9f721bd47eb14283 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b993f1c012d64e0d9f721bd47eb14283" member_type: VOTER last_known_addr { host: "127.0.222.1" port: 36605 } } peers { permanent_uuid: "a53ba76a284142b59b14269e0c4f5b79" member_type: VOTER last_known_addr { host: "127.0.222.2" port: 44389 } } peers { permanent_uuid: "4c1c66a977e240bd9c4ba8dee8a073b9" member_type: VOTER last_known_addr { host: "127.0.222.3" port: 36479 } }
I20260809 12:11:26.970839 12105 catalog_manager.cc:5495] T c706ff8c265d4ba5beb487f92dacff59 P b993f1c012d64e0d9f721bd47eb14283 reported cstate change: term changed from 0 to 1, leader changed from <none> to b993f1c012d64e0d9f721bd47eb14283 (127.0.222.1). New cstate: current_term: 1 leader_uuid: "b993f1c012d64e0d9f721bd47eb14283" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b993f1c012d64e0d9f721bd47eb14283" member_type: VOTER last_known_addr { host: "127.0.222.1" port: 36605 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "a53ba76a284142b59b14269e0c4f5b79" member_type: VOTER last_known_addr { host: "127.0.222.2" port: 44389 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "4c1c66a977e240bd9c4ba8dee8a073b9" member_type: VOTER last_known_addr { host: "127.0.222.3" port: 36479 } health_report { overall_health: UNKNOWN } } }
I20260809 12:11:27.027561 12614 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 12:11:27.022383 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47002 (local address 127.0.222.1:36605)
0809 12:11:27.022572 (+ 189us) server_negotiation.cc:206] Beginning negotiation
0809 12:11:27.022577 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0809 12:11:27.022595 (+ 18us) server_negotiation.cc:407] Connection header received
0809 12:11:27.022746 (+ 151us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 12:11:27.022750 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 12:11:27.022856 (+ 106us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 12:11:27.023051 (+ 195us) server_negotiation.cc:226] Negotiated authn=TOKEN
0809 12:11:27.023376 (+ 325us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 12:11:27.024082 (+ 706us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 12:11:27.025397 (+ 1315us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 12:11:27.025832 (+ 435us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 12:11:27.025929 (+ 97us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 12:11:27.026365 (+ 436us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0809 12:11:27.026977 (+ 612us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0809 12:11:27.027120 (+ 143us) server_negotiation.cc:1035] Waiting for connection context
0809 12:11:27.027242 (+ 122us) server_negotiation.cc:299] Negotiation successful
0809 12:11:27.027313 (+ 71us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":59}
I20260809 12:11:27.037509 12406 raft_consensus.cc:1261] T c706ff8c265d4ba5beb487f92dacff59 P a53ba76a284142b59b14269e0c4f5b79 [term 1 FOLLOWER]: Refusing update from remote peer b993f1c012d64e0d9f721bd47eb14283: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260809 12:11:27.037623 12549 raft_consensus.cc:1261] T c706ff8c265d4ba5beb487f92dacff59 P 4c1c66a977e240bd9c4ba8dee8a073b9 [term 1 FOLLOWER]: Refusing update from remote peer b993f1c012d64e0d9f721bd47eb14283: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260809 12:11:27.038575 12629 consensus_queue.cc:1035] T c706ff8c265d4ba5beb487f92dacff59 P b993f1c012d64e0d9f721bd47eb14283 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4c1c66a977e240bd9c4ba8dee8a073b9" member_type: VOTER last_known_addr { host: "127.0.222.3" port: 36479 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260809 12:11:27.038932 12626 consensus_queue.cc:1035] T c706ff8c265d4ba5beb487f92dacff59 P b993f1c012d64e0d9f721bd47eb14283 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a53ba76a284142b59b14269e0c4f5b79" member_type: VOTER last_known_addr { host: "127.0.222.2" port: 44389 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260809 12:11:27.053992 12632 mvcc.cc:205] Tried to move back new op lower bound from 7316592586895339520 to 7316592586618204160. Current Snapshot: MvccSnapshot[applied={T|T < 7316592586895339520}]
I20260809 12:11:27.057073 12633 mvcc.cc:205] Tried to move back new op lower bound from 7316592586895339520 to 7316592586618204160. Current Snapshot: MvccSnapshot[applied={T|T < 7316592586895339520}]
I20260809 12:11:27.067713 12105 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:41324:
table { table_name: "test-table" } modify_external_catalogs: true
I20260809 12:11:27.067971 12105 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:41324:
table { table_name: "test-table" } modify_external_catalogs: true
I20260809 12:11:27.071623 12105 catalog_manager.cc:5782] T 00000000000000000000000000000000 P d2968055ebf2465e9f45ae5dcd3d4250: Sending DeleteTablet for 3 replicas of tablet c706ff8c265d4ba5beb487f92dacff59
I20260809 12:11:27.072974 12241 tablet_service.cc:1514] Processing DeleteTablet for tablet c706ff8c265d4ba5beb487f92dacff59 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-09 12:11:27 UTC) from {username='kudu', principal='kudu/127.0.222.62@KRBTEST.COM'} at 127.0.0.1:46992
I20260809 12:11:27.073156 12386 tablet_service.cc:1514] Processing DeleteTablet for tablet c706ff8c265d4ba5beb487f92dacff59 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-09 12:11:27 UTC) from {username='kudu', principal='kudu/127.0.222.62@KRBTEST.COM'} at 127.0.0.1:38552
I20260809 12:11:27.073751 12529 tablet_service.cc:1514] Processing DeleteTablet for tablet c706ff8c265d4ba5beb487f92dacff59 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-09 12:11:27 UTC) from {username='kudu', principal='kudu/127.0.222.62@KRBTEST.COM'} at 127.0.0.1:41852
I20260809 12:11:27.074748 888 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskgH5Pjm/build/asan/bin/kudu pid 12175
I20260809 12:11:27.075913 12644 tablet_replica.cc:330] T c706ff8c265d4ba5beb487f92dacff59 P 4c1c66a977e240bd9c4ba8dee8a073b9: stopping tablet replica
I20260809 12:11:27.076443 12644 raft_consensus.cc:2229] T c706ff8c265d4ba5beb487f92dacff59 P 4c1c66a977e240bd9c4ba8dee8a073b9 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260809 12:11:27.077046 12644 raft_consensus.cc:2258] T c706ff8c265d4ba5beb487f92dacff59 P 4c1c66a977e240bd9c4ba8dee8a073b9 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260809 12:11:27.086901 12643 tablet_replica.cc:330] T c706ff8c265d4ba5beb487f92dacff59 P a53ba76a284142b59b14269e0c4f5b79: stopping tablet replica
I20260809 12:11:27.089249 12644 ts_tablet_manager.cc:1905] T c706ff8c265d4ba5beb487f92dacff59 P 4c1c66a977e240bd9c4ba8dee8a073b9: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260809 12:11:27.091065 12643 raft_consensus.cc:2229] T c706ff8c265d4ba5beb487f92dacff59 P a53ba76a284142b59b14269e0c4f5b79 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260809 12:11:27.091728 12643 raft_consensus.cc:2258] T c706ff8c265d4ba5beb487f92dacff59 P a53ba76a284142b59b14269e0c4f5b79 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260809 12:11:27.093065 12642 tablet_replica.cc:330] T c706ff8c265d4ba5beb487f92dacff59 P b993f1c012d64e0d9f721bd47eb14283: stopping tablet replica
I20260809 12:11:27.093477 12642 raft_consensus.cc:2229] T c706ff8c265d4ba5beb487f92dacff59 P b993f1c012d64e0d9f721bd47eb14283 [term 1 LEADER]: Raft consensus shutting down.
I20260809 12:11:27.094168 12642 raft_consensus.cc:2258] T c706ff8c265d4ba5beb487f92dacff59 P b993f1c012d64e0d9f721bd47eb14283 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260809 12:11:27.096472 12643 ts_tablet_manager.cc:1905] T c706ff8c265d4ba5beb487f92dacff59 P a53ba76a284142b59b14269e0c4f5b79: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260809 12:11:27.096592 12644 ts_tablet_manager.cc:1918] T c706ff8c265d4ba5beb487f92dacff59 P 4c1c66a977e240bd9c4ba8dee8a073b9: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260809 12:11:27.096773 12644 log.cc:1196] T c706ff8c265d4ba5beb487f92dacff59 P 4c1c66a977e240bd9c4ba8dee8a073b9: Deleting WAL directory at /tmp/dist-test-taskgH5Pjm/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786277382073938-888-0/minicluster-data/ts-2/wal/wals/c706ff8c265d4ba5beb487f92dacff59
I20260809 12:11:27.097419 12644 ts_tablet_manager.cc:1939] T c706ff8c265d4ba5beb487f92dacff59 P 4c1c66a977e240bd9c4ba8dee8a073b9: Deleting consensus metadata
I20260809 12:11:27.099531 12642 ts_tablet_manager.cc:1905] T c706ff8c265d4ba5beb487f92dacff59 P b993f1c012d64e0d9f721bd47eb14283: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260809 12:11:27.099494 12090 catalog_manager.cc:4841] TS 4c1c66a977e240bd9c4ba8dee8a073b9 (127.0.222.3:36479): tablet c706ff8c265d4ba5beb487f92dacff59 (table test-table [id=cfd81d5575e34e9ea46bae3f602f6c89]) successfully deleted
Aug 09 12:11:27 dist-test-slave-2w3w krb5kdc[12060](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786277484, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.222.1@KRBTEST.COM
I20260809 12:11:27.103629 12643 ts_tablet_manager.cc:1918] T c706ff8c265d4ba5beb487f92dacff59 P a53ba76a284142b59b14269e0c4f5b79: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260809 12:11:27.103813 12643 log.cc:1196] T c706ff8c265d4ba5beb487f92dacff59 P a53ba76a284142b59b14269e0c4f5b79: Deleting WAL directory at /tmp/dist-test-taskgH5Pjm/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786277382073938-888-0/minicluster-data/ts-1/wal/wals/c706ff8c265d4ba5beb487f92dacff59
I20260809 12:11:27.104429 12643 ts_tablet_manager.cc:1939] T c706ff8c265d4ba5beb487f92dacff59 P a53ba76a284142b59b14269e0c4f5b79: Deleting consensus metadata
I20260809 12:11:27.106402 12091 catalog_manager.cc:4841] TS a53ba76a284142b59b14269e0c4f5b79 (127.0.222.2:44389): tablet c706ff8c265d4ba5beb487f92dacff59 (table test-table [id=cfd81d5575e34e9ea46bae3f602f6c89]) successfully deleted
I20260809 12:11:27.108006 12642 ts_tablet_manager.cc:1918] T c706ff8c265d4ba5beb487f92dacff59 P b993f1c012d64e0d9f721bd47eb14283: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260809 12:11:27.108201 12642 log.cc:1196] T c706ff8c265d4ba5beb487f92dacff59 P b993f1c012d64e0d9f721bd47eb14283: Deleting WAL directory at /tmp/dist-test-taskgH5Pjm/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786277382073938-888-0/minicluster-data/ts-0/wal/wals/c706ff8c265d4ba5beb487f92dacff59
I20260809 12:11:27.108683 12642 ts_tablet_manager.cc:1939] T c706ff8c265d4ba5beb487f92dacff59 P b993f1c012d64e0d9f721bd47eb14283: Deleting consensus metadata
I20260809 12:11:27.110443 12093 catalog_manager.cc:4841] TS b993f1c012d64e0d9f721bd47eb14283 (127.0.222.1:36605): tablet c706ff8c265d4ba5beb487f92dacff59 (table test-table [id=cfd81d5575e34e9ea46bae3f602f6c89]) successfully deleted
I20260809 12:11:27.112130 12614 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 12:11:27.077575 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47010 (local address 127.0.222.1:36605)
0809 12:11:27.077797 (+ 222us) server_negotiation.cc:206] Beginning negotiation
0809 12:11:27.077806 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0809 12:11:27.087527 (+ 9721us) server_negotiation.cc:407] Connection header received
0809 12:11:27.087766 (+ 239us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 12:11:27.087775 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 12:11:27.087975 (+ 200us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 12:11:27.088121 (+ 146us) server_negotiation.cc:226] Negotiated authn=SASL
0809 12:11:27.090434 (+ 2313us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 12:11:27.092548 (+ 2114us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 12:11:27.094515 (+ 1967us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 12:11:27.095384 (+ 869us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 12:11:27.095521 (+ 137us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 12:11:27.103526 (+ 8005us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 12:11:27.103543 (+ 17us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 12:11:27.103561 (+ 18us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0809 12:11:27.103722 (+ 161us) server_negotiation.cc:922] calling sasl_server_start()
0809 12:11:27.108705 (+ 4983us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 12:11:27.109625 (+ 920us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 12:11:27.109635 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 12:11:27.109639 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0809 12:11:27.109832 (+ 193us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 12:11:27.110323 (+ 491us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 12:11:27.110330 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 12:11:27.110332 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0809 12:11:27.110809 (+ 477us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 12:11:27.111098 (+ 289us) server_negotiation.cc:1035] Waiting for connection context
0809 12:11:27.111541 (+ 443us) server_negotiation.cc:299] Negotiation successful
0809 12:11:27.111806 (+ 265us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":67}
I20260809 12:11:27.112814 12307 generic_service.cc:196] Checking for leaks (request via RPC)
I20260809 12:11:27.210448 888 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskgH5Pjm/build/asan/bin/kudu with pid 12175
I20260809 12:11:27.232730 888 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskgH5Pjm/build/asan/bin/kudu pid 12319
Aug 09 12:11:27 dist-test-slave-2w3w krb5kdc[12060](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786277484, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.222.2@KRBTEST.COM
I20260809 12:11:27.247947 12616 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 12:11:27.233555 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38554 (local address 127.0.222.2:44389)
0809 12:11:27.233741 (+ 186us) server_negotiation.cc:206] Beginning negotiation
0809 12:11:27.233750 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0809 12:11:27.233836 (+ 86us) server_negotiation.cc:407] Connection header received
0809 12:11:27.234060 (+ 224us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 12:11:27.234064 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 12:11:27.234132 (+ 68us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 12:11:27.234237 (+ 105us) server_negotiation.cc:226] Negotiated authn=SASL
0809 12:11:27.235603 (+ 1366us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 12:11:27.236339 (+ 736us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 12:11:27.237597 (+ 1258us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 12:11:27.238009 (+ 412us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 12:11:27.238147 (+ 138us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 12:11:27.242248 (+ 4101us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 12:11:27.242264 (+ 16us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 12:11:27.242280 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0809 12:11:27.242316 (+ 36us) server_negotiation.cc:922] calling sasl_server_start()
0809 12:11:27.245073 (+ 2757us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 12:11:27.245766 (+ 693us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 12:11:27.245781 (+ 15us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 12:11:27.245783 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0809 12:11:27.245915 (+ 132us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 12:11:27.246309 (+ 394us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 12:11:27.246315 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 12:11:27.246318 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0809 12:11:27.246706 (+ 388us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 12:11:27.247062 (+ 356us) server_negotiation.cc:1035] Waiting for connection context
0809 12:11:27.247362 (+ 300us) server_negotiation.cc:299] Negotiation successful
0809 12:11:27.247589 (+ 227us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":64}
I20260809 12:11:27.248639 12451 generic_service.cc:196] Checking for leaks (request via RPC)
I20260809 12:11:27.339903 888 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskgH5Pjm/build/asan/bin/kudu with pid 12319
I20260809 12:11:27.360092 888 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskgH5Pjm/build/asan/bin/kudu pid 12462
Aug 09 12:11:27 dist-test-slave-2w3w krb5kdc[12060](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786277484, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.222.3@KRBTEST.COM
I20260809 12:11:27.375671 12617 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 12:11:27.360952 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41866 (local address 127.0.222.3:36479)
0809 12:11:27.361162 (+ 210us) server_negotiation.cc:206] Beginning negotiation
0809 12:11:27.361169 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0809 12:11:27.361191 (+ 22us) server_negotiation.cc:407] Connection header received
0809 12:11:27.361390 (+ 199us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 12:11:27.361394 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 12:11:27.361472 (+ 78us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 12:11:27.361582 (+ 110us) server_negotiation.cc:226] Negotiated authn=SASL
0809 12:11:27.362905 (+ 1323us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 12:11:27.363873 (+ 968us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 12:11:27.365135 (+ 1262us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 12:11:27.365536 (+ 401us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 12:11:27.365671 (+ 135us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 12:11:27.370206 (+ 4535us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 12:11:27.370222 (+ 16us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 12:11:27.370238 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0809 12:11:27.370274 (+ 36us) server_negotiation.cc:922] calling sasl_server_start()
0809 12:11:27.372997 (+ 2723us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 12:11:27.373653 (+ 656us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 12:11:27.373661 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 12:11:27.373663 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0809 12:11:27.373835 (+ 172us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 12:11:27.374226 (+ 391us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 12:11:27.374229 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 12:11:27.374231 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0809 12:11:27.374577 (+ 346us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 12:11:27.374859 (+ 282us) server_negotiation.cc:1035] Waiting for connection context
0809 12:11:27.375171 (+ 312us) server_negotiation.cc:299] Negotiation successful
0809 12:11:27.375400 (+ 229us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":63}
I20260809 12:11:27.376327 12592 generic_service.cc:196] Checking for leaks (request via RPC)
I20260809 12:11:27.466583 888 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskgH5Pjm/build/asan/bin/kudu with pid 12462
I20260809 12:11:27.486938 888 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskgH5Pjm/build/asan/bin/kudu pid 12073
I20260809 12:11:27.488174 12142 generic_service.cc:196] Checking for leaks (request via RPC)
I20260809 12:11:27.600561 888 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskgH5Pjm/build/asan/bin/kudu with pid 12073
2026-08-09T12:11:27Z chronyd exiting
[ OK ] SecurityITest.TestRequireAuthenticationSecureCluster (4669 ms)
[ RUN ] SecurityITest.TestEncryptionWithKMSIntegration
Loading random data
Initializing database '/tmp/dist-test-taskgH5Pjm/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786277382073938-888-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 09 12:11:27 dist-test-slave-2w3w krb5kdc[12652](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 09 12:11:27 dist-test-slave-2w3w krb5kdc[12652](info): set up 2 sockets
Aug 09 12:11:27 dist-test-slave-2w3w krb5kdc[12652](info): commencing operation
krb5kdc: starting...
W20260809 12:11:28.772647 888 mini_kdc.cc:121] Time spent starting KDC: real 1.125s user 0.000s sys 0.041s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskgH5Pjm/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786277382073938-888-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskgH5Pjm/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786277382073938-888-0/krb5kdc/test-user.keytab.
Aug 09 12:11:28 dist-test-slave-2w3w krb5kdc[12652](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786277488, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-09T12:11:28Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-09T12:11:28Z Disabled control of system clock
WARNING: no policy specified for rangeradmin/127.0.222.20@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "rangeradmin/127.0.222.20@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal rangeradmin/127.0.222.20@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskgH5Pjm/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786277382073938-888-0/krb5kdc/rangeradmin_127.0.222.20@KRBTEST.COM.keytab.
Entry for principal rangeradmin/127.0.222.20@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskgH5Pjm/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786277382073938-888-0/krb5kdc/rangeradmin_127.0.222.20@KRBTEST.COM.keytab.
WARNING: no policy specified for rangerlookup/127.0.222.20@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "rangerlookup/127.0.222.20@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal rangerlookup/127.0.222.20@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskgH5Pjm/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786277382073938-888-0/krb5kdc/rangerlookup_127.0.222.20@KRBTEST.COM.keytab.
Entry for principal rangerlookup/127.0.222.20@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskgH5Pjm/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786277382073938-888-0/krb5kdc/rangerlookup_127.0.222.20@KRBTEST.COM.keytab.
WARNING: no policy specified for HTTP/127.0.222.20@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "HTTP/127.0.222.20@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.0.222.20@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskgH5Pjm/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786277382073938-888-0/krb5kdc/HTTP_127.0.222.20@KRBTEST.COM.keytab.
Entry for principal HTTP/127.0.222.20@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskgH5Pjm/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786277382073938-888-0/krb5kdc/HTTP_127.0.222.20@KRBTEST.COM.keytab.
I20260809 12:11:29.188071 888 mini_postgres.cc:62] Running initdb...
The files belonging to this database system will be owned by user "slave".
This user must also own the server process.
The database cluster will be initialized with locale "C".
The default database encoding has accordingly been set to "SQL_ASCII".
The default text search configuration will be set to "english".
Data page checksums are disabled.
creating directory /tmp/dist-test-taskgH5Pjm/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786277382073938-888-0/minicluster-data/postgres ... ok
creating subdirectories ... ok
selecting dynamic shared memory implementation ... posix
selecting default max_connections ... 100
selecting default shared_buffers ... 128MB
selecting default time zone ... Etc/UTC
creating configuration files ... ok
running bootstrap script ... ok
performing post-bootstrap initialization ... ok
syncing data to disk ... ok
initdb: warning: enabling "trust" authentication for local connections
You can change this by editing pg_hba.conf or using the option -A, or
--auth-local and --auth-host, the next time you run initdb.
Success. You can now start the database server using:
/tmp/dist-test-taskgH5Pjm/build/asan/bin/postgres/pg_ctl -D /tmp/dist-test-taskgH5Pjm/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786277382073938-888-0/minicluster-data/postgres -l logfile start
2026-08-09 12:11:31.801 UTC [12685] LOG: starting PostgreSQL 12.2 on x86_64-pc-linux-gnu, compiled by cc (GCC) 8.3.1 20190311 (Red Hat 8.3.1-3), 64-bit
2026-08-09 12:11:31.801 UTC [12685] LOG: listening on IPv4 address "127.0.222.20", port 49343
2026-08-09 12:11:31.805 UTC [12685] LOG: listening on Unix socket "/tmp/.s.PGSQL.49343"
2026-08-09 12:11:31.828 UTC [12688] LOG: database system was shut down at 2026-08-09 12:11:30 UTC
2026-08-09 12:11:31.831 UTC [12685] LOG: database system is ready to accept connections
I20260809 12:11:32.856544 888 mini_postgres.cc:96] Postgres bound to 49343
2026-08-09 12:11:32.883 UTC [12697] FATAL: database "slave" does not exist
127.0.222.20:49343 - accepting connections
I20260809 12:11:32.884079 888 mini_ranger.cc:161] Starting Ranger...
I20260809 12:11:32.920974 888 mini_ranger.cc:84] Created miniranger Postgres user
I20260809 12:11:33.414400 888 mini_ranger.cc:87] Created ranger Postgres database
I20260809 12:11:33.414525 888 mini_ranger.cc:178] Starting Ranger out of /tmp/dist-test-taskgH5Pjm/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786277382073938-888-0/minicluster-data/ranger-admin
2026-08-09 12:11:33,818 [I] DB FLAVOR :POSTGRES
2026-08-09 12:11:33,819 [I] --------- Verifying Ranger DB connection ---------
2026-08-09 12:11:33,819 [I] Checking connection..
2026-08-09 12:11:33,819 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskgH5Pjm/build/asan/bin/postgresql.jar:/tmp/dist-test-taskgH5Pjm/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.222.20:49343/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select 1;"
2026-08-09 12:11:34,079 [I] Checking connection passed.
2026-08-09 12:11:34,079 [I] --------- Verifying version history table ---------
2026-08-09 12:11:34,079 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskgH5Pjm/build/asan/bin/postgresql.jar:/tmp/dist-test-taskgH5Pjm/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.222.20:49343/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_db_version_h') as temp;"
2026-08-09 12:11:34,329 [I] Table x_db_version_h does not exist in database ranger
2026-08-09 12:11:34,329 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskgH5Pjm/build/asan/bin/postgresql.jar:/tmp/dist-test-taskgH5Pjm/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.222.20:49343/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_db_version_h') as temp;"
2026-08-09 12:11:34,593 [I] Table x_db_version_h does not exist in database ranger
2026-08-09 12:11:34,593 [I] Importing x_db_version_h table schema to database ranger from file: create_dbversion_catalog.sql
2026-08-09 12:11:34,594 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskgH5Pjm/build/asan/bin/postgresql.jar:/tmp/dist-test-taskgH5Pjm/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.222.20:49343/ranger -u miniranger -p '********' -noheader -trim -c \; -input /tmp/dist-test-taskgH5Pjm/thirdparty/src/ranger-2.1.0-admin/db/postgres/create_dbversion_catalog.sql
2026-08-09 12:11:34.856 UTC [12768] WARNING: there is no transaction in progress
2026-08-09 12:11:34,864 [I] create_dbversion_catalog.sql file imported successfully
2026-08-09 12:11:34,864 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskgH5Pjm/build/asan/bin/postgresql.jar:/tmp/dist-test-taskgH5Pjm/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.222.20:49343/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_db_version_h') as temp;"
2026-08-09 12:11:35,136 [I] Table x_db_version_h already exists in database 'ranger'
2026-08-09 12:11:35,136 [I] --------- Importing Ranger Core DB Schema ---------
2026-08-09 12:11:35,136 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskgH5Pjm/build/asan/bin/postgresql.jar:/tmp/dist-test-taskgH5Pjm/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.222.20:49343/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'CORE_DB_SCHEMA' and active = 'Y';"
2026-08-09 12:11:35,381 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskgH5Pjm/build/asan/bin/postgresql.jar:/tmp/dist-test-taskgH5Pjm/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.222.20:49343/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'CORE_DB_SCHEMA' and active = 'N';"
2026-08-09 12:11:35,628 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskgH5Pjm/build/asan/bin/postgresql.jar:/tmp/dist-test-taskgH5Pjm/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.222.20:49343/ranger -u miniranger -p '********' -noheader -trim -c \; -query "insert into x_db_version_h (version, inst_at, inst_by, updated_at, updated_by,active) values ('CORE_DB_SCHEMA', current_timestamp, 'Ranger 2.1.0', current_timestamp, 'dist-test-slave-2w3w.c.gcp-upstream.internal','N') ;"
2026-08-09 12:11:35,893 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskgH5Pjm/build/asan/bin/postgresql.jar:/tmp/dist-test-taskgH5Pjm/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.222.20:49343/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_portal_user') as temp;"
2026-08-09 12:11:36,164 [I] Table x_portal_user does not exist in database ranger
2026-08-09 12:11:36,165 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskgH5Pjm/build/asan/bin/postgresql.jar:/tmp/dist-test-taskgH5Pjm/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.222.20:49343/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_policy_ref_group') as temp;"
2026-08-09 12:11:36,413 [I] Table x_policy_ref_group does not exist in database ranger
2026-08-09 12:11:36,414 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskgH5Pjm/build/asan/bin/postgresql.jar:/tmp/dist-test-taskgH5Pjm/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.222.20:49343/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'DB_PATCHES' and active = 'Y';"
2026-08-09 12:11:36,682 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskgH5Pjm/build/asan/bin/postgresql.jar:/tmp/dist-test-taskgH5Pjm/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.222.20:49343/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'JAVA_PATCHES' and active = 'Y';"
2026-08-09 12:11:36,923 [I] Importing DB schema to database ranger from file: ranger_core_db_postgres.sql
2026-08-09 12:11:36,923 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskgH5Pjm/build/asan/bin/postgresql.jar:/tmp/dist-test-taskgH5Pjm/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.222.20:49343/ranger -u miniranger -p '********' -noheader -trim -c \; -input /tmp/dist-test-taskgH5Pjm/thirdparty/src/ranger-2.1.0-admin/db/postgres/optimized/current/ranger_core_db_postgres.sql
2026-08-09 12:11:37.894 UTC [12883] WARNING: there is no transaction in progress
2026-08-09 12:11:37.913 UTC [12883] WARNING: there is no transaction in progress
2026-08-09 12:11:37.930 UTC [12883] WARNING: there is no transaction in progress
2026-08-09 12:11:37.947 UTC [12883] WARNING: there is no transaction in progress
2026-08-09 12:11:37.964 UTC [12883] WARNING: there is no transaction in progress
2026-08-09 12:11:37.985 UTC [12883] WARNING: there is no transaction in progress
2026-08-09 12:11:38.041 UTC [12883] WARNING: there is no transaction in progress
2026-08-09 12:11:38.049 UTC [12883] WARNING: there is no transaction in progress
2026-08-09 12:11:38.063 UTC [12883] WARNING: there is no transaction in progress
2026-08-09 12:11:38.074 UTC [12883] WARNING: there is no transaction in progress
2026-08-09 12:11:38.084 UTC [12883] WARNING: there is no transaction in progress
2026-08-09 12:11:38.099 UTC [12883] WARNING: there is no transaction in progress
2026-08-09 12:11:38.108 UTC [12883] WARNING: there is no transaction in progress
2026-08-09 12:11:38.115 UTC [12883] WARNING: there is no transaction in progress
2026-08-09 12:11:38.123 UTC [12883] WARNING: there is no transaction in progress
2026-08-09 12:11:38.130 UTC [12883] WARNING: there is no transaction in progress
2026-08-09 12:11:38,766 [I] ranger_core_db_postgres.sql file imported successfully
2026-08-09 12:11:38,767 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskgH5Pjm/build/asan/bin/postgresql.jar:/tmp/dist-test-taskgH5Pjm/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.222.20:49343/ranger -u miniranger -p '********' -noheader -trim -c \; -query "update x_db_version_h set inst_by='Ranger 2.1.0' where active='Y' and updated_by='localhost';"
2026-08-09 12:11:39,012 [I] Patches status entries updated from base ranger version to current installed ranger version:Ranger 2.1.0
2026-08-09 12:11:39,012 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskgH5Pjm/build/asan/bin/postgresql.jar:/tmp/dist-test-taskgH5Pjm/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.222.20:49343/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_portal_user') as temp;"
2026-08-09 12:11:39,281 [I] Table x_portal_user already exists in database 'ranger'
2026-08-09 12:11:39,281 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskgH5Pjm/build/asan/bin/postgresql.jar:/tmp/dist-test-taskgH5Pjm/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.222.20:49343/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_policy_ref_group') as temp;"
2026-08-09 12:11:39,540 [I] Table x_policy_ref_group already exists in database 'ranger'
2026-08-09 12:11:39,541 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskgH5Pjm/build/asan/bin/postgresql.jar:/tmp/dist-test-taskgH5Pjm/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.222.20:49343/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'DB_PATCHES' and active = 'Y';"
2026-08-09 12:11:39,802 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskgH5Pjm/build/asan/bin/postgresql.jar:/tmp/dist-test-taskgH5Pjm/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.222.20:49343/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'JAVA_PATCHES' and active = 'Y';"
2026-08-09 12:11:40,041 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskgH5Pjm/build/asan/bin/postgresql.jar:/tmp/dist-test-taskgH5Pjm/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.222.20:49343/ranger -u miniranger -p '********' -noheader -trim -c \; -query "update x_db_version_h set active='Y' where version='CORE_DB_SCHEMA' and active='N' and updated_by='dist-test-slave-2w3w.c.gcp-upstream.internal';"
2026-08-09 12:11:40,299 [I] CORE_DB_SCHEMA import status has been updated
2026-08-09 12:11:40,300 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskgH5Pjm/build/asan/bin/postgresql.jar:/tmp/dist-test-taskgH5Pjm/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.222.20:49343/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'DB_PATCHES' and inst_by = 'Ranger 2.1.0' and active = 'Y';"
2026-08-09 12:11:40,552 [I] DB_PATCHES have already been applied
I20260809 12:11:40.558760 888 mini_ranger.cc:191] Using Ranger class path: /tmp/dist-test-taskgH5Pjm/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786277382073938-888-0/minicluster-data/ranger-admin:/tmp/dist-test-taskgH5Pjm/thirdparty/src/ranger-2.1.0-admin/ews/lib/*:/usr/lib/jvm/java-1.8.0-openjdk-amd64/lib/*:/tmp/dist-test-taskgH5Pjm/thirdparty/src/hadoop-3.2.0/*:/tmp/dist-test-taskgH5Pjm/build/asan/bin/postgresql.jar:/tmp/dist-test-taskgH5Pjm/thirdparty/src/ranger-2.1.0-admin/ews/webapp
I20260809 12:11:40.558851 888 mini_ranger.cc:193] Using host: 127.0.222.20
I20260809 12:11:40.581454 888 mini_ranger.cc:227] Ranger admin URL: http://127.0.222.20:49993
Aug 09, 2026 12:11:40 PM org.apache.ranger.server.tomcat.EmbeddedServer getKeyManagers
WARNING: Config 'ranger.keystore.file' or 'ranger.service.https.attrib.keystore.file' is not found or contains blank value
Aug 09, 2026 12:11:40 PM org.apache.ranger.server.tomcat.EmbeddedServer getTrustManagers
WARNING: Config 'ranger.truststore.file' is not found or contains blank value!
Aug 09, 2026 12:11:41 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Deriving webapp folder from catalina.base property. folder=/tmp/dist-test-taskgH5Pjm/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786277382073938-888-0/minicluster-data/ranger-admin/ews/webapp
Aug 09, 2026 12:11:41 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Webapp file =/tmp/dist-test-taskgH5Pjm/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786277382073938-888-0/minicluster-data/ranger-admin/ews/webapp, webAppName = /
Aug 09, 2026 12:11:41 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Adding webapp [/] = path [/tmp/dist-test-taskgH5Pjm/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786277382073938-888-0/minicluster-data/ranger-admin/ews/webapp] .....
Aug 09, 2026 12:11:41 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Finished init of webapp [/] = path [/tmp/dist-test-taskgH5Pjm/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786277382073938-888-0/minicluster-data/ranger-admin/ews/webapp].
Aug 09, 2026 12:11:41 PM org.apache.ranger.server.tomcat.EmbeddedServer startServer
INFO: Server Name : miniranger
I20260809 12:11:41.785774 888 mini_ranger.cc:160] Time spent starting Ranger: real 8.902s user 0.002s sys 0.066s
Aug 09, 2026 12:12:02 PM com.sun.jersey.api.core.PackagesResourceConfig init
INFO: Scanning for root resource and provider classes in the packages:
org.apache.ranger.rest
org.apache.ranger.common
xa.rest
Aug 09, 2026 12:12:02 PM com.sun.jersey.api.core.ScanningResourceConfig logClasses
INFO: Root resource classes found:
class org.apache.ranger.rest.MetricsREST
class org.apache.ranger.rest.XKeyREST
class org.apache.ranger.rest.XAuditREST
class org.apache.ranger.rest.XUserREST
class org.apache.ranger.rest.UserREST
class org.apache.ranger.rest.ServiceREST
class org.apache.ranger.rest.PublicAPIs
class org.apache.ranger.rest.TagREST
class org.apache.ranger.rest.AssetREST
class org.apache.ranger.rest.RoleREST
class org.apache.ranger.rest.SecurityZoneREST
class org.apache.ranger.rest.PublicAPIsv2
Aug 09, 2026 12:12:02 PM com.sun.jersey.api.core.ScanningResourceConfig logClasses
INFO: Provider classes found:
class org.apache.ranger.common.RangerJAXBContextResolver
Aug 09, 2026 12:12:02 PM com.sun.jersey.spi.spring.container.servlet.SpringServlet getContext
INFO: Using default applicationContext
Aug 09, 2026 12:12:02 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, assetREST, of type org.apache.ranger.rest.AssetREST as a root resource class
Aug 09, 2026 12:12:02 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, metricsREST, of type org.apache.ranger.rest.MetricsREST as a root resource class
Aug 09, 2026 12:12:02 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, publicAPIs, of type org.apache.ranger.rest.PublicAPIs as a root resource class
Aug 09, 2026 12:12:02 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, publicAPIsv2, of type org.apache.ranger.rest.PublicAPIsv2 as a root resource class
Aug 09, 2026 12:12:02 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, roleREST, of type org.apache.ranger.rest.RoleREST as a root resource class
Aug 09, 2026 12:12:02 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, securityZoneREST, of type org.apache.ranger.rest.SecurityZoneREST as a root resource class
Aug 09, 2026 12:12:02 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, serviceREST, of type org.apache.ranger.rest.ServiceREST as a root resource class
Aug 09, 2026 12:12:02 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, tagREST, of type org.apache.ranger.rest.TagREST as a root resource class
Aug 09, 2026 12:12:02 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, userREST, of type org.apache.ranger.rest.UserREST as a root resource class
Aug 09, 2026 12:12:02 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, XAuditREST, of type org.apache.ranger.rest.XAuditREST as a root resource class
Aug 09, 2026 12:12:02 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, XKeyREST, of type org.apache.ranger.rest.XKeyREST as a root resource class
Aug 09, 2026 12:12:02 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, XUserREST, of type org.apache.ranger.rest.XUserREST as a root resource class
Aug 09, 2026 12:12:02 PM com.sun.jersey.server.impl.application.WebApplicationImpl _initiate
INFO: Initiating Jersey application, version 'Jersey: 1.19.3 10/24/2016 03:58 PM'
Aug 09, 2026 12:12:03 PM com.sun.jersey.spi.inject.Errors processErrorMessages
WARNING: The following warnings have been detected with resource and/or provider classes:
WARNING: A HTTP GET method, public void org.apache.ranger.rest.ServiceREST.getPoliciesInExcel(javax.servlet.http.HttpServletRequest,javax.servlet.http.HttpServletResponse), MUST return a non-void type.
WARNING: A HTTP GET method, public void org.apache.ranger.rest.ServiceREST.getPoliciesInJson(javax.servlet.http.HttpServletRequest,javax.servlet.http.HttpServletResponse,java.lang.Boolean), MUST return a non-void type.
WARNING: A HTTP GET method, public void org.apache.ranger.rest.ServiceREST.getPoliciesInCsv(javax.servlet.http.HttpServletRequest,javax.servlet.http.HttpServletResponse) throws java.io.IOException, MUST return a non-void type.
I20260809 12:12:04.121249 888 mini_ranger.cc:261] Created Kudu service
WARNING: no policy specified for rangerkms/127.0.222.20@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "rangerkms/127.0.222.20@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal rangerkms/127.0.222.20@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskgH5Pjm/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786277382073938-888-0/krb5kdc/rangerkms_127.0.222.20@KRBTEST.COM.keytab.
Entry for principal rangerkms/127.0.222.20@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskgH5Pjm/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786277382073938-888-0/krb5kdc/rangerkms_127.0.222.20@KRBTEST.COM.keytab.
WARNING: no policy specified for HTTP/127.0.222.20@KRBTEST.COM; defaulting to no policy
add_principal: Principal or policy already exists while creating "HTTP/127.0.222.20@KRBTEST.COM".
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.0.222.20@KRBTEST.COM with kvno 3, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskgH5Pjm/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786277382073938-888-0/krb5kdc/HTTP_127.0.222.20@KRBTEST.COM.keytab.
Entry for principal HTTP/127.0.222.20@KRBTEST.COM with kvno 3, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskgH5Pjm/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786277382073938-888-0/krb5kdc/HTTP_127.0.222.20@KRBTEST.COM.keytab.
WARNING: no policy specified for keyadmin@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "keyadmin@KRBTEST.COM" created.
Aug 09 12:12:04 dist-test-slave-2w3w krb5kdc[12652](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786277524, etypes {rep=17 tkt=17 ses=17}, keyadmin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for keyadmin@KRBTEST.COM:
I20260809 12:12:04.343518 888 mini_ranger_kms.cc:208] Starting Ranger KMS...
I20260809 12:12:04.373088 888 mini_ranger_kms.cc:78] Created minirangerkms Postgres user
I20260809 12:12:05.248647 888 mini_ranger_kms.cc:81] Created rangerkms Postgres database
I20260809 12:12:05.248828 888 mini_ranger_kms.cc:226] Starting Ranger KMS out of /tmp/dist-test-taskgH5Pjm/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786277382073938-888-0/minicluster-data/ranger-kms
I20260809 12:12:05.248986 888 mini_ranger_kms.cc:227] Using postgres at 127.0.222.20:49343
2026-08-09 12:12:05,739 [I] DB FLAVOR :POSTGRES
2026-08-09 12:12:05,739 [I] --------- Verifying Ranger DB connection ---------
2026-08-09 12:12:05,739 [I] Checking connection
2026-08-09 12:12:05,739 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskgH5Pjm/build/asan/bin/postgresql.jar:/tmp/dist-test-taskgH5Pjm/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786277382073938-888-0/minicluster-data/ranger-kms/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.222.20:49343/rangerkms -u rangerkms -p '********' -noheader -trim -c \; -query "SELECT 1;"
2026-08-09 12:12:05,985 [I] connection success
2026-08-09 12:12:05,985 [I] --------- Verifying Ranger DB tables ---------
2026-08-09 12:12:05,985 [I] Verifying table ranger_masterkey in database rangerkms
2026-08-09 12:12:05,985 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskgH5Pjm/build/asan/bin/postgresql.jar:/tmp/dist-test-taskgH5Pjm/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786277382073938-888-0/minicluster-data/ranger-kms/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.222.20:49343/rangerkms -u rangerkms -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='rangerkms' and table_name = 'ranger_masterkey') as temp;"
2026-08-09 12:12:06,242 [I] Table ranger_masterkey does not exist in database rangerkms
2026-08-09 12:12:06,242 [I] --------- Importing Ranger Core DB Schema ---------
2026-08-09 12:12:06,243 [I] Importing db schema to database rangerkms from file: kms_core_db_postgres.sql
2026-08-09 12:12:06,243 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskgH5Pjm/build/asan/bin/postgresql.jar:/tmp/dist-test-taskgH5Pjm/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786277382073938-888-0/minicluster-data/ranger-kms/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.222.20:49343/rangerkms -u rangerkms -p '********' -noheader -trim -c \; -input /tmp/dist-test-taskgH5Pjm/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786277382073938-888-0/minicluster-data/ranger-kms/db/postgres/kms_core_db_postgres.sql
2026-08-09 12:12:06,510 [I] kms_core_db_postgres.sql DB schema imported successfully
I20260809 12:12:07.005321 888 mini_ranger_kms.cc:310] Created kms service
I20260809 12:12:07.235175 888 mini_ranger_kms.cc:326] Created kudu user
I20260809 12:12:07.285446 888 mini_ranger_kms.cc:343] Created rangerkms user
Aug 09 12:12:07 dist-test-slave-2w3w krb5kdc[12652](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786277524, etypes {rep=17 tkt=17 ses=17}, keyadmin@KRBTEST.COM for HTTP/127.0.222.20@KRBTEST.COM
I20260809 12:12:07.719189 888 mini_ranger_kms.cc:384] Added ranger policy
I20260809 12:12:07.719604 888 mini_ranger_kms.cc:238] Using RangerKMS classpath: /tmp/dist-test-taskgH5Pjm/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786277382073938-888-0/minicluster-data/ranger-kms:/tmp/dist-test-taskgH5Pjm/build/asan/bin/postgresql.jar:/tmp/dist-test-taskgH5Pjm/thirdparty/src/ranger-2.1.0-kms/ews/webapp/WEB-INF/classes/lib/*:/tmp/dist-test-taskgH5Pjm/thirdparty/src/ranger-2.1.0-kms/ews/webapp/lib/*:/usr/lib/jvm/java-1.8.0-openjdk-amd64/lib/*
I20260809 12:12:07.719683 888 mini_ranger_kms.cc:240] Using host: 127.0.222.20
I20260809 12:12:07.749362 888 mini_ranger_kms.cc:276] Ranger KMS PID: 13076
I20260809 12:12:07.749575 888 mini_ranger_kms.cc:277] Ranger KMS URL: http://127.0.222.20:46617
Aug 09, 2026 12:12:08 PM org.apache.ranger.server.tomcat.EmbeddedServer getKeyManagers
WARNING: Config 'ranger.keystore.file' or 'ranger.service.https.attrib.keystore.file' is not found or contains blank value
Aug 09, 2026 12:12:08 PM org.apache.ranger.server.tomcat.EmbeddedServer getTrustManagers
WARNING: Config 'ranger.truststore.file' is not found or contains blank value!
Aug 09, 2026 12:12:08 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Webapp file =/tmp/dist-test-taskgH5Pjm/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786277382073938-888-0/minicluster-data/ranger-kms/ews/webapp, webAppName = /kms
Aug 09, 2026 12:12:08 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Adding webapp [/kms] = path [/tmp/dist-test-taskgH5Pjm/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786277382073938-888-0/minicluster-data/ranger-kms/ews/webapp] .....
Aug 09, 2026 12:12:09 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Finished init of webapp [/kms] = path [/tmp/dist-test-taskgH5Pjm/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786277382073938-888-0/minicluster-data/ranger-kms/ews/webapp].
Aug 09, 2026 12:12:09 PM org.apache.ranger.server.tomcat.EmbeddedServer startServer
INFO: Server Name : minirangerkms
Aug 09 12:12:10 dist-test-slave-2w3w krb5kdc[12652](info): AS_REQ (1 etypes {17}) 127.0.0.1: ISSUE: authtime 1786277530, etypes {rep=17 tkt=17 ses=17}, rangerkms/127.0.222.20@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260809 12:12:10.163278 888 mini_ranger_kms.cc:207] Time spent starting Ranger KMS: real 5.820s user 0.116s sys 0.424s
I20260809 12:12:10.163655 888 mini_ranger_kms.cc:397] {"name":"kuduclusterkey","cipher":"AES/CTR/NoPadding","length":128,"description":"kuduclusterkey"}
I20260809 12:12:10.163900 888 mini_ranger_kms.cc:401] 127.0.222.20:46617/kms/v1/keys
Aug 09 12:12:10 dist-test-slave-2w3w krb5kdc[12652](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786277530, etypes {rep=17 tkt=17 ses=17}, rangerkms/127.0.222.20@KRBTEST.COM for HTTP/127.0.222.20@KRBTEST.COM
Aug 09 12:12:10 dist-test-slave-2w3w krb5kdc[12652](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786277530, etypes {rep=17 tkt=17 ses=17}, rangerkms/127.0.222.20@KRBTEST.COM for HTTP/127.0.222.20@KRBTEST.COM
Aug 09 12:12:13 dist-test-slave-2w3w krb5kdc[12652](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786277533, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
WARNING: no policy specified for kudu/127.0.222.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.222.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.222.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskgH5Pjm/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786277382073938-888-0/krb5kdc/kudu_127.0.222.62.keytab.
Entry for principal kudu/127.0.222.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskgH5Pjm/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786277382073938-888-0/krb5kdc/kudu_127.0.222.62.keytab.
I20260809 12:12:13.356014 888 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskgH5Pjm/build/asan/bin/kudu
/tmp/dist-test-taskgH5Pjm/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskgH5Pjm/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786277382073938-888-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskgH5Pjm/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786277382073938-888-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskgH5Pjm/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786277382073938-888-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskgH5Pjm/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786277382073938-888-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.222.62:39139
--webserver_interface=127.0.222.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskgH5Pjm/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786277382073938-888-0/krb5kdc/kudu_127.0.222.62.keytab
--principal=kudu/127.0.222.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.222.20:37617
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.222.62:39139
--ranger_config_path=/tmp/dist-test-taskgH5Pjm/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786277382073938-888-0/minicluster-data/ranger-client
--trusted_user_acl=test-admin
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.0.222.20:46617/kms
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskgH5Pjm/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786277382073938-888-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskgH5Pjm/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786277382073938-888-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskgH5Pjm/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786277382073938-888-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260809 12:12:14.042090 13128 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260809 12:12:14.042541 13128 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260809 12:12:14.042801 13128 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260809 12:12:14.053342 13128 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260809 12:12:14.053601 13128 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260809 12:12:14.053774 13128 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260809 12:12:14.054014 13128 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260809 12:12:14.054158 13128 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260809 12:12:14.068238 13128 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.222.20:37617
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskgH5Pjm/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786277382073938-888-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskgH5Pjm/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786277382073938-888-0/minicluster-data/master-0/wal
--ranger_kms_url=127.0.222.20:46617/kms
--trusted_user_acl=<redacted>
--ipki_ca_key_size=768
--master_addresses=127.0.222.62:39139
--ranger_config_path=/tmp/dist-test-taskgH5Pjm/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786277382073938-888-0/minicluster-data/ranger-client
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskgH5Pjm/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786277382073938-888-0/krb5kdc/kudu_127.0.222.62.keytab
--principal=kudu/127.0.222.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.222.62:39139
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskgH5Pjm/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786277382073938-888-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.222.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskgH5Pjm/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786277382073938-888-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 09 Aug 2026 11:06:41 UTC on ccycloud.quasar-bwoogy.root.comops.site
build id 4575991
ASAN enabled
I20260809 12:12:14.069815 13128 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260809 12:12:14.071465 13128 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260809 12:12:14.093947 13133 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 12:12:14.094842 13134 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 12:12:14.095028 13136 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260809 12:12:14.100803 13128 server_base.cc:1034] running on GCE node
I20260809 12:12:14.101936 13128 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260809 12:12:14.103994 13128 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260809 12:12:14.105307 13128 hybrid_clock.cc:648] HybridClock initialized: now 1786277534105207 us; error 39 us; skew 500 ppm
I20260809 12:12:14.105842 13128 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 09 12:12:14 dist-test-slave-2w3w krb5kdc[12652](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786277534, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.222.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260809 12:12:14.115108 13128 init.cc:377] Logged in from keytab as kudu/127.0.222.62@KRBTEST.COM (short username kudu)
I20260809 12:12:14.117781 13128 webserver.cc:460] Webserver started at http://127.0.222.62:34057/ using document root <none> and password file <none>
I20260809 12:12:14.118748 13128 fs_manager.cc:359] Metadata directory not provided
I20260809 12:12:14.119073 13128 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260809 12:12:14.119529 13128 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Aug 09 12:12:14 dist-test-slave-2w3w krb5kdc[12652](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786277534, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.222.62@KRBTEST.COM for HTTP/127.0.222.20@KRBTEST.COM
I20260809 12:12:14.184206 13128 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskgH5Pjm/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786277382073938-888-0/minicluster-data/master-0/data/instance:
uuid: "667e56ed1b824dfab8ec0522ff0f8b2b"
format_stamp: "Formatted at 2026-08-09 12:12:14 on dist-test-slave-2w3w"
server_key: "4923eb57b2fc03406d1be29d051320b1"
server_key_iv: "7df7ead4c48e089cea91fbb310d2bf58"
server_key_version: "kuduclusterkey@0"
I20260809 12:12:14.185328 13128 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskgH5Pjm/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786277382073938-888-0/minicluster-data/master-0/wal/instance:
uuid: "667e56ed1b824dfab8ec0522ff0f8b2b"
format_stamp: "Formatted at 2026-08-09 12:12:14 on dist-test-slave-2w3w"
server_key: "4923eb57b2fc03406d1be29d051320b1"
server_key_iv: "7df7ead4c48e089cea91fbb310d2bf58"
server_key_version: "kuduclusterkey@0"
I20260809 12:12:14.192942 13128 fs_manager.cc:691] Time spent creating directory manager: real 0.007s user 0.000s sys 0.008s
I20260809 12:12:14.217542 13144 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260809 12:12:14.219424 13128 fs_manager.cc:725] Time spent opening block manager: real 0.005s user 0.002s sys 0.005s
I20260809 12:12:14.219662 13128 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskgH5Pjm/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786277382073938-888-0/minicluster-data/master-0/data,/tmp/dist-test-taskgH5Pjm/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786277382073938-888-0/minicluster-data/master-0/wal
uuid: "667e56ed1b824dfab8ec0522ff0f8b2b"
format_stamp: "Formatted at 2026-08-09 12:12:14 on dist-test-slave-2w3w"
server_key: "4923eb57b2fc03406d1be29d051320b1"
server_key_iv: "7df7ead4c48e089cea91fbb310d2bf58"
server_key_version: "kuduclusterkey@0"
I20260809 12:12:14.220026 13128 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskgH5Pjm/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786277382073938-888-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskgH5Pjm/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786277382073938-888-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskgH5Pjm/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786277382073938-888-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260809 12:12:14.236088 13128 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260809 12:12:14.241164 13128 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260809 12:12:14.241757 13128 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260809 12:12:14.284129 13128 rpc_server.cc:307] RPC server started. Bound to: 127.0.222.62:39139
I20260809 12:12:14.284276 13198 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.222.62:39139 every 8 connection(s)
I20260809 12:12:14.285843 13128 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskgH5Pjm/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786277382073938-888-0/minicluster-data/master-0/data/info.pb
I20260809 12:12:14.291035 13199 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260809 12:12:14.294301 888 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskgH5Pjm/build/asan/bin/kudu as pid 13128
I20260809 12:12:14.294508 888 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskgH5Pjm/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786277382073938-888-0/minicluster-data/master-0/wal/instance
I20260809 12:12:14.295316 888 external_mini_cluster.cc:1428] Setting key 6309c17d98d6296a4731c8b72f390a9b
I20260809 12:12:14.307922 13199 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 667e56ed1b824dfab8ec0522ff0f8b2b: Bootstrap starting.
I20260809 12:12:14.313450 13199 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 667e56ed1b824dfab8ec0522ff0f8b2b: Neither blocks nor log segments found. Creating new log.
Aug 09 12:12:14 dist-test-slave-2w3w krb5kdc[12652](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786277533, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.222.62@KRBTEST.COM
I20260809 12:12:14.315181 13199 log.cc:824] T 00000000000000000000000000000000 P 667e56ed1b824dfab8ec0522ff0f8b2b: Log is configured to *not* fsync() on all Append() calls
I20260809 12:12:14.319507 13199 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 667e56ed1b824dfab8ec0522ff0f8b2b: No bootstrap required, opened a new log
I20260809 12:12:14.324853 13202 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 12:12:14.297827 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50226 (local address 127.0.222.62:39139)
0809 12:12:14.299565 (+ 1738us) server_negotiation.cc:206] Beginning negotiation
0809 12:12:14.299577 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0809 12:12:14.299628 (+ 51us) server_negotiation.cc:407] Connection header received
0809 12:12:14.302431 (+ 2803us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 12:12:14.302454 (+ 23us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 12:12:14.302843 (+ 389us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 12:12:14.303284 (+ 441us) server_negotiation.cc:226] Negotiated authn=SASL
0809 12:12:14.305224 (+ 1940us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 12:12:14.306920 (+ 1696us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 12:12:14.308865 (+ 1945us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 12:12:14.309615 (+ 750us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 12:12:14.309752 (+ 137us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 12:12:14.315183 (+ 5431us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 12:12:14.315193 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 12:12:14.315208 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0809 12:12:14.315246 (+ 38us) server_negotiation.cc:922] calling sasl_server_start()
0809 12:12:14.319716 (+ 4470us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 12:12:14.320428 (+ 712us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 12:12:14.320439 (+ 11us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 12:12:14.320462 (+ 23us) server_negotiation.cc:967] calling sasl_server_step()
0809 12:12:14.320700 (+ 238us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 12:12:14.321171 (+ 471us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 12:12:14.321176 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 12:12:14.321180 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0809 12:12:14.321935 (+ 755us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 12:12:14.322396 (+ 461us) server_negotiation.cc:1035] Waiting for connection context
0809 12:12:14.322856 (+ 460us) server_negotiation.cc:299] Negotiation successful
0809 12:12:14.323335 (+ 479us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1490,"thread_start_us":1078,"threads_started":1}
I20260809 12:12:14.331023 13199 raft_consensus.cc:348] T 00000000000000000000000000000000 P 667e56ed1b824dfab8ec0522ff0f8b2b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "667e56ed1b824dfab8ec0522ff0f8b2b" member_type: VOTER last_known_addr { host: "127.0.222.62" port: 39139 } }
I20260809 12:12:14.331511 13199 raft_consensus.cc:374] T 00000000000000000000000000000000 P 667e56ed1b824dfab8ec0522ff0f8b2b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260809 12:12:14.331671 13199 raft_consensus.cc:729] T 00000000000000000000000000000000 P 667e56ed1b824dfab8ec0522ff0f8b2b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 667e56ed1b824dfab8ec0522ff0f8b2b, State: Initialized, Role: FOLLOWER
I20260809 12:12:14.332469 13199 consensus_queue.cc:260] T 00000000000000000000000000000000 P 667e56ed1b824dfab8ec0522ff0f8b2b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "667e56ed1b824dfab8ec0522ff0f8b2b" member_type: VOTER last_known_addr { host: "127.0.222.62" port: 39139 } }
I20260809 12:12:14.332943 13199 raft_consensus.cc:388] T 00000000000000000000000000000000 P 667e56ed1b824dfab8ec0522ff0f8b2b [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260809 12:12:14.333191 13199 raft_consensus.cc:482] T 00000000000000000000000000000000 P 667e56ed1b824dfab8ec0522ff0f8b2b [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260809 12:12:14.333488 13199 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 667e56ed1b824dfab8ec0522ff0f8b2b [term 0 FOLLOWER]: Advancing to term 1
I20260809 12:12:14.336815 13199 raft_consensus.cc:504] T 00000000000000000000000000000000 P 667e56ed1b824dfab8ec0522ff0f8b2b [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "667e56ed1b824dfab8ec0522ff0f8b2b" member_type: VOTER last_known_addr { host: "127.0.222.62" port: 39139 } }
I20260809 12:12:14.337486 13199 leader_election.cc:302] T 00000000000000000000000000000000 P 667e56ed1b824dfab8ec0522ff0f8b2b [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 667e56ed1b824dfab8ec0522ff0f8b2b; no voters:
I20260809 12:12:14.338680 13199 leader_election.cc:288] T 00000000000000000000000000000000 P 667e56ed1b824dfab8ec0522ff0f8b2b [CANDIDATE]: Term 1 election: Requested vote from peers
I20260809 12:12:14.338800 13204 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 667e56ed1b824dfab8ec0522ff0f8b2b [term 1 FOLLOWER]: Leader election won for term 1
I20260809 12:12:14.340871 13204 raft_consensus.cc:686] T 00000000000000000000000000000000 P 667e56ed1b824dfab8ec0522ff0f8b2b [term 1 LEADER]: Becoming Leader. State: Replica: 667e56ed1b824dfab8ec0522ff0f8b2b, State: Running, Role: LEADER
I20260809 12:12:14.341692 13204 consensus_queue.cc:237] T 00000000000000000000000000000000 P 667e56ed1b824dfab8ec0522ff0f8b2b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "667e56ed1b824dfab8ec0522ff0f8b2b" member_type: VOTER last_known_addr { host: "127.0.222.62" port: 39139 } }
I20260809 12:12:14.342243 13199 sys_catalog.cc:564] T 00000000000000000000000000000000 P 667e56ed1b824dfab8ec0522ff0f8b2b [sys.catalog]: configured and running, proceeding with master startup.
I20260809 12:12:14.345716 13205 sys_catalog.cc:455] T 00000000000000000000000000000000 P 667e56ed1b824dfab8ec0522ff0f8b2b [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "667e56ed1b824dfab8ec0522ff0f8b2b" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "667e56ed1b824dfab8ec0522ff0f8b2b" member_type: VOTER last_known_addr { host: "127.0.222.62" port: 39139 } } }
I20260809 12:12:14.347098 13205 sys_catalog.cc:458] T 00000000000000000000000000000000 P 667e56ed1b824dfab8ec0522ff0f8b2b [sys.catalog]: This master's current role is: LEADER
I20260809 12:12:14.346463 13206 sys_catalog.cc:455] T 00000000000000000000000000000000 P 667e56ed1b824dfab8ec0522ff0f8b2b [sys.catalog]: SysCatalogTable state changed. Reason: New leader 667e56ed1b824dfab8ec0522ff0f8b2b. Latest consensus state: current_term: 1 leader_uuid: "667e56ed1b824dfab8ec0522ff0f8b2b" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "667e56ed1b824dfab8ec0522ff0f8b2b" member_type: VOTER last_known_addr { host: "127.0.222.62" port: 39139 } } }
I20260809 12:12:14.347564 13206 sys_catalog.cc:458] T 00000000000000000000000000000000 P 667e56ed1b824dfab8ec0522ff0f8b2b [sys.catalog]: This master's current role is: LEADER
I20260809 12:12:14.359385 13209 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260809 12:12:14.370167 13209 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260809 12:12:14.377108 13199 ranger_client.cc:318] Using new properties file: /tmp/dist-test-taskgH5Pjm/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786277382073938-888-0/minicluster-data/master-0/logs/kudu-ranger-subprocess-log4j2.properties
I20260809 12:12:14.395562 13209 catalog_manager.cc:1347] Generated new cluster ID: 6dfafffe0ec440858fa6618b20c93fc7
I20260809 12:12:14.396207 13209 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260809 12:12:14.431814 13209 catalog_manager.cc:1370] Generated new certificate authority record
I20260809 12:12:14.433804 13209 catalog_manager.cc:1504] Loading token signing keys...
I20260809 12:12:14.441850 13209 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 667e56ed1b824dfab8ec0522ff0f8b2b: Generated new TSK 0
I20260809 12:12:14.443006 13209 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.0.222.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.222.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.222.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskgH5Pjm/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786277382073938-888-0/krb5kdc/kudu_127.0.222.1.keytab.
Entry for principal kudu/127.0.222.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskgH5Pjm/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786277382073938-888-0/krb5kdc/kudu_127.0.222.1.keytab.
I20260809 12:12:15.350281 888 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskgH5Pjm/build/asan/bin/kudu
/tmp/dist-test-taskgH5Pjm/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskgH5Pjm/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786277382073938-888-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskgH5Pjm/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786277382073938-888-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskgH5Pjm/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786277382073938-888-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskgH5Pjm/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786277382073938-888-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.0.222.20:46617/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.222.1:0
--local_ip_for_outbound_sockets=127.0.222.1
--webserver_interface=127.0.222.1
--webserver_port=0
--tserver_master_addrs=127.0.222.62:39139
--keytab_file=/tmp/dist-test-taskgH5Pjm/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786277382073938-888-0/krb5kdc/kudu_127.0.222.1.keytab
--principal=kudu/127.0.222.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.222.20:37617
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskgH5Pjm/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786277382073938-888-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskgH5Pjm/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786277382073938-888-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskgH5Pjm/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786277382073938-888-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Aug 09 12:12:15 dist-test-slave-2w3w krb5kdc[12652](info): AS_REQ (1 etypes {17}) 127.0.0.1: ISSUE: authtime 1786277535, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.222.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
W20260809 12:12:15.975425 13244 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260809 12:12:15.975984 13244 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260809 12:12:15.976337 13244 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260809 12:12:15.990128 13244 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260809 12:12:15.990343 13244 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260809 12:12:15.990557 13244 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.222.1
I20260809 12:12:16.006366 13244 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.222.20:37617
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskgH5Pjm/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786277382073938-888-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskgH5Pjm/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786277382073938-888-0/minicluster-data/ts-0/wal
--ranger_kms_url=127.0.222.20:46617/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskgH5Pjm/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786277382073938-888-0/krb5kdc/kudu_127.0.222.1.keytab
--principal=kudu/127.0.222.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.222.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskgH5Pjm/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786277382073938-888-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.222.1
--webserver_port=0
--tserver_master_addrs=127.0.222.62:39139
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.222.1
--log_dir=/tmp/dist-test-taskgH5Pjm/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786277382073938-888-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 09 Aug 2026 11:06:41 UTC on ccycloud.quasar-bwoogy.root.comops.site
build id 4575991
ASAN enabled
I20260809 12:12:16.008129 13244 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260809 12:12:16.009709 13244 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260809 12:12:16.037526 13254 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 12:12:16.037829 13255 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 12:12:16.039217 13257 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260809 12:12:16.040611 13244 server_base.cc:1034] running on GCE node
I20260809 12:12:16.041344 13244 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260809 12:12:16.042680 13244 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260809 12:12:16.043982 13244 hybrid_clock.cc:648] HybridClock initialized: now 1786277536043847 us; error 84 us; skew 500 ppm
I20260809 12:12:16.044539 13244 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 09 12:12:16 dist-test-slave-2w3w krb5kdc[12652](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786277536, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.222.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260809 12:12:16.049535 13244 init.cc:377] Logged in from keytab as kudu/127.0.222.1@KRBTEST.COM (short username kudu)
I20260809 12:12:16.053892 13244 webserver.cc:460] Webserver started at http://127.0.222.1:40997/ using document root <none> and password file <none>
I20260809 12:12:16.055380 13244 fs_manager.cc:359] Metadata directory not provided
I20260809 12:12:16.055723 13244 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260809 12:12:16.056452 13244 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Aug 09 12:12:16 dist-test-slave-2w3w krb5kdc[12652](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786277536, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.222.1@KRBTEST.COM for HTTP/127.0.222.20@KRBTEST.COM
I20260809 12:12:16.080768 13244 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskgH5Pjm/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786277382073938-888-0/minicluster-data/ts-0/data/instance:
uuid: "7e10508ab9f84a08b31e30eb4cba88ba"
format_stamp: "Formatted at 2026-08-09 12:12:16 on dist-test-slave-2w3w"
server_key: "f16edaf4c694338b80604c51100faed5"
server_key_iv: "b8b79cfe47aabfd7e5a279a4f22d124b"
server_key_version: "kuduclusterkey@0"
I20260809 12:12:16.082296 13244 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskgH5Pjm/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786277382073938-888-0/minicluster-data/ts-0/wal/instance:
uuid: "7e10508ab9f84a08b31e30eb4cba88ba"
format_stamp: "Formatted at 2026-08-09 12:12:16 on dist-test-slave-2w3w"
server_key: "f16edaf4c694338b80604c51100faed5"
server_key_iv: "b8b79cfe47aabfd7e5a279a4f22d124b"
server_key_version: "kuduclusterkey@0"
I20260809 12:12:16.095403 13244 fs_manager.cc:691] Time spent creating directory manager: real 0.012s user 0.007s sys 0.001s
I20260809 12:12:16.132215 13266 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260809 12:12:16.135262 13244 fs_manager.cc:725] Time spent opening block manager: real 0.018s user 0.009s sys 0.008s
I20260809 12:12:16.135607 13244 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskgH5Pjm/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786277382073938-888-0/minicluster-data/ts-0/data,/tmp/dist-test-taskgH5Pjm/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786277382073938-888-0/minicluster-data/ts-0/wal
uuid: "7e10508ab9f84a08b31e30eb4cba88ba"
format_stamp: "Formatted at 2026-08-09 12:12:16 on dist-test-slave-2w3w"
server_key: "f16edaf4c694338b80604c51100faed5"
server_key_iv: "b8b79cfe47aabfd7e5a279a4f22d124b"
server_key_version: "kuduclusterkey@0"
I20260809 12:12:16.136026 13244 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskgH5Pjm/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786277382073938-888-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskgH5Pjm/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786277382073938-888-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskgH5Pjm/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786277382073938-888-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260809 12:12:16.235535 13244 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260809 12:12:16.243602 13244 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260809 12:12:16.244160 13244 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260809 12:12:16.252578 13244 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260809 12:12:16.276335 13244 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260809 12:12:16.276575 13244 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.001s sys 0.001s
I20260809 12:12:16.276854 13244 ts_tablet_manager.cc:610] Registered 0 tablets
I20260809 12:12:16.277004 13244 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Aug 09 12:12:16 dist-test-slave-2w3w krb5kdc[12652](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786277535, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.222.62@KRBTEST.COM for HTTP/127.0.222.20@KRBTEST.COM
I20260809 12:12:16.599335 13244 rpc_server.cc:307] RPC server started. Bound to: 127.0.222.1:41929
I20260809 12:12:16.599426 13386 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.222.1:41929 every 8 connection(s)
I20260809 12:12:16.602890 13244 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskgH5Pjm/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786277382073938-888-0/minicluster-data/ts-0/data/info.pb
I20260809 12:12:16.608603 888 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskgH5Pjm/build/asan/bin/kudu as pid 13244
I20260809 12:12:16.608978 888 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskgH5Pjm/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786277382073938-888-0/minicluster-data/ts-0/wal/instance
I20260809 12:12:16.610657 888 external_mini_cluster.cc:1428] Setting key db44f0deecbe19a1aa4a667b3a2584ff
Aug 09 12:12:16 dist-test-slave-2w3w krb5kdc[12652](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786277536, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.222.1@KRBTEST.COM for kudu/127.0.222.62@KRBTEST.COM
I20260809 12:12:16.666695 13389 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 12:12:16.607633 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.222.1:42795 (local address 127.0.222.62:39139)
0809 12:12:16.635025 (+ 27392us) server_negotiation.cc:206] Beginning negotiation
0809 12:12:16.635034 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0809 12:12:16.636913 (+ 1879us) server_negotiation.cc:407] Connection header received
0809 12:12:16.639880 (+ 2967us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 12:12:16.639886 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 12:12:16.639978 (+ 92us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 12:12:16.640250 (+ 272us) server_negotiation.cc:226] Negotiated authn=SASL
0809 12:12:16.643508 (+ 3258us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 12:12:16.645107 (+ 1599us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 12:12:16.648029 (+ 2922us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 12:12:16.648955 (+ 926us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 12:12:16.649079 (+ 124us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 12:12:16.656041 (+ 6962us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 12:12:16.656049 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 12:12:16.656054 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0809 12:12:16.656096 (+ 42us) server_negotiation.cc:922] calling sasl_server_start()
0809 12:12:16.661608 (+ 5512us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 12:12:16.662884 (+ 1276us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 12:12:16.662889 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 12:12:16.662893 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0809 12:12:16.663212 (+ 319us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 12:12:16.663994 (+ 782us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 12:12:16.664001 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 12:12:16.664004 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0809 12:12:16.664499 (+ 495us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 12:12:16.664644 (+ 145us) server_negotiation.cc:1035] Waiting for connection context
0809 12:12:16.665786 (+ 1142us) server_negotiation.cc:299] Negotiation successful
0809 12:12:16.666037 (+ 251us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":27030,"thread_start_us":26825,"threads_started":1}
I20260809 12:12:16.669719 13390 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 12:12:16.608259 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.222.62:39139 (local address 127.0.222.1:42795)
0809 12:12:16.635859 (+ 27600us) negotiation.cc:107] Waiting for socket to connect
0809 12:12:16.635935 (+ 76us) client_negotiation.cc:174] Beginning negotiation
0809 12:12:16.639492 (+ 3557us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 12:12:16.640924 (+ 1432us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 12:12:16.640938 (+ 14us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 12:12:16.641888 (+ 950us) client_negotiation.cc:189] Negotiated authn=SASL
0809 12:12:16.643305 (+ 1417us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 12:12:16.643333 (+ 28us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 12:12:16.645403 (+ 2070us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 12:12:16.645408 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 12:12:16.647863 (+ 2455us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 12:12:16.647877 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 12:12:16.649478 (+ 1601us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 12:12:16.649485 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 12:12:16.649866 (+ 381us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 12:12:16.650952 (+ 1086us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0809 12:12:16.651005 (+ 53us) client_negotiation.cc:656] calling sasl_client_start()
0809 12:12:16.655736 (+ 4731us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0809 12:12:16.661831 (+ 6095us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0809 12:12:16.661841 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0809 12:12:16.661859 (+ 18us) client_negotiation.cc:760] calling sasl_client_step()
0809 12:12:16.662717 (+ 858us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0809 12:12:16.663384 (+ 667us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0809 12:12:16.663389 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0809 12:12:16.663392 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0809 12:12:16.663851 (+ 459us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0809 12:12:16.664752 (+ 901us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0809 12:12:16.664760 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0809 12:12:16.665243 (+ 483us) client_negotiation.cc:769] Sending connection context
0809 12:12:16.667099 (+ 1856us) client_negotiation.cc:240] Negotiation successful
0809 12:12:16.667717 (+ 618us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":27153,"thread_start_us":26802,"threads_started":1}
I20260809 12:12:16.672380 13387 heartbeater.cc:347] Connected to a master server at 127.0.222.62:39139
I20260809 12:12:16.672851 13387 heartbeater.cc:464] Registering TS with master...
I20260809 12:12:16.674332 13387 heartbeater.cc:511] Master 127.0.222.62:39139 requested a full tablet report, sending...
I20260809 12:12:16.677829 13161 ts_manager.cc:194] Registered new tserver with Master: 7e10508ab9f84a08b31e30eb4cba88ba (127.0.222.1:41929)
I20260809 12:12:16.680124 13161 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.222.1@KRBTEST.COM'} at 127.0.222.1:42795
WARNING: no policy specified for kudu/127.0.222.2@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.222.2@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.222.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskgH5Pjm/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786277382073938-888-0/krb5kdc/kudu_127.0.222.2.keytab.
Entry for principal kudu/127.0.222.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskgH5Pjm/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786277382073938-888-0/krb5kdc/kudu_127.0.222.2.keytab.
W20260809 12:12:16.730486 888 mini_kdc.cc:262] Time spent creating service keytab for kudu/127.0.222.2: real 0.105s user 0.000s sys 0.045s
I20260809 12:12:16.731436 888 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskgH5Pjm/build/asan/bin/kudu
/tmp/dist-test-taskgH5Pjm/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskgH5Pjm/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786277382073938-888-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskgH5Pjm/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786277382073938-888-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskgH5Pjm/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786277382073938-888-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskgH5Pjm/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786277382073938-888-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.0.222.20:46617/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.222.2:0
--local_ip_for_outbound_sockets=127.0.222.2
--webserver_interface=127.0.222.2
--webserver_port=0
--tserver_master_addrs=127.0.222.62:39139
--keytab_file=/tmp/dist-test-taskgH5Pjm/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786277382073938-888-0/krb5kdc/kudu_127.0.222.2.keytab
--principal=kudu/127.0.222.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.222.20:37617
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskgH5Pjm/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786277382073938-888-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskgH5Pjm/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786277382073938-888-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskgH5Pjm/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786277382073938-888-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Aug 09 12:12:16 dist-test-slave-2w3w krb5kdc[12652](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786277535, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.222.62@KRBTEST.COM for HTTP/127.0.222.20@KRBTEST.COM
W20260809 12:12:17.499330 13393 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260809 12:12:17.499749 13393 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260809 12:12:17.500023 13393 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260809 12:12:17.510229 13393 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260809 12:12:17.510468 13393 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260809 12:12:17.510700 13393 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.222.2
I20260809 12:12:17.524305 13393 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.222.20:37617
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskgH5Pjm/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786277382073938-888-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskgH5Pjm/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786277382073938-888-0/minicluster-data/ts-1/wal
--ranger_kms_url=127.0.222.20:46617/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskgH5Pjm/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786277382073938-888-0/krb5kdc/kudu_127.0.222.2.keytab
--principal=kudu/127.0.222.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.222.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskgH5Pjm/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786277382073938-888-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.222.2
--webserver_port=0
--tserver_master_addrs=127.0.222.62:39139
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.222.2
--log_dir=/tmp/dist-test-taskgH5Pjm/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786277382073938-888-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 09 Aug 2026 11:06:41 UTC on ccycloud.quasar-bwoogy.root.comops.site
build id 4575991
ASAN enabled
I20260809 12:12:17.525897 13393 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260809 12:12:17.527878 13393 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260809 12:12:17.544853 13408 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 12:12:17.544961 13410 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 12:12:17.545343 13407 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260809 12:12:17.548467 13393 server_base.cc:1034] running on GCE node
I20260809 12:12:17.549129 13393 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260809 12:12:17.550312 13393 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260809 12:12:17.551574 13393 hybrid_clock.cc:648] HybridClock initialized: now 1786277537551489 us; error 47 us; skew 500 ppm
I20260809 12:12:17.551971 13393 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 09 12:12:17 dist-test-slave-2w3w krb5kdc[12652](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786277537, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.222.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260809 12:12:17.555833 13393 init.cc:377] Logged in from keytab as kudu/127.0.222.2@KRBTEST.COM (short username kudu)
I20260809 12:12:17.557511 13393 webserver.cc:460] Webserver started at http://127.0.222.2:42911/ using document root <none> and password file <none>
I20260809 12:12:17.558274 13393 fs_manager.cc:359] Metadata directory not provided
I20260809 12:12:17.558404 13393 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260809 12:12:17.558748 13393 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Aug 09 12:12:17 dist-test-slave-2w3w krb5kdc[12652](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786277537, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.222.2@KRBTEST.COM for HTTP/127.0.222.20@KRBTEST.COM
I20260809 12:12:17.577936 13393 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskgH5Pjm/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786277382073938-888-0/minicluster-data/ts-1/data/instance:
uuid: "3828629cc754491cb361a1b3e4912a83"
format_stamp: "Formatted at 2026-08-09 12:12:17 on dist-test-slave-2w3w"
server_key: "7f1d7034f61635a0a8ae526a36c26029"
server_key_iv: "c1828e9af744b2b34878cbd0070191a9"
server_key_version: "kuduclusterkey@0"
I20260809 12:12:17.579607 13393 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskgH5Pjm/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786277382073938-888-0/minicluster-data/ts-1/wal/instance:
uuid: "3828629cc754491cb361a1b3e4912a83"
format_stamp: "Formatted at 2026-08-09 12:12:17 on dist-test-slave-2w3w"
server_key: "7f1d7034f61635a0a8ae526a36c26029"
server_key_iv: "c1828e9af744b2b34878cbd0070191a9"
server_key_version: "kuduclusterkey@0"
I20260809 12:12:17.587659 13393 fs_manager.cc:691] Time spent creating directory manager: real 0.007s user 0.006s sys 0.002s
I20260809 12:12:17.606164 13418 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260809 12:12:17.607796 13393 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.001s sys 0.001s
I20260809 12:12:17.608011 13393 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskgH5Pjm/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786277382073938-888-0/minicluster-data/ts-1/data,/tmp/dist-test-taskgH5Pjm/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786277382073938-888-0/minicluster-data/ts-1/wal
uuid: "3828629cc754491cb361a1b3e4912a83"
format_stamp: "Formatted at 2026-08-09 12:12:17 on dist-test-slave-2w3w"
server_key: "7f1d7034f61635a0a8ae526a36c26029"
server_key_iv: "c1828e9af744b2b34878cbd0070191a9"
server_key_version: "kuduclusterkey@0"
I20260809 12:12:17.608275 13393 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskgH5Pjm/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786277382073938-888-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskgH5Pjm/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786277382073938-888-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskgH5Pjm/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786277382073938-888-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260809 12:12:17.636154 13393 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260809 12:12:17.642057 13393 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260809 12:12:17.642700 13393 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260809 12:12:17.644660 13393 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260809 12:12:17.647033 13393 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260809 12:12:17.647226 13393 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.001s
I20260809 12:12:17.647488 13393 ts_tablet_manager.cc:610] Registered 0 tablets
I20260809 12:12:17.647668 13393 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260809 12:12:17.685338 13387 heartbeater.cc:503] Master 127.0.222.62:39139 was elected leader, sending a full tablet report...
I20260809 12:12:17.746292 13393 rpc_server.cc:307] RPC server started. Bound to: 127.0.222.2:37561
I20260809 12:12:17.746400 13537 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.222.2:37561 every 8 connection(s)
I20260809 12:12:17.748454 13393 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskgH5Pjm/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786277382073938-888-0/minicluster-data/ts-1/data/info.pb
I20260809 12:12:17.754899 888 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskgH5Pjm/build/asan/bin/kudu as pid 13393
I20260809 12:12:17.755221 888 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskgH5Pjm/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786277382073938-888-0/minicluster-data/ts-1/wal/instance
I20260809 12:12:17.755965 888 external_mini_cluster.cc:1428] Setting key 55375a1edc3c1f8a828478401ce84a03
Aug 09 12:12:17 dist-test-slave-2w3w krb5kdc[12652](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786277537, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.222.2@KRBTEST.COM for kudu/127.0.222.62@KRBTEST.COM
I20260809 12:12:17.784584 13540 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 12:12:17.753525 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.222.2:36941 (local address 127.0.222.62:39139)
0809 12:12:17.754369 (+ 844us) server_negotiation.cc:206] Beginning negotiation
0809 12:12:17.754375 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0809 12:12:17.756480 (+ 2105us) server_negotiation.cc:407] Connection header received
0809 12:12:17.760144 (+ 3664us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 12:12:17.760153 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 12:12:17.760257 (+ 104us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 12:12:17.760587 (+ 330us) server_negotiation.cc:226] Negotiated authn=SASL
0809 12:12:17.763390 (+ 2803us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 12:12:17.765314 (+ 1924us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 12:12:17.768440 (+ 3126us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 12:12:17.769540 (+ 1100us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 12:12:17.769727 (+ 187us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 12:12:17.775798 (+ 6071us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 12:12:17.775813 (+ 15us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 12:12:17.775819 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0809 12:12:17.775906 (+ 87us) server_negotiation.cc:922] calling sasl_server_start()
0809 12:12:17.780787 (+ 4881us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 12:12:17.781697 (+ 910us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 12:12:17.781706 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 12:12:17.781712 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0809 12:12:17.781881 (+ 169us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 12:12:17.782504 (+ 623us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 12:12:17.782508 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 12:12:17.782510 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0809 12:12:17.782829 (+ 319us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 12:12:17.783059 (+ 230us) server_negotiation.cc:1035] Waiting for connection context
0809 12:12:17.783886 (+ 827us) server_negotiation.cc:299] Negotiation successful
0809 12:12:17.784116 (+ 230us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":672,"thread_start_us":480,"threads_started":1}
I20260809 12:12:17.785943 13541 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 12:12:17.754381 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.222.62:39139 (local address 127.0.222.2:36941)
0809 12:12:17.756244 (+ 1863us) negotiation.cc:107] Waiting for socket to connect
0809 12:12:17.756295 (+ 51us) client_negotiation.cc:174] Beginning negotiation
0809 12:12:17.759804 (+ 3509us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 12:12:17.760879 (+ 1075us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 12:12:17.760892 (+ 13us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 12:12:17.761907 (+ 1015us) client_negotiation.cc:189] Negotiated authn=SASL
0809 12:12:17.763160 (+ 1253us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 12:12:17.763188 (+ 28us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 12:12:17.765514 (+ 2326us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 12:12:17.765520 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 12:12:17.768251 (+ 2731us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 12:12:17.768276 (+ 25us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 12:12:17.769857 (+ 1581us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 12:12:17.769866 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 12:12:17.770277 (+ 411us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 12:12:17.771360 (+ 1083us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0809 12:12:17.771410 (+ 50us) client_negotiation.cc:656] calling sasl_client_start()
0809 12:12:17.775421 (+ 4011us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0809 12:12:17.780952 (+ 5531us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0809 12:12:17.780959 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0809 12:12:17.780974 (+ 15us) client_negotiation.cc:760] calling sasl_client_step()
0809 12:12:17.781531 (+ 557us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0809 12:12:17.782004 (+ 473us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0809 12:12:17.782010 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0809 12:12:17.782013 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0809 12:12:17.782360 (+ 347us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0809 12:12:17.783001 (+ 641us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0809 12:12:17.783011 (+ 10us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0809 12:12:17.783403 (+ 392us) client_negotiation.cc:769] Sending connection context
0809 12:12:17.783740 (+ 337us) client_negotiation.cc:240] Negotiation successful
0809 12:12:17.784364 (+ 624us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1498,"thread_start_us":951,"threads_started":1}
I20260809 12:12:17.787890 13538 heartbeater.cc:347] Connected to a master server at 127.0.222.62:39139
I20260809 12:12:17.788267 13538 heartbeater.cc:464] Registering TS with master...
I20260809 12:12:17.789115 13538 heartbeater.cc:511] Master 127.0.222.62:39139 requested a full tablet report, sending...
I20260809 12:12:17.790720 13161 ts_manager.cc:194] Registered new tserver with Master: 3828629cc754491cb361a1b3e4912a83 (127.0.222.2:37561)
I20260809 12:12:17.791802 13161 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.222.2@KRBTEST.COM'} at 127.0.222.2:36941
WARNING: no policy specified for kudu/127.0.222.3@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.222.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.222.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskgH5Pjm/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786277382073938-888-0/krb5kdc/kudu_127.0.222.3.keytab.
Entry for principal kudu/127.0.222.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskgH5Pjm/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786277382073938-888-0/krb5kdc/kudu_127.0.222.3.keytab.
I20260809 12:12:17.850790 888 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskgH5Pjm/build/asan/bin/kudu
/tmp/dist-test-taskgH5Pjm/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskgH5Pjm/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786277382073938-888-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskgH5Pjm/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786277382073938-888-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskgH5Pjm/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786277382073938-888-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskgH5Pjm/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786277382073938-888-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.0.222.20:46617/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.222.3:0
--local_ip_for_outbound_sockets=127.0.222.3
--webserver_interface=127.0.222.3
--webserver_port=0
--tserver_master_addrs=127.0.222.62:39139
--keytab_file=/tmp/dist-test-taskgH5Pjm/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786277382073938-888-0/krb5kdc/kudu_127.0.222.3.keytab
--principal=kudu/127.0.222.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.222.20:37617
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskgH5Pjm/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786277382073938-888-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskgH5Pjm/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786277382073938-888-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskgH5Pjm/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786277382073938-888-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260809 12:12:18.409379 13544 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260809 12:12:18.409754 13544 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260809 12:12:18.409961 13544 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260809 12:12:18.418244 13544 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260809 12:12:18.418439 13544 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260809 12:12:18.418656 13544 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.222.3
I20260809 12:12:18.429972 13544 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.222.20:37617
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskgH5Pjm/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786277382073938-888-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskgH5Pjm/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786277382073938-888-0/minicluster-data/ts-2/wal
--ranger_kms_url=127.0.222.20:46617/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskgH5Pjm/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786277382073938-888-0/krb5kdc/kudu_127.0.222.3.keytab
--principal=kudu/127.0.222.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.222.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskgH5Pjm/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786277382073938-888-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.222.3
--webserver_port=0
--tserver_master_addrs=127.0.222.62:39139
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.222.3
--log_dir=/tmp/dist-test-taskgH5Pjm/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786277382073938-888-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 09 Aug 2026 11:06:41 UTC on ccycloud.quasar-bwoogy.root.comops.site
build id 4575991
ASAN enabled
I20260809 12:12:18.431504 13544 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260809 12:12:18.432900 13544 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260809 12:12:18.447532 13552 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 12:12:18.447803 13550 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 12:12:18.447901 13549 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260809 12:12:18.449298 13544 server_base.cc:1034] running on GCE node
I20260809 12:12:18.450284 13544 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260809 12:12:18.451702 13544 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260809 12:12:18.453166 13544 hybrid_clock.cc:648] HybridClock initialized: now 1786277538453003 us; error 55 us; skew 500 ppm
I20260809 12:12:18.453761 13544 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 09 12:12:18 dist-test-slave-2w3w krb5kdc[12652](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786277538, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.222.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260809 12:12:18.458005 13544 init.cc:377] Logged in from keytab as kudu/127.0.222.3@KRBTEST.COM (short username kudu)
I20260809 12:12:18.459805 13544 webserver.cc:460] Webserver started at http://127.0.222.3:41637/ using document root <none> and password file <none>
I20260809 12:12:18.460526 13544 fs_manager.cc:359] Metadata directory not provided
I20260809 12:12:18.460685 13544 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260809 12:12:18.461021 13544 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Aug 09 12:12:18 dist-test-slave-2w3w krb5kdc[12652](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786277538, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.222.3@KRBTEST.COM for HTTP/127.0.222.20@KRBTEST.COM
I20260809 12:12:18.478343 13544 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskgH5Pjm/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786277382073938-888-0/minicluster-data/ts-2/data/instance:
uuid: "b856ed095c0441a2b4407a94090ab4db"
format_stamp: "Formatted at 2026-08-09 12:12:18 on dist-test-slave-2w3w"
server_key: "52599db70b27501a15db17e6e3c3a48b"
server_key_iv: "0b203443ed4d795b845292863a2c946f"
server_key_version: "kuduclusterkey@0"
I20260809 12:12:18.479753 13544 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskgH5Pjm/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786277382073938-888-0/minicluster-data/ts-2/wal/instance:
uuid: "b856ed095c0441a2b4407a94090ab4db"
format_stamp: "Formatted at 2026-08-09 12:12:18 on dist-test-slave-2w3w"
server_key: "52599db70b27501a15db17e6e3c3a48b"
server_key_iv: "0b203443ed4d795b845292863a2c946f"
server_key_version: "kuduclusterkey@0"
I20260809 12:12:18.487701 13544 fs_manager.cc:691] Time spent creating directory manager: real 0.007s user 0.006s sys 0.002s
I20260809 12:12:18.506264 13559 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260809 12:12:18.508023 13544 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.001s sys 0.001s
I20260809 12:12:18.508262 13544 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskgH5Pjm/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786277382073938-888-0/minicluster-data/ts-2/data,/tmp/dist-test-taskgH5Pjm/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786277382073938-888-0/minicluster-data/ts-2/wal
uuid: "b856ed095c0441a2b4407a94090ab4db"
format_stamp: "Formatted at 2026-08-09 12:12:18 on dist-test-slave-2w3w"
server_key: "52599db70b27501a15db17e6e3c3a48b"
server_key_iv: "0b203443ed4d795b845292863a2c946f"
server_key_version: "kuduclusterkey@0"
I20260809 12:12:18.508524 13544 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskgH5Pjm/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786277382073938-888-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskgH5Pjm/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786277382073938-888-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskgH5Pjm/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786277382073938-888-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260809 12:12:18.533957 13544 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260809 12:12:18.538436 13544 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260809 12:12:18.538834 13544 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260809 12:12:18.540030 13544 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260809 12:12:18.541916 13544 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260809 12:12:18.542002 13544 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260809 12:12:18.542199 13544 ts_tablet_manager.cc:610] Registered 0 tablets
I20260809 12:12:18.542295 13544 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260809 12:12:18.601541 13544 rpc_server.cc:307] RPC server started. Bound to: 127.0.222.3:38419
I20260809 12:12:18.601620 13681 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.222.3:38419 every 8 connection(s)
I20260809 12:12:18.602936 13544 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskgH5Pjm/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786277382073938-888-0/minicluster-data/ts-2/data/info.pb
I20260809 12:12:18.612861 888 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskgH5Pjm/build/asan/bin/kudu as pid 13544
I20260809 12:12:18.613080 888 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskgH5Pjm/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786277382073938-888-0/minicluster-data/ts-2/wal/instance
I20260809 12:12:18.613725 888 external_mini_cluster.cc:1428] Setting key 7873b79d210d7a303ff13dccc9e98ea1
Aug 09 12:12:18 dist-test-slave-2w3w krb5kdc[12652](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786277538, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.222.3@KRBTEST.COM for kudu/127.0.222.62@KRBTEST.COM
I20260809 12:12:18.626719 13684 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 12:12:18.606072 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.222.3:50743 (local address 127.0.222.62:39139)
0809 12:12:18.606720 (+ 648us) server_negotiation.cc:206] Beginning negotiation
0809 12:12:18.606726 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0809 12:12:18.607886 (+ 1160us) server_negotiation.cc:407] Connection header received
0809 12:12:18.610189 (+ 2303us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 12:12:18.610193 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 12:12:18.610259 (+ 66us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 12:12:18.610440 (+ 181us) server_negotiation.cc:226] Negotiated authn=SASL
0809 12:12:18.612233 (+ 1793us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 12:12:18.613691 (+ 1458us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 12:12:18.615767 (+ 2076us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 12:12:18.616385 (+ 618us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 12:12:18.616506 (+ 121us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 12:12:18.620343 (+ 3837us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 12:12:18.620347 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 12:12:18.620350 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0809 12:12:18.620389 (+ 39us) server_negotiation.cc:922] calling sasl_server_start()
0809 12:12:18.623503 (+ 3114us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 12:12:18.624288 (+ 785us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 12:12:18.624293 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 12:12:18.624295 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0809 12:12:18.624464 (+ 169us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 12:12:18.624956 (+ 492us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 12:12:18.624961 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 12:12:18.624964 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0809 12:12:18.625277 (+ 313us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 12:12:18.625458 (+ 181us) server_negotiation.cc:1035] Waiting for connection context
0809 12:12:18.626116 (+ 658us) server_negotiation.cc:299] Negotiation successful
0809 12:12:18.626321 (+ 205us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":493,"thread_start_us":355,"threads_started":1}
I20260809 12:12:18.627944 13685 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 12:12:18.606574 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.222.62:39139 (local address 127.0.222.3:50743)
0809 12:12:18.607709 (+ 1135us) negotiation.cc:107] Waiting for socket to connect
0809 12:12:18.607756 (+ 47us) client_negotiation.cc:174] Beginning negotiation
0809 12:12:18.609811 (+ 2055us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 12:12:18.610672 (+ 861us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 12:12:18.610681 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 12:12:18.611322 (+ 641us) client_negotiation.cc:189] Negotiated authn=SASL
0809 12:12:18.612062 (+ 740us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 12:12:18.612081 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 12:12:18.613911 (+ 1830us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 12:12:18.613915 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 12:12:18.615607 (+ 1692us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 12:12:18.615620 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 12:12:18.616661 (+ 1041us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 12:12:18.616672 (+ 11us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 12:12:18.617021 (+ 349us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 12:12:18.617734 (+ 713us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0809 12:12:18.617755 (+ 21us) client_negotiation.cc:656] calling sasl_client_start()
0809 12:12:18.620093 (+ 2338us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0809 12:12:18.623636 (+ 3543us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0809 12:12:18.623644 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0809 12:12:18.623660 (+ 16us) client_negotiation.cc:760] calling sasl_client_step()
0809 12:12:18.624156 (+ 496us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0809 12:12:18.624574 (+ 418us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0809 12:12:18.624579 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0809 12:12:18.624583 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0809 12:12:18.624786 (+ 203us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0809 12:12:18.625430 (+ 644us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0809 12:12:18.625438 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0809 12:12:18.625751 (+ 313us) client_negotiation.cc:769] Sending connection context
0809 12:12:18.626024 (+ 273us) client_negotiation.cc:240] Negotiation successful
0809 12:12:18.626547 (+ 523us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":860,"thread_start_us":577,"threads_started":1}
I20260809 12:12:18.629822 13682 heartbeater.cc:347] Connected to a master server at 127.0.222.62:39139
I20260809 12:12:18.630224 13682 heartbeater.cc:464] Registering TS with master...
I20260809 12:12:18.631556 13682 heartbeater.cc:511] Master 127.0.222.62:39139 requested a full tablet report, sending...
I20260809 12:12:18.633546 13160 ts_manager.cc:194] Registered new tserver with Master: b856ed095c0441a2b4407a94090ab4db (127.0.222.3:38419)
I20260809 12:12:18.634557 13160 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.222.3@KRBTEST.COM'} at 127.0.222.3:50743
I20260809 12:12:18.641798 888 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260809 12:12:18.663673 13684 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 12:12:18.647906 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44962 (local address 127.0.222.62:39139)
0809 12:12:18.648146 (+ 240us) server_negotiation.cc:206] Beginning negotiation
0809 12:12:18.648154 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0809 12:12:18.649027 (+ 873us) server_negotiation.cc:407] Connection header received
0809 12:12:18.649208 (+ 181us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 12:12:18.649212 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 12:12:18.649294 (+ 82us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 12:12:18.649375 (+ 81us) server_negotiation.cc:226] Negotiated authn=SASL
0809 12:12:18.651092 (+ 1717us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 12:12:18.651881 (+ 789us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 12:12:18.653613 (+ 1732us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 12:12:18.654119 (+ 506us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 12:12:18.654215 (+ 96us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 12:12:18.657234 (+ 3019us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 12:12:18.657239 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 12:12:18.657243 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0809 12:12:18.657277 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0809 12:12:18.660465 (+ 3188us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 12:12:18.661317 (+ 852us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 12:12:18.661321 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 12:12:18.661324 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0809 12:12:18.661508 (+ 184us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 12:12:18.661962 (+ 454us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 12:12:18.661966 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 12:12:18.661968 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0809 12:12:18.662346 (+ 378us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 12:12:18.662587 (+ 241us) server_negotiation.cc:1035] Waiting for connection context
0809 12:12:18.662955 (+ 368us) server_negotiation.cc:299] Negotiation successful
0809 12:12:18.663217 (+ 262us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":89}
I20260809 12:12:18.668411 13160 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:44962:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260809 12:12:18.681377 13160 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260809 12:12:18.713436 13699 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 12:12:18.698926 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46436 (local address 127.0.222.2:37561)
0809 12:12:18.701253 (+ 2327us) server_negotiation.cc:206] Beginning negotiation
0809 12:12:18.701262 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0809 12:12:18.701291 (+ 29us) server_negotiation.cc:407] Connection header received
0809 12:12:18.701392 (+ 101us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 12:12:18.701401 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 12:12:18.701696 (+ 295us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 12:12:18.703464 (+ 1768us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0809 12:12:18.703703 (+ 239us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 12:12:18.705592 (+ 1889us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 12:12:18.709147 (+ 3555us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 12:12:18.711482 (+ 2335us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 12:12:18.711669 (+ 187us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 12:12:18.711951 (+ 282us) server_negotiation.cc:1035] Waiting for connection context
0809 12:12:18.712641 (+ 690us) server_negotiation.cc:299] Negotiation successful
0809 12:12:18.712838 (+ 197us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":2130,"thread_start_us":2607,"threads_started":1}
I20260809 12:12:18.714334 13698 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 12:12:18.698604 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.222.2:37561 (local address 127.0.0.1:46436)
0809 12:12:18.700085 (+ 1481us) negotiation.cc:107] Waiting for socket to connect
0809 12:12:18.700143 (+ 58us) client_negotiation.cc:174] Beginning negotiation
0809 12:12:18.700464 (+ 321us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 12:12:18.702161 (+ 1697us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 12:12:18.702169 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 12:12:18.702229 (+ 60us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0809 12:12:18.703246 (+ 1017us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 12:12:18.703261 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 12:12:18.705872 (+ 2611us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 12:12:18.705878 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 12:12:18.708807 (+ 2929us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 12:12:18.708823 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 12:12:18.711782 (+ 2959us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 12:12:18.711790 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 12:12:18.712107 (+ 317us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 12:12:18.712241 (+ 134us) client_negotiation.cc:769] Sending connection context
0809 12:12:18.712367 (+ 126us) client_negotiation.cc:240] Negotiation successful
0809 12:12:18.713592 (+ 1225us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1136,"thread_start_us":3863,"threads_started":1}
I20260809 12:12:18.719094 13700 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 12:12:18.699859 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53616 (local address 127.0.222.3:38419)
0809 12:12:18.708423 (+ 8564us) server_negotiation.cc:206] Beginning negotiation
0809 12:12:18.708432 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0809 12:12:18.708494 (+ 62us) server_negotiation.cc:407] Connection header received
0809 12:12:18.708628 (+ 134us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 12:12:18.708639 (+ 11us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 12:12:18.709067 (+ 428us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 12:12:18.709570 (+ 503us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0809 12:12:18.711507 (+ 1937us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 12:12:18.713306 (+ 1799us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 12:12:18.716372 (+ 3066us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 12:12:18.717604 (+ 1232us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 12:12:18.717768 (+ 164us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 12:12:18.718065 (+ 297us) server_negotiation.cc:1035] Waiting for connection context
0809 12:12:18.718414 (+ 349us) server_negotiation.cc:299] Negotiation successful
0809 12:12:18.718583 (+ 169us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":8210,"thread_start_us":7856,"threads_started":1}
I20260809 12:12:18.719086 13697 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 12:12:18.698904 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.222.3:38419 (local address 127.0.0.1:53616)
0809 12:12:18.703066 (+ 4162us) negotiation.cc:107] Waiting for socket to connect
0809 12:12:18.703089 (+ 23us) client_negotiation.cc:174] Beginning negotiation
0809 12:12:18.703301 (+ 212us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 12:12:18.709430 (+ 6129us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 12:12:18.709436 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 12:12:18.709469 (+ 33us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0809 12:12:18.711175 (+ 1706us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 12:12:18.711188 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 12:12:18.713489 (+ 2301us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 12:12:18.713504 (+ 15us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 12:12:18.716190 (+ 2686us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 12:12:18.716202 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 12:12:18.717934 (+ 1732us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 12:12:18.717941 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 12:12:18.718200 (+ 259us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 12:12:18.718235 (+ 35us) client_negotiation.cc:769] Sending connection context
0809 12:12:18.718352 (+ 117us) client_negotiation.cc:240] Negotiation successful
0809 12:12:18.718499 (+ 147us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":3921,"thread_start_us":1501,"threads_started":1}
I20260809 12:12:18.720042 13469 tablet_service.cc:1467] Processing CreateTablet for tablet 64d33967c1b542cfb1a90f22af62f412 (DEFAULT_TABLE table=test-table [id=ee8bd4b380c040d6b78c289366cb913f]), partition=RANGE (key) PARTITION UNBOUNDED
I20260809 12:12:18.721837 13469 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 64d33967c1b542cfb1a90f22af62f412. 1 dirs total, 0 dirs full, 0 dirs failed
I20260809 12:12:18.723721 13701 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 12:12:18.703616 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39242 (local address 127.0.222.1:41929)
0809 12:12:18.706108 (+ 2492us) server_negotiation.cc:206] Beginning negotiation
0809 12:12:18.706119 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0809 12:12:18.711753 (+ 5634us) server_negotiation.cc:407] Connection header received
0809 12:12:18.712196 (+ 443us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 12:12:18.712202 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 12:12:18.712409 (+ 207us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 12:12:18.712655 (+ 246us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0809 12:12:18.714711 (+ 2056us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 12:12:18.717186 (+ 2475us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 12:12:18.720557 (+ 3371us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 12:12:18.722283 (+ 1726us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 12:12:18.722432 (+ 149us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 12:12:18.722721 (+ 289us) server_negotiation.cc:1035] Waiting for connection context
0809 12:12:18.723023 (+ 302us) server_negotiation.cc:299] Negotiation successful
0809 12:12:18.723263 (+ 240us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":2224,"thread_start_us":1954,"threads_started":1}
I20260809 12:12:18.723162 13612 tablet_service.cc:1467] Processing CreateTablet for tablet 64d33967c1b542cfb1a90f22af62f412 (DEFAULT_TABLE table=test-table [id=ee8bd4b380c040d6b78c289366cb913f]), partition=RANGE (key) PARTITION UNBOUNDED
I20260809 12:12:18.724694 13702 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 12:12:18.705200 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.222.1:41929 (local address 127.0.0.1:39242)
0809 12:12:18.711578 (+ 6378us) negotiation.cc:107] Waiting for socket to connect
0809 12:12:18.711599 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0809 12:12:18.711965 (+ 366us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 12:12:18.712637 (+ 672us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 12:12:18.712641 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 12:12:18.712668 (+ 27us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0809 12:12:18.714377 (+ 1709us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 12:12:18.714395 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 12:12:18.717482 (+ 3087us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 12:12:18.717488 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 12:12:18.720355 (+ 2867us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 12:12:18.720372 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 12:12:18.722576 (+ 2204us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 12:12:18.722584 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 12:12:18.722777 (+ 193us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 12:12:18.722809 (+ 32us) client_negotiation.cc:769] Sending connection context
0809 12:12:18.724005 (+ 1196us) client_negotiation.cc:240] Negotiation successful
0809 12:12:18.724155 (+ 150us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":6128,"thread_start_us":5698,"threads_started":1}
I20260809 12:12:18.725003 13612 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 64d33967c1b542cfb1a90f22af62f412. 1 dirs total, 0 dirs full, 0 dirs failed
I20260809 12:12:18.729413 13315 tablet_service.cc:1467] Processing CreateTablet for tablet 64d33967c1b542cfb1a90f22af62f412 (DEFAULT_TABLE table=test-table [id=ee8bd4b380c040d6b78c289366cb913f]), partition=RANGE (key) PARTITION UNBOUNDED
I20260809 12:12:18.731403 13315 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 64d33967c1b542cfb1a90f22af62f412. 1 dirs total, 0 dirs full, 0 dirs failed
I20260809 12:12:18.741168 13703 tablet_bootstrap.cc:492] T 64d33967c1b542cfb1a90f22af62f412 P b856ed095c0441a2b4407a94090ab4db: Bootstrap starting.
I20260809 12:12:18.744288 13704 tablet_bootstrap.cc:492] T 64d33967c1b542cfb1a90f22af62f412 P 3828629cc754491cb361a1b3e4912a83: Bootstrap starting.
I20260809 12:12:18.749289 13704 tablet_bootstrap.cc:654] T 64d33967c1b542cfb1a90f22af62f412 P 3828629cc754491cb361a1b3e4912a83: Neither blocks nor log segments found. Creating new log.
I20260809 12:12:18.749670 13703 tablet_bootstrap.cc:654] T 64d33967c1b542cfb1a90f22af62f412 P b856ed095c0441a2b4407a94090ab4db: Neither blocks nor log segments found. Creating new log.
I20260809 12:12:18.751302 13703 log.cc:824] T 64d33967c1b542cfb1a90f22af62f412 P b856ed095c0441a2b4407a94090ab4db: Log is configured to *not* fsync() on all Append() calls
I20260809 12:12:18.751454 13704 log.cc:824] T 64d33967c1b542cfb1a90f22af62f412 P 3828629cc754491cb361a1b3e4912a83: Log is configured to *not* fsync() on all Append() calls
I20260809 12:12:18.767370 13703 tablet_bootstrap.cc:492] T 64d33967c1b542cfb1a90f22af62f412 P b856ed095c0441a2b4407a94090ab4db: No bootstrap required, opened a new log
I20260809 12:12:18.767748 13703 ts_tablet_manager.cc:1397] T 64d33967c1b542cfb1a90f22af62f412 P b856ed095c0441a2b4407a94090ab4db: Time spent bootstrapping tablet: real 0.027s user 0.011s sys 0.010s
I20260809 12:12:18.768040 13707 tablet_bootstrap.cc:492] T 64d33967c1b542cfb1a90f22af62f412 P 7e10508ab9f84a08b31e30eb4cba88ba: Bootstrap starting.
I20260809 12:12:18.768260 13704 tablet_bootstrap.cc:492] T 64d33967c1b542cfb1a90f22af62f412 P 3828629cc754491cb361a1b3e4912a83: No bootstrap required, opened a new log
I20260809 12:12:18.768548 13704 ts_tablet_manager.cc:1397] T 64d33967c1b542cfb1a90f22af62f412 P 3828629cc754491cb361a1b3e4912a83: Time spent bootstrapping tablet: real 0.025s user 0.009s sys 0.013s
I20260809 12:12:18.773510 13707 tablet_bootstrap.cc:654] T 64d33967c1b542cfb1a90f22af62f412 P 7e10508ab9f84a08b31e30eb4cba88ba: Neither blocks nor log segments found. Creating new log.
I20260809 12:12:18.775336 13707 log.cc:824] T 64d33967c1b542cfb1a90f22af62f412 P 7e10508ab9f84a08b31e30eb4cba88ba: Log is configured to *not* fsync() on all Append() calls
I20260809 12:12:18.778191 13704 raft_consensus.cc:348] T 64d33967c1b542cfb1a90f22af62f412 P 3828629cc754491cb361a1b3e4912a83 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3828629cc754491cb361a1b3e4912a83" member_type: VOTER last_known_addr { host: "127.0.222.2" port: 37561 } } peers { permanent_uuid: "b856ed095c0441a2b4407a94090ab4db" member_type: VOTER last_known_addr { host: "127.0.222.3" port: 38419 } } peers { permanent_uuid: "7e10508ab9f84a08b31e30eb4cba88ba" member_type: VOTER last_known_addr { host: "127.0.222.1" port: 41929 } }
I20260809 12:12:18.778716 13704 raft_consensus.cc:374] T 64d33967c1b542cfb1a90f22af62f412 P 3828629cc754491cb361a1b3e4912a83 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260809 12:12:18.778849 13704 raft_consensus.cc:729] T 64d33967c1b542cfb1a90f22af62f412 P 3828629cc754491cb361a1b3e4912a83 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3828629cc754491cb361a1b3e4912a83, State: Initialized, Role: FOLLOWER
I20260809 12:12:18.780030 13707 tablet_bootstrap.cc:492] T 64d33967c1b542cfb1a90f22af62f412 P 7e10508ab9f84a08b31e30eb4cba88ba: No bootstrap required, opened a new log
I20260809 12:12:18.779664 13704 consensus_queue.cc:260] T 64d33967c1b542cfb1a90f22af62f412 P 3828629cc754491cb361a1b3e4912a83 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3828629cc754491cb361a1b3e4912a83" member_type: VOTER last_known_addr { host: "127.0.222.2" port: 37561 } } peers { permanent_uuid: "b856ed095c0441a2b4407a94090ab4db" member_type: VOTER last_known_addr { host: "127.0.222.3" port: 38419 } } peers { permanent_uuid: "7e10508ab9f84a08b31e30eb4cba88ba" member_type: VOTER last_known_addr { host: "127.0.222.1" port: 41929 } }
I20260809 12:12:18.780476 13707 ts_tablet_manager.cc:1397] T 64d33967c1b542cfb1a90f22af62f412 P 7e10508ab9f84a08b31e30eb4cba88ba: Time spent bootstrapping tablet: real 0.013s user 0.005s sys 0.005s
I20260809 12:12:18.781576 13538 heartbeater.cc:503] Master 127.0.222.62:39139 was elected leader, sending a full tablet report...
I20260809 12:12:18.781804 13703 raft_consensus.cc:348] T 64d33967c1b542cfb1a90f22af62f412 P b856ed095c0441a2b4407a94090ab4db [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3828629cc754491cb361a1b3e4912a83" member_type: VOTER last_known_addr { host: "127.0.222.2" port: 37561 } } peers { permanent_uuid: "b856ed095c0441a2b4407a94090ab4db" member_type: VOTER last_known_addr { host: "127.0.222.3" port: 38419 } } peers { permanent_uuid: "7e10508ab9f84a08b31e30eb4cba88ba" member_type: VOTER last_known_addr { host: "127.0.222.1" port: 41929 } }
I20260809 12:12:18.782290 13704 ts_tablet_manager.cc:1428] T 64d33967c1b542cfb1a90f22af62f412 P 3828629cc754491cb361a1b3e4912a83: Time spent starting tablet: real 0.014s user 0.014s sys 0.000s
I20260809 12:12:18.782305 13703 raft_consensus.cc:374] T 64d33967c1b542cfb1a90f22af62f412 P b856ed095c0441a2b4407a94090ab4db [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260809 12:12:18.782459 13703 raft_consensus.cc:729] T 64d33967c1b542cfb1a90f22af62f412 P b856ed095c0441a2b4407a94090ab4db [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b856ed095c0441a2b4407a94090ab4db, State: Initialized, Role: FOLLOWER
I20260809 12:12:18.783444 13703 consensus_queue.cc:260] T 64d33967c1b542cfb1a90f22af62f412 P b856ed095c0441a2b4407a94090ab4db [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3828629cc754491cb361a1b3e4912a83" member_type: VOTER last_known_addr { host: "127.0.222.2" port: 37561 } } peers { permanent_uuid: "b856ed095c0441a2b4407a94090ab4db" member_type: VOTER last_known_addr { host: "127.0.222.3" port: 38419 } } peers { permanent_uuid: "7e10508ab9f84a08b31e30eb4cba88ba" member_type: VOTER last_known_addr { host: "127.0.222.1" port: 41929 } }
I20260809 12:12:18.785239 13682 heartbeater.cc:503] Master 127.0.222.62:39139 was elected leader, sending a full tablet report...
I20260809 12:12:18.785856 13703 ts_tablet_manager.cc:1428] T 64d33967c1b542cfb1a90f22af62f412 P b856ed095c0441a2b4407a94090ab4db: Time spent starting tablet: real 0.018s user 0.014s sys 0.003s
I20260809 12:12:18.793299 13707 raft_consensus.cc:348] T 64d33967c1b542cfb1a90f22af62f412 P 7e10508ab9f84a08b31e30eb4cba88ba [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3828629cc754491cb361a1b3e4912a83" member_type: VOTER last_known_addr { host: "127.0.222.2" port: 37561 } } peers { permanent_uuid: "b856ed095c0441a2b4407a94090ab4db" member_type: VOTER last_known_addr { host: "127.0.222.3" port: 38419 } } peers { permanent_uuid: "7e10508ab9f84a08b31e30eb4cba88ba" member_type: VOTER last_known_addr { host: "127.0.222.1" port: 41929 } }
I20260809 12:12:18.793816 13707 raft_consensus.cc:374] T 64d33967c1b542cfb1a90f22af62f412 P 7e10508ab9f84a08b31e30eb4cba88ba [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260809 12:12:18.793960 13707 raft_consensus.cc:729] T 64d33967c1b542cfb1a90f22af62f412 P 7e10508ab9f84a08b31e30eb4cba88ba [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7e10508ab9f84a08b31e30eb4cba88ba, State: Initialized, Role: FOLLOWER
I20260809 12:12:18.794872 13707 consensus_queue.cc:260] T 64d33967c1b542cfb1a90f22af62f412 P 7e10508ab9f84a08b31e30eb4cba88ba [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3828629cc754491cb361a1b3e4912a83" member_type: VOTER last_known_addr { host: "127.0.222.2" port: 37561 } } peers { permanent_uuid: "b856ed095c0441a2b4407a94090ab4db" member_type: VOTER last_known_addr { host: "127.0.222.3" port: 38419 } } peers { permanent_uuid: "7e10508ab9f84a08b31e30eb4cba88ba" member_type: VOTER last_known_addr { host: "127.0.222.1" port: 41929 } }
I20260809 12:12:18.797367 13707 ts_tablet_manager.cc:1428] T 64d33967c1b542cfb1a90f22af62f412 P 7e10508ab9f84a08b31e30eb4cba88ba: Time spent starting tablet: real 0.017s user 0.010s sys 0.007s
W20260809 12:12:18.855216 13683 tablet.cc:2369] T 64d33967c1b542cfb1a90f22af62f412 P b856ed095c0441a2b4407a94090ab4db: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260809 12:12:18.859622 13388 tablet.cc:2369] T 64d33967c1b542cfb1a90f22af62f412 P 7e10508ab9f84a08b31e30eb4cba88ba: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260809 12:12:18.957813 13711 raft_consensus.cc:482] T 64d33967c1b542cfb1a90f22af62f412 P 7e10508ab9f84a08b31e30eb4cba88ba [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260809 12:12:18.958113 13711 raft_consensus.cc:504] T 64d33967c1b542cfb1a90f22af62f412 P 7e10508ab9f84a08b31e30eb4cba88ba [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3828629cc754491cb361a1b3e4912a83" member_type: VOTER last_known_addr { host: "127.0.222.2" port: 37561 } } peers { permanent_uuid: "b856ed095c0441a2b4407a94090ab4db" member_type: VOTER last_known_addr { host: "127.0.222.3" port: 38419 } } peers { permanent_uuid: "7e10508ab9f84a08b31e30eb4cba88ba" member_type: VOTER last_known_addr { host: "127.0.222.1" port: 41929 } }
I20260809 12:12:18.960503 13711 leader_election.cc:288] T 64d33967c1b542cfb1a90f22af62f412 P 7e10508ab9f84a08b31e30eb4cba88ba [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 3828629cc754491cb361a1b3e4912a83 (127.0.222.2:37561), b856ed095c0441a2b4407a94090ab4db (127.0.222.3:38419)
I20260809 12:12:18.973289 13700 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 12:12:18.960940 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.222.1:40355 (local address 127.0.222.3:38419)
0809 12:12:18.961284 (+ 344us) server_negotiation.cc:206] Beginning negotiation
0809 12:12:18.961294 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0809 12:12:18.963810 (+ 2516us) server_negotiation.cc:407] Connection header received
0809 12:12:18.964044 (+ 234us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 12:12:18.964049 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 12:12:18.964120 (+ 71us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 12:12:18.964195 (+ 75us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0809 12:12:18.965592 (+ 1397us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 12:12:18.967113 (+ 1521us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 12:12:18.970247 (+ 3134us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 12:12:18.971909 (+ 1662us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 12:12:18.972131 (+ 222us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 12:12:18.972255 (+ 124us) server_negotiation.cc:1035] Waiting for connection context
0809 12:12:18.972797 (+ 542us) server_negotiation.cc:299] Negotiation successful
0809 12:12:18.972932 (+ 135us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":125}
I20260809 12:12:18.973311 13699 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 12:12:18.961303 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.222.1:33491 (local address 127.0.222.2:37561)
0809 12:12:18.961584 (+ 281us) server_negotiation.cc:206] Beginning negotiation
0809 12:12:18.961592 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0809 12:12:18.963171 (+ 1579us) server_negotiation.cc:407] Connection header received
0809 12:12:18.963584 (+ 413us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 12:12:18.963589 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 12:12:18.963656 (+ 67us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 12:12:18.963738 (+ 82us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0809 12:12:18.965161 (+ 1423us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 12:12:18.966710 (+ 1549us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 12:12:18.970112 (+ 3402us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 12:12:18.971979 (+ 1867us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 12:12:18.972166 (+ 187us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 12:12:18.972255 (+ 89us) server_negotiation.cc:1035] Waiting for connection context
0809 12:12:18.972843 (+ 588us) server_negotiation.cc:299] Negotiation successful
0809 12:12:18.972967 (+ 124us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":90}
I20260809 12:12:18.973380 13713 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 12:12:18.961828 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.222.3:38419 (local address 127.0.222.1:40355)
0809 12:12:18.963610 (+ 1782us) negotiation.cc:107] Waiting for socket to connect
0809 12:12:18.963625 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0809 12:12:18.963871 (+ 246us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 12:12:18.964272 (+ 401us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 12:12:18.964285 (+ 13us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 12:12:18.964310 (+ 25us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0809 12:12:18.965388 (+ 1078us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 12:12:18.965399 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 12:12:18.967267 (+ 1868us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 12:12:18.967273 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 12:12:18.970081 (+ 2808us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 12:12:18.970092 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 12:12:18.972350 (+ 2258us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 12:12:18.972374 (+ 24us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 12:12:18.972597 (+ 223us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 12:12:18.972635 (+ 38us) client_negotiation.cc:769] Sending connection context
0809 12:12:18.972743 (+ 108us) client_negotiation.cc:240] Negotiation successful
0809 12:12:18.972886 (+ 143us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1601,"thread_start_us":1155,"threads_started":1}
I20260809 12:12:18.973775 13712 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 12:12:18.960688 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.222.2:37561 (local address 127.0.222.1:33491)
0809 12:12:18.962845 (+ 2157us) negotiation.cc:107] Waiting for socket to connect
0809 12:12:18.962879 (+ 34us) client_negotiation.cc:174] Beginning negotiation
0809 12:12:18.963326 (+ 447us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 12:12:18.963812 (+ 486us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 12:12:18.963819 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 12:12:18.963942 (+ 123us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0809 12:12:18.964940 (+ 998us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 12:12:18.964965 (+ 25us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 12:12:18.966925 (+ 1960us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 12:12:18.966929 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 12:12:18.969910 (+ 2981us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 12:12:18.969932 (+ 22us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 12:12:18.972489 (+ 2557us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 12:12:18.972496 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 12:12:18.972674 (+ 178us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 12:12:18.972716 (+ 42us) client_negotiation.cc:769] Sending connection context
0809 12:12:18.972798 (+ 82us) client_negotiation.cc:240] Negotiation successful
0809 12:12:18.972907 (+ 109us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1910,"mutex_wait_us":4,"spinlock_wait_cycles":24832,"thread_start_us":1398,"threads_started":1}
I20260809 12:12:18.974808 13633 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "64d33967c1b542cfb1a90f22af62f412" candidate_uuid: "7e10508ab9f84a08b31e30eb4cba88ba" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b856ed095c0441a2b4407a94090ab4db" is_pre_election: true
I20260809 12:12:18.975212 13489 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "64d33967c1b542cfb1a90f22af62f412" candidate_uuid: "7e10508ab9f84a08b31e30eb4cba88ba" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3828629cc754491cb361a1b3e4912a83" is_pre_election: true
I20260809 12:12:18.975780 13633 raft_consensus.cc:2454] T 64d33967c1b542cfb1a90f22af62f412 P b856ed095c0441a2b4407a94090ab4db [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 7e10508ab9f84a08b31e30eb4cba88ba in term 0.
I20260809 12:12:18.976302 13489 raft_consensus.cc:2454] T 64d33967c1b542cfb1a90f22af62f412 P 3828629cc754491cb361a1b3e4912a83 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 7e10508ab9f84a08b31e30eb4cba88ba in term 0.
I20260809 12:12:18.976878 13267 leader_election.cc:302] T 64d33967c1b542cfb1a90f22af62f412 P 7e10508ab9f84a08b31e30eb4cba88ba [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 7e10508ab9f84a08b31e30eb4cba88ba, b856ed095c0441a2b4407a94090ab4db; no voters:
I20260809 12:12:18.977415 13711 raft_consensus.cc:2781] T 64d33967c1b542cfb1a90f22af62f412 P 7e10508ab9f84a08b31e30eb4cba88ba [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260809 12:12:18.977615 13711 raft_consensus.cc:482] T 64d33967c1b542cfb1a90f22af62f412 P 7e10508ab9f84a08b31e30eb4cba88ba [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260809 12:12:18.977751 13711 raft_consensus.cc:3037] T 64d33967c1b542cfb1a90f22af62f412 P 7e10508ab9f84a08b31e30eb4cba88ba [term 0 FOLLOWER]: Advancing to term 1
I20260809 12:12:18.980657 13711 raft_consensus.cc:504] T 64d33967c1b542cfb1a90f22af62f412 P 7e10508ab9f84a08b31e30eb4cba88ba [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3828629cc754491cb361a1b3e4912a83" member_type: VOTER last_known_addr { host: "127.0.222.2" port: 37561 } } peers { permanent_uuid: "b856ed095c0441a2b4407a94090ab4db" member_type: VOTER last_known_addr { host: "127.0.222.3" port: 38419 } } peers { permanent_uuid: "7e10508ab9f84a08b31e30eb4cba88ba" member_type: VOTER last_known_addr { host: "127.0.222.1" port: 41929 } }
I20260809 12:12:18.981469 13711 leader_election.cc:288] T 64d33967c1b542cfb1a90f22af62f412 P 7e10508ab9f84a08b31e30eb4cba88ba [CANDIDATE]: Term 1 election: Requested vote from peers 3828629cc754491cb361a1b3e4912a83 (127.0.222.2:37561), b856ed095c0441a2b4407a94090ab4db (127.0.222.3:38419)
I20260809 12:12:18.981926 13489 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "64d33967c1b542cfb1a90f22af62f412" candidate_uuid: "7e10508ab9f84a08b31e30eb4cba88ba" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3828629cc754491cb361a1b3e4912a83"
I20260809 12:12:18.981992 13633 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "64d33967c1b542cfb1a90f22af62f412" candidate_uuid: "7e10508ab9f84a08b31e30eb4cba88ba" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b856ed095c0441a2b4407a94090ab4db"
I20260809 12:12:18.982216 13489 raft_consensus.cc:3037] T 64d33967c1b542cfb1a90f22af62f412 P 3828629cc754491cb361a1b3e4912a83 [term 0 FOLLOWER]: Advancing to term 1
I20260809 12:12:18.982241 13633 raft_consensus.cc:3037] T 64d33967c1b542cfb1a90f22af62f412 P b856ed095c0441a2b4407a94090ab4db [term 0 FOLLOWER]: Advancing to term 1
I20260809 12:12:18.985172 13633 raft_consensus.cc:2454] T 64d33967c1b542cfb1a90f22af62f412 P b856ed095c0441a2b4407a94090ab4db [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 7e10508ab9f84a08b31e30eb4cba88ba in term 1.
I20260809 12:12:18.985285 13489 raft_consensus.cc:2454] T 64d33967c1b542cfb1a90f22af62f412 P 3828629cc754491cb361a1b3e4912a83 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 7e10508ab9f84a08b31e30eb4cba88ba in term 1.
I20260809 12:12:18.985639 13267 leader_election.cc:302] T 64d33967c1b542cfb1a90f22af62f412 P 7e10508ab9f84a08b31e30eb4cba88ba [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 7e10508ab9f84a08b31e30eb4cba88ba, b856ed095c0441a2b4407a94090ab4db; no voters:
I20260809 12:12:18.986091 13711 raft_consensus.cc:2781] T 64d33967c1b542cfb1a90f22af62f412 P 7e10508ab9f84a08b31e30eb4cba88ba [term 1 FOLLOWER]: Leader election won for term 1
I20260809 12:12:18.986899 13711 raft_consensus.cc:686] T 64d33967c1b542cfb1a90f22af62f412 P 7e10508ab9f84a08b31e30eb4cba88ba [term 1 LEADER]: Becoming Leader. State: Replica: 7e10508ab9f84a08b31e30eb4cba88ba, State: Running, Role: LEADER
I20260809 12:12:18.987629 13711 consensus_queue.cc:237] T 64d33967c1b542cfb1a90f22af62f412 P 7e10508ab9f84a08b31e30eb4cba88ba [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3828629cc754491cb361a1b3e4912a83" member_type: VOTER last_known_addr { host: "127.0.222.2" port: 37561 } } peers { permanent_uuid: "b856ed095c0441a2b4407a94090ab4db" member_type: VOTER last_known_addr { host: "127.0.222.3" port: 38419 } } peers { permanent_uuid: "7e10508ab9f84a08b31e30eb4cba88ba" member_type: VOTER last_known_addr { host: "127.0.222.1" port: 41929 } }
I20260809 12:12:18.993484 13161 catalog_manager.cc:5495] T 64d33967c1b542cfb1a90f22af62f412 P 7e10508ab9f84a08b31e30eb4cba88ba reported cstate change: term changed from 0 to 1, leader changed from <none> to 7e10508ab9f84a08b31e30eb4cba88ba (127.0.222.1). New cstate: current_term: 1 leader_uuid: "7e10508ab9f84a08b31e30eb4cba88ba" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3828629cc754491cb361a1b3e4912a83" member_type: VOTER last_known_addr { host: "127.0.222.2" port: 37561 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "b856ed095c0441a2b4407a94090ab4db" member_type: VOTER last_known_addr { host: "127.0.222.3" port: 38419 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "7e10508ab9f84a08b31e30eb4cba88ba" member_type: VOTER last_known_addr { host: "127.0.222.1" port: 41929 } health_report { overall_health: HEALTHY } } }
W20260809 12:12:19.002738 13539 tablet.cc:2369] T 64d33967c1b542cfb1a90f22af62f412 P 3828629cc754491cb361a1b3e4912a83: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260809 12:12:19.079846 13701 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 12:12:19.073427 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39248 (local address 127.0.222.1:41929)
0809 12:12:19.073686 (+ 259us) server_negotiation.cc:206] Beginning negotiation
0809 12:12:19.073694 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0809 12:12:19.073712 (+ 18us) server_negotiation.cc:407] Connection header received
0809 12:12:19.073811 (+ 99us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 12:12:19.073815 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 12:12:19.073916 (+ 101us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 12:12:19.074112 (+ 196us) server_negotiation.cc:226] Negotiated authn=TOKEN
0809 12:12:19.074736 (+ 624us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 12:12:19.075635 (+ 899us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 12:12:19.077301 (+ 1666us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 12:12:19.077833 (+ 532us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 12:12:19.077977 (+ 144us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 12:12:19.078575 (+ 598us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0809 12:12:19.079209 (+ 634us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0809 12:12:19.079373 (+ 164us) server_negotiation.cc:1035] Waiting for connection context
0809 12:12:19.079435 (+ 62us) server_negotiation.cc:299] Negotiation successful
0809 12:12:19.079525 (+ 90us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":77}
I20260809 12:12:19.089555 13489 raft_consensus.cc:1261] T 64d33967c1b542cfb1a90f22af62f412 P 3828629cc754491cb361a1b3e4912a83 [term 1 FOLLOWER]: Refusing update from remote peer 7e10508ab9f84a08b31e30eb4cba88ba: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260809 12:12:19.089555 13633 raft_consensus.cc:1261] T 64d33967c1b542cfb1a90f22af62f412 P b856ed095c0441a2b4407a94090ab4db [term 1 FOLLOWER]: Refusing update from remote peer 7e10508ab9f84a08b31e30eb4cba88ba: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260809 12:12:19.090766 13714 consensus_queue.cc:1035] T 64d33967c1b542cfb1a90f22af62f412 P 7e10508ab9f84a08b31e30eb4cba88ba [LEADER]: Connected to new peer: Peer: permanent_uuid: "3828629cc754491cb361a1b3e4912a83" member_type: VOTER last_known_addr { host: "127.0.222.2" port: 37561 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260809 12:12:19.091353 13711 consensus_queue.cc:1035] T 64d33967c1b542cfb1a90f22af62f412 P 7e10508ab9f84a08b31e30eb4cba88ba [LEADER]: Connected to new peer: Peer: permanent_uuid: "b856ed095c0441a2b4407a94090ab4db" member_type: VOTER last_known_addr { host: "127.0.222.3" port: 38419 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260809 12:12:19.107771 13719 mvcc.cc:205] Tried to move back new op lower bound from 7316592800098557952 to 7316592799697903616. Current Snapshot: MvccSnapshot[applied={T|T < 7316592800098557952}]
I20260809 12:12:19.123912 13161 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:44962:
table { table_name: "test-table" } modify_external_catalogs: true
I20260809 12:12:19.124432 13161 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:44962:
table { table_name: "test-table" } modify_external_catalogs: true
I20260809 12:12:19.128998 13161 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 667e56ed1b824dfab8ec0522ff0f8b2b: Sending DeleteTablet for 3 replicas of tablet 64d33967c1b542cfb1a90f22af62f412
I20260809 12:12:19.130308 13612 tablet_service.cc:1514] Processing DeleteTablet for tablet 64d33967c1b542cfb1a90f22af62f412 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-09 12:12:19 UTC) from {username='kudu', principal='kudu/127.0.222.62@KRBTEST.COM'} at 127.0.0.1:53616
I20260809 12:12:19.130471 13469 tablet_service.cc:1514] Processing DeleteTablet for tablet 64d33967c1b542cfb1a90f22af62f412 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-09 12:12:19 UTC) from {username='kudu', principal='kudu/127.0.222.62@KRBTEST.COM'} at 127.0.0.1:46436
I20260809 12:12:19.130740 13315 tablet_service.cc:1514] Processing DeleteTablet for tablet 64d33967c1b542cfb1a90f22af62f412 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-09 12:12:19 UTC) from {username='kudu', principal='kudu/127.0.222.62@KRBTEST.COM'} at 127.0.0.1:39242
I20260809 12:12:19.135165 888 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskgH5Pjm/build/asan/bin/kudu pid 13244
I20260809 12:12:19.143844 13728 tablet_replica.cc:330] T 64d33967c1b542cfb1a90f22af62f412 P 7e10508ab9f84a08b31e30eb4cba88ba: stopping tablet replica
I20260809 12:12:19.144445 13728 raft_consensus.cc:2229] T 64d33967c1b542cfb1a90f22af62f412 P 7e10508ab9f84a08b31e30eb4cba88ba [term 1 LEADER]: Raft consensus shutting down.
I20260809 12:12:19.145372 13728 raft_consensus.cc:2258] T 64d33967c1b542cfb1a90f22af62f412 P 7e10508ab9f84a08b31e30eb4cba88ba [term 1 FOLLOWER]: Raft consensus is shut down!
I20260809 12:12:19.151849 13727 tablet_replica.cc:330] T 64d33967c1b542cfb1a90f22af62f412 P 3828629cc754491cb361a1b3e4912a83: stopping tablet replica
I20260809 12:12:19.152462 13727 raft_consensus.cc:2229] T 64d33967c1b542cfb1a90f22af62f412 P 3828629cc754491cb361a1b3e4912a83 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260809 12:12:19.153280 13727 raft_consensus.cc:2258] T 64d33967c1b542cfb1a90f22af62f412 P 3828629cc754491cb361a1b3e4912a83 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260809 12:12:19.155754 13728 ts_tablet_manager.cc:1905] T 64d33967c1b542cfb1a90f22af62f412 P 7e10508ab9f84a08b31e30eb4cba88ba: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260809 12:12:19.155936 13727 ts_tablet_manager.cc:1905] T 64d33967c1b542cfb1a90f22af62f412 P 3828629cc754491cb361a1b3e4912a83: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260809 12:12:19.158258 13726 tablet_replica.cc:330] T 64d33967c1b542cfb1a90f22af62f412 P b856ed095c0441a2b4407a94090ab4db: stopping tablet replica
I20260809 12:12:19.158804 13726 raft_consensus.cc:2229] T 64d33967c1b542cfb1a90f22af62f412 P b856ed095c0441a2b4407a94090ab4db [term 1 FOLLOWER]: Raft consensus shutting down.
I20260809 12:12:19.159801 13726 raft_consensus.cc:2258] T 64d33967c1b542cfb1a90f22af62f412 P b856ed095c0441a2b4407a94090ab4db [term 1 FOLLOWER]: Raft consensus is shut down!
I20260809 12:12:19.163362 13726 ts_tablet_manager.cc:1905] T 64d33967c1b542cfb1a90f22af62f412 P b856ed095c0441a2b4407a94090ab4db: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260809 12:12:19.165150 13727 ts_tablet_manager.cc:1918] T 64d33967c1b542cfb1a90f22af62f412 P 3828629cc754491cb361a1b3e4912a83: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260809 12:12:19.165381 13727 log.cc:1196] T 64d33967c1b542cfb1a90f22af62f412 P 3828629cc754491cb361a1b3e4912a83: Deleting WAL directory at /tmp/dist-test-taskgH5Pjm/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786277382073938-888-0/minicluster-data/ts-1/wal/wals/64d33967c1b542cfb1a90f22af62f412
I20260809 12:12:19.166109 13727 ts_tablet_manager.cc:1939] T 64d33967c1b542cfb1a90f22af62f412 P 3828629cc754491cb361a1b3e4912a83: Deleting consensus metadata
Aug 09 12:12:19 dist-test-slave-2w3w krb5kdc[12652](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786277533, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.222.1@KRBTEST.COM
I20260809 12:12:19.168519 13147 catalog_manager.cc:4841] TS 3828629cc754491cb361a1b3e4912a83 (127.0.222.2:37561): tablet 64d33967c1b542cfb1a90f22af62f412 (table test-table [id=ee8bd4b380c040d6b78c289366cb913f]) successfully deleted
I20260809 12:12:19.171432 13726 ts_tablet_manager.cc:1918] T 64d33967c1b542cfb1a90f22af62f412 P b856ed095c0441a2b4407a94090ab4db: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260809 12:12:19.171666 13726 log.cc:1196] T 64d33967c1b542cfb1a90f22af62f412 P b856ed095c0441a2b4407a94090ab4db: Deleting WAL directory at /tmp/dist-test-taskgH5Pjm/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786277382073938-888-0/minicluster-data/ts-2/wal/wals/64d33967c1b542cfb1a90f22af62f412
I20260809 12:12:19.172209 13726 ts_tablet_manager.cc:1939] T 64d33967c1b542cfb1a90f22af62f412 P b856ed095c0441a2b4407a94090ab4db: Deleting consensus metadata
I20260809 12:12:19.174136 13728 ts_tablet_manager.cc:1918] T 64d33967c1b542cfb1a90f22af62f412 P 7e10508ab9f84a08b31e30eb4cba88ba: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260809 12:12:19.174454 13728 log.cc:1196] T 64d33967c1b542cfb1a90f22af62f412 P 7e10508ab9f84a08b31e30eb4cba88ba: Deleting WAL directory at /tmp/dist-test-taskgH5Pjm/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786277382073938-888-0/minicluster-data/ts-0/wal/wals/64d33967c1b542cfb1a90f22af62f412
I20260809 12:12:19.174683 13145 catalog_manager.cc:4841] TS b856ed095c0441a2b4407a94090ab4db (127.0.222.3:38419): tablet 64d33967c1b542cfb1a90f22af62f412 (table test-table [id=ee8bd4b380c040d6b78c289366cb913f]) successfully deleted
I20260809 12:12:19.175652 13728 ts_tablet_manager.cc:1939] T 64d33967c1b542cfb1a90f22af62f412 P 7e10508ab9f84a08b31e30eb4cba88ba: Deleting consensus metadata
I20260809 12:12:19.178042 13147 catalog_manager.cc:4841] TS 7e10508ab9f84a08b31e30eb4cba88ba (127.0.222.1:41929): tablet 64d33967c1b542cfb1a90f22af62f412 (table test-table [id=ee8bd4b380c040d6b78c289366cb913f]) successfully deleted
I20260809 12:12:19.182096 13701 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 12:12:19.136958 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39262 (local address 127.0.222.1:41929)
0809 12:12:19.137156 (+ 198us) server_negotiation.cc:206] Beginning negotiation
0809 12:12:19.137162 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0809 12:12:19.139619 (+ 2457us) server_negotiation.cc:407] Connection header received
0809 12:12:19.139850 (+ 231us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 12:12:19.139855 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 12:12:19.139943 (+ 88us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 12:12:19.140066 (+ 123us) server_negotiation.cc:226] Negotiated authn=SASL
0809 12:12:19.146550 (+ 6484us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 12:12:19.148255 (+ 1705us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 12:12:19.156007 (+ 7752us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 12:12:19.157057 (+ 1050us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 12:12:19.157276 (+ 219us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 12:12:19.169971 (+ 12695us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 12:12:19.169992 (+ 21us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 12:12:19.170026 (+ 34us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0809 12:12:19.170106 (+ 80us) server_negotiation.cc:922] calling sasl_server_start()
0809 12:12:19.177509 (+ 7403us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 12:12:19.179172 (+ 1663us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 12:12:19.179182 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 12:12:19.179185 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0809 12:12:19.179406 (+ 221us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 12:12:19.179924 (+ 518us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 12:12:19.179929 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 12:12:19.179932 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0809 12:12:19.180419 (+ 487us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 12:12:19.180901 (+ 482us) server_negotiation.cc:1035] Waiting for connection context
0809 12:12:19.181210 (+ 309us) server_negotiation.cc:299] Negotiation successful
0809 12:12:19.181530 (+ 320us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":67}
I20260809 12:12:19.183246 13382 generic_service.cc:196] Checking for leaks (request via RPC)
I20260809 12:12:19.286684 888 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskgH5Pjm/build/asan/bin/kudu with pid 13244
I20260809 12:12:19.308223 888 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskgH5Pjm/build/asan/bin/kudu pid 13393
Aug 09 12:12:19 dist-test-slave-2w3w krb5kdc[12652](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786277533, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.222.2@KRBTEST.COM
I20260809 12:12:19.324093 13699 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 12:12:19.309015 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46446 (local address 127.0.222.2:37561)
0809 12:12:19.309214 (+ 199us) server_negotiation.cc:206] Beginning negotiation
0809 12:12:19.309222 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0809 12:12:19.309246 (+ 24us) server_negotiation.cc:407] Connection header received
0809 12:12:19.309477 (+ 231us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 12:12:19.309483 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 12:12:19.309559 (+ 76us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 12:12:19.309670 (+ 111us) server_negotiation.cc:226] Negotiated authn=SASL
0809 12:12:19.310951 (+ 1281us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 12:12:19.311692 (+ 741us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 12:12:19.313079 (+ 1387us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 12:12:19.313593 (+ 514us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 12:12:19.313751 (+ 158us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 12:12:19.318284 (+ 4533us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 12:12:19.318298 (+ 14us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 12:12:19.318314 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0809 12:12:19.318348 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0809 12:12:19.321115 (+ 2767us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 12:12:19.321827 (+ 712us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 12:12:19.321840 (+ 13us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 12:12:19.321842 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0809 12:12:19.321969 (+ 127us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 12:12:19.322378 (+ 409us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 12:12:19.322383 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 12:12:19.322386 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0809 12:12:19.322774 (+ 388us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 12:12:19.323294 (+ 520us) server_negotiation.cc:1035] Waiting for connection context
0809 12:12:19.323471 (+ 177us) server_negotiation.cc:299] Negotiation successful
0809 12:12:19.323717 (+ 246us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":71}
I20260809 12:12:19.324960 13533 generic_service.cc:196] Checking for leaks (request via RPC)
I20260809 12:12:19.416019 888 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskgH5Pjm/build/asan/bin/kudu with pid 13393
I20260809 12:12:19.436666 888 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskgH5Pjm/build/asan/bin/kudu pid 13544
Aug 09 12:12:19 dist-test-slave-2w3w krb5kdc[12652](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786277533, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.222.3@KRBTEST.COM
I20260809 12:12:19.451745 13700 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 12:12:19.437444 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53630 (local address 127.0.222.3:38419)
0809 12:12:19.437609 (+ 165us) server_negotiation.cc:206] Beginning negotiation
0809 12:12:19.437613 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0809 12:12:19.437632 (+ 19us) server_negotiation.cc:407] Connection header received
0809 12:12:19.437713 (+ 81us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 12:12:19.437717 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 12:12:19.437778 (+ 61us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 12:12:19.437871 (+ 93us) server_negotiation.cc:226] Negotiated authn=SASL
0809 12:12:19.439271 (+ 1400us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 12:12:19.440015 (+ 744us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 12:12:19.441283 (+ 1268us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 12:12:19.441714 (+ 431us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 12:12:19.441839 (+ 125us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 12:12:19.446264 (+ 4425us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 12:12:19.446278 (+ 14us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 12:12:19.446294 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0809 12:12:19.446339 (+ 45us) server_negotiation.cc:922] calling sasl_server_start()
0809 12:12:19.449196 (+ 2857us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 12:12:19.449834 (+ 638us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 12:12:19.449841 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 12:12:19.449843 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0809 12:12:19.449984 (+ 141us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 12:12:19.450325 (+ 341us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 12:12:19.450328 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 12:12:19.450330 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0809 12:12:19.450691 (+ 361us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 12:12:19.450929 (+ 238us) server_negotiation.cc:1035] Waiting for connection context
0809 12:12:19.451217 (+ 288us) server_negotiation.cc:299] Negotiation successful
0809 12:12:19.451380 (+ 163us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":57}
I20260809 12:12:19.452569 13677 generic_service.cc:196] Checking for leaks (request via RPC)
I20260809 12:12:19.542002 888 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskgH5Pjm/build/asan/bin/kudu with pid 13544
I20260809 12:12:19.562455 888 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskgH5Pjm/build/asan/bin/kudu pid 13128
I20260809 12:12:19.563747 13194 generic_service.cc:196] Checking for leaks (request via RPC)
I20260809 12:12:19.681895 888 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskgH5Pjm/build/asan/bin/kudu with pid 13128
I20260809 12:12:19.708767 888 mini_ranger_kms.cc:60] Stopping Ranger KMS...
I20260809 12:12:20.053937 888 mini_ranger_kms.cc:62] Stopped Ranger KMS
I20260809 12:12:20.054265 888 mini_ranger.cc:66] Stopping Ranger...
I20260809 12:12:20.399662 888 mini_ranger.cc:68] Stopped Ranger
2026-08-09 12:12:20.399 UTC [12685] LOG: received smart shutdown request
2026-08-09 12:12:20.404 UTC [12685] LOG: background worker "logical replication launcher" (PID 12695) exited with exit code 1
2026-08-09 12:12:20.404 UTC [12690] LOG: shutting down
2026-08-09 12:12:20.580 UTC [12685] LOG: database system is shut down
2026-08-09T12:12:20Z chronyd exiting
[ OK ] SecurityITest.TestEncryptionWithKMSIntegration (53112 ms)
[ RUN ] SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers
Loading random data
Initializing database '/tmp/dist-test-taskgH5Pjm/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786277382073938-888-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 09 12:12:20 dist-test-slave-2w3w krb5kdc[13742](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 09 12:12:20 dist-test-slave-2w3w krb5kdc[13742](info): set up 2 sockets
Aug 09 12:12:20 dist-test-slave-2w3w krb5kdc[13742](info): commencing operation
krb5kdc: starting...
W20260809 12:12:21.937523 888 mini_kdc.cc:121] Time spent starting KDC: real 1.177s user 0.004s sys 0.033s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskgH5Pjm/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786277382073938-888-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskgH5Pjm/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786277382073938-888-0/krb5kdc/test-user.keytab.
Aug 09 12:12:22 dist-test-slave-2w3w krb5kdc[13742](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786277542, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-09T12:12:22Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-09T12:12:22Z Disabled control of system clock
WARNING: no policy specified for rangeradmin/127.0.222.20@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "rangeradmin/127.0.222.20@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal rangeradmin/127.0.222.20@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskgH5Pjm/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786277382073938-888-0/krb5kdc/rangeradmin_127.0.222.20@KRBTEST.COM.keytab.
Entry for principal rangeradmin/127.0.222.20@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskgH5Pjm/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786277382073938-888-0/krb5kdc/rangeradmin_127.0.222.20@KRBTEST.COM.keytab.
WARNING: no policy specified for rangerlookup/127.0.222.20@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "rangerlookup/127.0.222.20@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal rangerlookup/127.0.222.20@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskgH5Pjm/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786277382073938-888-0/krb5kdc/rangerlookup_127.0.222.20@KRBTEST.COM.keytab.
Entry for principal rangerlookup/127.0.222.20@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskgH5Pjm/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786277382073938-888-0/krb5kdc/rangerlookup_127.0.222.20@KRBTEST.COM.keytab.
WARNING: no policy specified for HTTP/127.0.222.20@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "HTTP/127.0.222.20@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.0.222.20@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskgH5Pjm/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786277382073938-888-0/krb5kdc/HTTP_127.0.222.20@KRBTEST.COM.keytab.
Entry for principal HTTP/127.0.222.20@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskgH5Pjm/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786277382073938-888-0/krb5kdc/HTTP_127.0.222.20@KRBTEST.COM.keytab.
I20260809 12:12:22.343284 888 mini_postgres.cc:62] Running initdb...
The files belonging to this database system will be owned by user "slave".
This user must also own the server process.
The database cluster will be initialized with locale "C".
The default database encoding has accordingly been set to "SQL_ASCII".
The default text search configuration will be set to "english".
Data page checksums are disabled.
creating directory /tmp/dist-test-taskgH5Pjm/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786277382073938-888-0/minicluster-data/postgres ... ok
creating subdirectories ... ok
selecting dynamic shared memory implementation ... posix
selecting default max_connections ... 100
selecting default shared_buffers ... 128MB
selecting default time zone ... Etc/UTC
creating configuration files ... ok
running bootstrap script ... ok
performing post-bootstrap initialization ... ok
syncing data to disk ... ok
initdb: warning: enabling "trust" authentication for local connections
You can change this by editing pg_hba.conf or using the option -A, or
--auth-local and --auth-host, the next time you run initdb.
Success. You can now start the database server using:
/tmp/dist-test-taskgH5Pjm/build/asan/bin/postgres/pg_ctl -D /tmp/dist-test-taskgH5Pjm/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786277382073938-888-0/minicluster-data/postgres -l logfile start
2026-08-09 12:12:24.890 UTC [13775] LOG: starting PostgreSQL 12.2 on x86_64-pc-linux-gnu, compiled by cc (GCC) 8.3.1 20190311 (Red Hat 8.3.1-3), 64-bit
2026-08-09 12:12:24.890 UTC [13775] LOG: listening on IPv4 address "127.0.222.20", port 47669
2026-08-09 12:12:24.894 UTC [13775] LOG: listening on Unix socket "/tmp/.s.PGSQL.47669"
2026-08-09 12:12:24.920 UTC [13778] LOG: database system was shut down at 2026-08-09 12:12:23 UTC
2026-08-09 12:12:24.924 UTC [13775] LOG: database system is ready to accept connections
I20260809 12:12:25.952730 888 mini_postgres.cc:96] Postgres bound to 47669
2026-08-09 12:12:25.976 UTC [13786] FATAL: database "slave" does not exist
127.0.222.20:47669 - accepting connections
I20260809 12:12:25.977435 888 mini_ranger.cc:161] Starting Ranger...
I20260809 12:12:26.009681 888 mini_ranger.cc:84] Created miniranger Postgres user
I20260809 12:12:26.501577 888 mini_ranger.cc:87] Created ranger Postgres database
I20260809 12:12:26.501686 888 mini_ranger.cc:178] Starting Ranger out of /tmp/dist-test-taskgH5Pjm/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786277382073938-888-0/minicluster-data/ranger-admin
2026-08-09 12:12:26,791 [I] DB FLAVOR :POSTGRES
2026-08-09 12:12:26,791 [I] --------- Verifying Ranger DB connection ---------
2026-08-09 12:12:26,792 [I] Checking connection..
2026-08-09 12:12:26,792 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskgH5Pjm/build/asan/bin/postgresql.jar:/tmp/dist-test-taskgH5Pjm/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.222.20:47669/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select 1;"
2026-08-09 12:12:27,057 [I] Checking connection passed.
2026-08-09 12:12:27,057 [I] --------- Verifying version history table ---------
2026-08-09 12:12:27,057 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskgH5Pjm/build/asan/bin/postgresql.jar:/tmp/dist-test-taskgH5Pjm/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.222.20:47669/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_db_version_h') as temp;"
2026-08-09 12:12:27,306 [I] Table x_db_version_h does not exist in database ranger
2026-08-09 12:12:27,307 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskgH5Pjm/build/asan/bin/postgresql.jar:/tmp/dist-test-taskgH5Pjm/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.222.20:47669/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_db_version_h') as temp;"
2026-08-09 12:12:27,579 [I] Table x_db_version_h does not exist in database ranger
2026-08-09 12:12:27,579 [I] Importing x_db_version_h table schema to database ranger from file: create_dbversion_catalog.sql
2026-08-09 12:12:27,579 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskgH5Pjm/build/asan/bin/postgresql.jar:/tmp/dist-test-taskgH5Pjm/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.222.20:47669/ranger -u miniranger -p '********' -noheader -trim -c \; -input /tmp/dist-test-taskgH5Pjm/thirdparty/src/ranger-2.1.0-admin/db/postgres/create_dbversion_catalog.sql
2026-08-09 12:12:27.843 UTC [13853] WARNING: there is no transaction in progress
2026-08-09 12:12:27,851 [I] create_dbversion_catalog.sql file imported successfully
2026-08-09 12:12:27,851 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskgH5Pjm/build/asan/bin/postgresql.jar:/tmp/dist-test-taskgH5Pjm/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.222.20:47669/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_db_version_h') as temp;"
2026-08-09 12:12:28,100 [I] Table x_db_version_h already exists in database 'ranger'
2026-08-09 12:12:28,100 [I] --------- Importing Ranger Core DB Schema ---------
2026-08-09 12:12:28,100 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskgH5Pjm/build/asan/bin/postgresql.jar:/tmp/dist-test-taskgH5Pjm/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.222.20:47669/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'CORE_DB_SCHEMA' and active = 'Y';"
2026-08-09 12:12:28,340 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskgH5Pjm/build/asan/bin/postgresql.jar:/tmp/dist-test-taskgH5Pjm/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.222.20:47669/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'CORE_DB_SCHEMA' and active = 'N';"
2026-08-09 12:12:28,610 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskgH5Pjm/build/asan/bin/postgresql.jar:/tmp/dist-test-taskgH5Pjm/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.222.20:47669/ranger -u miniranger -p '********' -noheader -trim -c \; -query "insert into x_db_version_h (version, inst_at, inst_by, updated_at, updated_by,active) values ('CORE_DB_SCHEMA', current_timestamp, 'Ranger 2.1.0', current_timestamp, 'dist-test-slave-2w3w.c.gcp-upstream.internal','N') ;"
2026-08-09 12:12:28,871 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskgH5Pjm/build/asan/bin/postgresql.jar:/tmp/dist-test-taskgH5Pjm/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.222.20:47669/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_portal_user') as temp;"
2026-08-09 12:12:29,136 [I] Table x_portal_user does not exist in database ranger
2026-08-09 12:12:29,137 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskgH5Pjm/build/asan/bin/postgresql.jar:/tmp/dist-test-taskgH5Pjm/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.222.20:47669/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_policy_ref_group') as temp;"
2026-08-09 12:12:29,409 [I] Table x_policy_ref_group does not exist in database ranger
2026-08-09 12:12:29,409 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskgH5Pjm/build/asan/bin/postgresql.jar:/tmp/dist-test-taskgH5Pjm/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.222.20:47669/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'DB_PATCHES' and active = 'Y';"
2026-08-09 12:12:29,672 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskgH5Pjm/build/asan/bin/postgresql.jar:/tmp/dist-test-taskgH5Pjm/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.222.20:47669/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'JAVA_PATCHES' and active = 'Y';"
2026-08-09 12:12:29,931 [I] Importing DB schema to database ranger from file: ranger_core_db_postgres.sql
2026-08-09 12:12:29,931 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskgH5Pjm/build/asan/bin/postgresql.jar:/tmp/dist-test-taskgH5Pjm/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.222.20:47669/ranger -u miniranger -p '********' -noheader -trim -c \; -input /tmp/dist-test-taskgH5Pjm/thirdparty/src/ranger-2.1.0-admin/db/postgres/optimized/current/ranger_core_db_postgres.sql
2026-08-09 12:12:30.922 UTC [13966] WARNING: there is no transaction in progress
2026-08-09 12:12:30.939 UTC [13966] WARNING: there is no transaction in progress
2026-08-09 12:12:30.955 UTC [13966] WARNING: there is no transaction in progress
2026-08-09 12:12:30.971 UTC [13966] WARNING: there is no transaction in progress
2026-08-09 12:12:30.987 UTC [13966] WARNING: there is no transaction in progress
2026-08-09 12:12:31.005 UTC [13966] WARNING: there is no transaction in progress
2026-08-09 12:12:31.061 UTC [13966] WARNING: there is no transaction in progress
2026-08-09 12:12:31.068 UTC [13966] WARNING: there is no transaction in progress
2026-08-09 12:12:31.081 UTC [13966] WARNING: there is no transaction in progress
2026-08-09 12:12:31.091 UTC [13966] WARNING: there is no transaction in progress
2026-08-09 12:12:31.101 UTC [13966] WARNING: there is no transaction in progress
2026-08-09 12:12:31.114 UTC [13966] WARNING: there is no transaction in progress
2026-08-09 12:12:31.124 UTC [13966] WARNING: there is no transaction in progress
2026-08-09 12:12:31.132 UTC [13966] WARNING: there is no transaction in progress
2026-08-09 12:12:31.139 UTC [13966] WARNING: there is no transaction in progress
2026-08-09 12:12:31.146 UTC [13966] WARNING: there is no transaction in progress
2026-08-09 12:12:31,794 [I] ranger_core_db_postgres.sql file imported successfully
2026-08-09 12:12:31,795 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskgH5Pjm/build/asan/bin/postgresql.jar:/tmp/dist-test-taskgH5Pjm/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.222.20:47669/ranger -u miniranger -p '********' -noheader -trim -c \; -query "update x_db_version_h set inst_by='Ranger 2.1.0' where active='Y' and updated_by='localhost';"
2026-08-09 12:12:32,056 [I] Patches status entries updated from base ranger version to current installed ranger version:Ranger 2.1.0
2026-08-09 12:12:32,056 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskgH5Pjm/build/asan/bin/postgresql.jar:/tmp/dist-test-taskgH5Pjm/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.222.20:47669/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_portal_user') as temp;"
2026-08-09 12:12:32,327 [I] Table x_portal_user already exists in database 'ranger'
2026-08-09 12:12:32,328 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskgH5Pjm/build/asan/bin/postgresql.jar:/tmp/dist-test-taskgH5Pjm/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.222.20:47669/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_policy_ref_group') as temp;"
2026-08-09 12:12:32,567 [I] Table x_policy_ref_group already exists in database 'ranger'
2026-08-09 12:12:32,567 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskgH5Pjm/build/asan/bin/postgresql.jar:/tmp/dist-test-taskgH5Pjm/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.222.20:47669/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'DB_PATCHES' and active = 'Y';"
2026-08-09 12:12:32,833 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskgH5Pjm/build/asan/bin/postgresql.jar:/tmp/dist-test-taskgH5Pjm/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.222.20:47669/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'JAVA_PATCHES' and active = 'Y';"
2026-08-09 12:12:33,091 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskgH5Pjm/build/asan/bin/postgresql.jar:/tmp/dist-test-taskgH5Pjm/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.222.20:47669/ranger -u miniranger -p '********' -noheader -trim -c \; -query "update x_db_version_h set active='Y' where version='CORE_DB_SCHEMA' and active='N' and updated_by='dist-test-slave-2w3w.c.gcp-upstream.internal';"
2026-08-09 12:12:33,352 [I] CORE_DB_SCHEMA import status has been updated
2026-08-09 12:12:33,352 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskgH5Pjm/build/asan/bin/postgresql.jar:/tmp/dist-test-taskgH5Pjm/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.222.20:47669/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'DB_PATCHES' and inst_by = 'Ranger 2.1.0' and active = 'Y';"
2026-08-09 12:12:33,619 [I] DB_PATCHES have already been applied
I20260809 12:12:33.625340 888 mini_ranger.cc:191] Using Ranger class path: /tmp/dist-test-taskgH5Pjm/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786277382073938-888-0/minicluster-data/ranger-admin:/tmp/dist-test-taskgH5Pjm/thirdparty/src/ranger-2.1.0-admin/ews/lib/*:/usr/lib/jvm/java-1.8.0-openjdk-amd64/lib/*:/tmp/dist-test-taskgH5Pjm/thirdparty/src/hadoop-3.2.0/*:/tmp/dist-test-taskgH5Pjm/build/asan/bin/postgresql.jar:/tmp/dist-test-taskgH5Pjm/thirdparty/src/ranger-2.1.0-admin/ews/webapp
I20260809 12:12:33.625430 888 mini_ranger.cc:193] Using host: 127.0.222.20
I20260809 12:12:33.645221 888 mini_ranger.cc:227] Ranger admin URL: http://127.0.222.20:51411
Aug 09, 2026 12:12:33 PM org.apache.ranger.server.tomcat.EmbeddedServer getKeyManagers
WARNING: Config 'ranger.keystore.file' or 'ranger.service.https.attrib.keystore.file' is not found or contains blank value
Aug 09, 2026 12:12:33 PM org.apache.ranger.server.tomcat.EmbeddedServer getTrustManagers
WARNING: Config 'ranger.truststore.file' is not found or contains blank value!
Aug 09, 2026 12:12:34 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Deriving webapp folder from catalina.base property. folder=/tmp/dist-test-taskgH5Pjm/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786277382073938-888-0/minicluster-data/ranger-admin/ews/webapp
Aug 09, 2026 12:12:34 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Webapp file =/tmp/dist-test-taskgH5Pjm/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786277382073938-888-0/minicluster-data/ranger-admin/ews/webapp, webAppName = /
Aug 09, 2026 12:12:34 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Adding webapp [/] = path [/tmp/dist-test-taskgH5Pjm/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786277382073938-888-0/minicluster-data/ranger-admin/ews/webapp] .....
Aug 09, 2026 12:12:34 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Finished init of webapp [/] = path [/tmp/dist-test-taskgH5Pjm/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786277382073938-888-0/minicluster-data/ranger-admin/ews/webapp].
Aug 09, 2026 12:12:34 PM org.apache.ranger.server.tomcat.EmbeddedServer startServer
INFO: Server Name : miniranger
I20260809 12:12:34.858310 888 mini_ranger.cc:160] Time spent starting Ranger: real 8.881s user 0.000s sys 0.059s
Aug 09, 2026 12:12:55 PM com.sun.jersey.api.core.PackagesResourceConfig init
INFO: Scanning for root resource and provider classes in the packages:
org.apache.ranger.rest
org.apache.ranger.common
xa.rest
Aug 09, 2026 12:12:55 PM com.sun.jersey.api.core.ScanningResourceConfig logClasses
INFO: Root resource classes found:
class org.apache.ranger.rest.XAuditREST
class org.apache.ranger.rest.SecurityZoneREST
class org.apache.ranger.rest.PublicAPIsv2
class org.apache.ranger.rest.XKeyREST
class org.apache.ranger.rest.RoleREST
class org.apache.ranger.rest.ServiceREST
class org.apache.ranger.rest.AssetREST
class org.apache.ranger.rest.XUserREST
class org.apache.ranger.rest.TagREST
class org.apache.ranger.rest.MetricsREST
class org.apache.ranger.rest.PublicAPIs
class org.apache.ranger.rest.UserREST
Aug 09, 2026 12:12:55 PM com.sun.jersey.api.core.ScanningResourceConfig logClasses
INFO: Provider classes found:
class org.apache.ranger.common.RangerJAXBContextResolver
Aug 09, 2026 12:12:55 PM com.sun.jersey.spi.spring.container.servlet.SpringServlet getContext
INFO: Using default applicationContext
Aug 09, 2026 12:12:55 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, assetREST, of type org.apache.ranger.rest.AssetREST as a root resource class
Aug 09, 2026 12:12:55 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, metricsREST, of type org.apache.ranger.rest.MetricsREST as a root resource class
Aug 09, 2026 12:12:55 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, publicAPIs, of type org.apache.ranger.rest.PublicAPIs as a root resource class
Aug 09, 2026 12:12:55 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, publicAPIsv2, of type org.apache.ranger.rest.PublicAPIsv2 as a root resource class
Aug 09, 2026 12:12:55 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, roleREST, of type org.apache.ranger.rest.RoleREST as a root resource class
Aug 09, 2026 12:12:55 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, securityZoneREST, of type org.apache.ranger.rest.SecurityZoneREST as a root resource class
Aug 09, 2026 12:12:55 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, serviceREST, of type org.apache.ranger.rest.ServiceREST as a root resource class
Aug 09, 2026 12:12:55 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, tagREST, of type org.apache.ranger.rest.TagREST as a root resource class
Aug 09, 2026 12:12:55 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, userREST, of type org.apache.ranger.rest.UserREST as a root resource class
Aug 09, 2026 12:12:55 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, XAuditREST, of type org.apache.ranger.rest.XAuditREST as a root resource class
Aug 09, 2026 12:12:55 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, XKeyREST, of type org.apache.ranger.rest.XKeyREST as a root resource class
Aug 09, 2026 12:12:55 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, XUserREST, of type org.apache.ranger.rest.XUserREST as a root resource class
Aug 09, 2026 12:12:55 PM com.sun.jersey.server.impl.application.WebApplicationImpl _initiate
INFO: Initiating Jersey application, version 'Jersey: 1.19.3 10/24/2016 03:58 PM'
Aug 09, 2026 12:12:56 PM com.sun.jersey.spi.inject.Errors processErrorMessages
WARNING: The following warnings have been detected with resource and/or provider classes:
WARNING: A HTTP GET method, public void org.apache.ranger.rest.ServiceREST.getPoliciesInExcel(javax.servlet.http.HttpServletRequest,javax.servlet.http.HttpServletResponse), MUST return a non-void type.
WARNING: A HTTP GET method, public void org.apache.ranger.rest.ServiceREST.getPoliciesInJson(javax.servlet.http.HttpServletRequest,javax.servlet.http.HttpServletResponse,java.lang.Boolean), MUST return a non-void type.
WARNING: A HTTP GET method, public void org.apache.ranger.rest.ServiceREST.getPoliciesInCsv(javax.servlet.http.HttpServletRequest,javax.servlet.http.HttpServletResponse) throws java.io.IOException, MUST return a non-void type.
I20260809 12:12:56.855161 888 mini_ranger.cc:261] Created Kudu service
WARNING: no policy specified for rangerkms/127.0.222.20@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "rangerkms/127.0.222.20@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal rangerkms/127.0.222.20@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskgH5Pjm/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786277382073938-888-0/krb5kdc/rangerkms_127.0.222.20@KRBTEST.COM.keytab.
Entry for principal rangerkms/127.0.222.20@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskgH5Pjm/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786277382073938-888-0/krb5kdc/rangerkms_127.0.222.20@KRBTEST.COM.keytab.
WARNING: no policy specified for HTTP/127.0.222.20@KRBTEST.COM; defaulting to no policy
add_principal: Principal or policy already exists while creating "HTTP/127.0.222.20@KRBTEST.COM".
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.0.222.20@KRBTEST.COM with kvno 3, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskgH5Pjm/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786277382073938-888-0/krb5kdc/HTTP_127.0.222.20@KRBTEST.COM.keytab.
Entry for principal HTTP/127.0.222.20@KRBTEST.COM with kvno 3, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskgH5Pjm/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786277382073938-888-0/krb5kdc/HTTP_127.0.222.20@KRBTEST.COM.keytab.
WARNING: no policy specified for keyadmin@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "keyadmin@KRBTEST.COM" created.
Aug 09 12:12:57 dist-test-slave-2w3w krb5kdc[13742](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786277577, etypes {rep=17 tkt=17 ses=17}, keyadmin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for keyadmin@KRBTEST.COM:
I20260809 12:12:57.078738 888 mini_ranger_kms.cc:208] Starting Ranger KMS...
I20260809 12:12:57.105988 888 mini_ranger_kms.cc:78] Created minirangerkms Postgres user
I20260809 12:12:57.972720 888 mini_ranger_kms.cc:81] Created rangerkms Postgres database
I20260809 12:12:57.972888 888 mini_ranger_kms.cc:226] Starting Ranger KMS out of /tmp/dist-test-taskgH5Pjm/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786277382073938-888-0/minicluster-data/ranger-kms
I20260809 12:12:57.972950 888 mini_ranger_kms.cc:227] Using postgres at 127.0.222.20:47669
2026-08-09 12:12:58,346 [I] DB FLAVOR :POSTGRES
2026-08-09 12:12:58,346 [I] --------- Verifying Ranger DB connection ---------
2026-08-09 12:12:58,346 [I] Checking connection
2026-08-09 12:12:58,346 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskgH5Pjm/build/asan/bin/postgresql.jar:/tmp/dist-test-taskgH5Pjm/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786277382073938-888-0/minicluster-data/ranger-kms/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.222.20:47669/rangerkms -u rangerkms -p '********' -noheader -trim -c \; -query "SELECT 1;"
2026-08-09 12:12:58,579 [I] connection success
2026-08-09 12:12:58,579 [I] --------- Verifying Ranger DB tables ---------
2026-08-09 12:12:58,579 [I] Verifying table ranger_masterkey in database rangerkms
2026-08-09 12:12:58,579 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskgH5Pjm/build/asan/bin/postgresql.jar:/tmp/dist-test-taskgH5Pjm/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786277382073938-888-0/minicluster-data/ranger-kms/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.222.20:47669/rangerkms -u rangerkms -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='rangerkms' and table_name = 'ranger_masterkey') as temp;"
2026-08-09 12:12:58,821 [I] Table ranger_masterkey does not exist in database rangerkms
2026-08-09 12:12:58,821 [I] --------- Importing Ranger Core DB Schema ---------
2026-08-09 12:12:58,821 [I] Importing db schema to database rangerkms from file: kms_core_db_postgres.sql
2026-08-09 12:12:58,821 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskgH5Pjm/build/asan/bin/postgresql.jar:/tmp/dist-test-taskgH5Pjm/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786277382073938-888-0/minicluster-data/ranger-kms/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.222.20:47669/rangerkms -u rangerkms -p '********' -noheader -trim -c \; -input /tmp/dist-test-taskgH5Pjm/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786277382073938-888-0/minicluster-data/ranger-kms/db/postgres/kms_core_db_postgres.sql
2026-08-09 12:12:59,114 [I] kms_core_db_postgres.sql DB schema imported successfully
I20260809 12:12:59.609328 888 mini_ranger_kms.cc:310] Created kms service
I20260809 12:12:59.780637 888 mini_ranger_kms.cc:326] Created kudu user
I20260809 12:12:59.828378 888 mini_ranger_kms.cc:343] Created rangerkms user
Aug 09 12:12:59 dist-test-slave-2w3w krb5kdc[13742](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786277577, etypes {rep=17 tkt=17 ses=17}, keyadmin@KRBTEST.COM for HTTP/127.0.222.20@KRBTEST.COM
I20260809 12:13:00.269131 888 mini_ranger_kms.cc:384] Added ranger policy
I20260809 12:13:00.269484 888 mini_ranger_kms.cc:238] Using RangerKMS classpath: /tmp/dist-test-taskgH5Pjm/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786277382073938-888-0/minicluster-data/ranger-kms:/tmp/dist-test-taskgH5Pjm/build/asan/bin/postgresql.jar:/tmp/dist-test-taskgH5Pjm/thirdparty/src/ranger-2.1.0-kms/ews/webapp/WEB-INF/classes/lib/*:/tmp/dist-test-taskgH5Pjm/thirdparty/src/ranger-2.1.0-kms/ews/webapp/lib/*:/usr/lib/jvm/java-1.8.0-openjdk-amd64/lib/*
I20260809 12:13:00.269551 888 mini_ranger_kms.cc:240] Using host: 127.0.222.20
I20260809 12:13:00.295323 888 mini_ranger_kms.cc:276] Ranger KMS PID: 14152
I20260809 12:13:00.295466 888 mini_ranger_kms.cc:277] Ranger KMS URL: http://127.0.222.20:55861
Aug 09, 2026 12:13:01 PM org.apache.ranger.server.tomcat.EmbeddedServer getKeyManagers
WARNING: Config 'ranger.keystore.file' or 'ranger.service.https.attrib.keystore.file' is not found or contains blank value
Aug 09, 2026 12:13:01 PM org.apache.ranger.server.tomcat.EmbeddedServer getTrustManagers
WARNING: Config 'ranger.truststore.file' is not found or contains blank value!
Aug 09, 2026 12:13:01 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Webapp file =/tmp/dist-test-taskgH5Pjm/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786277382073938-888-0/minicluster-data/ranger-kms/ews/webapp, webAppName = /kms
Aug 09, 2026 12:13:01 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Adding webapp [/kms] = path [/tmp/dist-test-taskgH5Pjm/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786277382073938-888-0/minicluster-data/ranger-kms/ews/webapp] .....
Aug 09, 2026 12:13:01 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Finished init of webapp [/kms] = path [/tmp/dist-test-taskgH5Pjm/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786277382073938-888-0/minicluster-data/ranger-kms/ews/webapp].
Aug 09, 2026 12:13:01 PM org.apache.ranger.server.tomcat.EmbeddedServer startServer
INFO: Server Name : minirangerkms
Aug 09 12:13:02 dist-test-slave-2w3w krb5kdc[13742](info): AS_REQ (1 etypes {17}) 127.0.0.1: ISSUE: authtime 1786277582, etypes {rep=17 tkt=17 ses=17}, rangerkms/127.0.222.20@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260809 12:13:02.951774 888 mini_ranger_kms.cc:207] Time spent starting Ranger KMS: real 5.873s user 0.067s sys 0.342s
I20260809 12:13:02.952042 888 mini_ranger_kms.cc:397] {"name":"kuduclusterkey","cipher":"AES/CTR/NoPadding","length":128,"description":"kuduclusterkey"}
I20260809 12:13:02.952224 888 mini_ranger_kms.cc:401] 127.0.222.20:55861/kms/v1/keys
Aug 09 12:13:03 dist-test-slave-2w3w krb5kdc[13742](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786277582, etypes {rep=17 tkt=17 ses=17}, rangerkms/127.0.222.20@KRBTEST.COM for HTTP/127.0.222.20@KRBTEST.COM
Aug 09 12:13:03 dist-test-slave-2w3w krb5kdc[13742](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786277582, etypes {rep=17 tkt=17 ses=17}, rangerkms/127.0.222.20@KRBTEST.COM for HTTP/127.0.222.20@KRBTEST.COM
Aug 09 12:13:05 dist-test-slave-2w3w krb5kdc[13742](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786277585, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
WARNING: no policy specified for kudu/127.0.222.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.222.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.222.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskgH5Pjm/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786277382073938-888-0/krb5kdc/kudu_127.0.222.62.keytab.
Entry for principal kudu/127.0.222.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskgH5Pjm/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786277382073938-888-0/krb5kdc/kudu_127.0.222.62.keytab.
I20260809 12:13:05.938498 888 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskgH5Pjm/build/asan/bin/kudu
/tmp/dist-test-taskgH5Pjm/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskgH5Pjm/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786277382073938-888-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskgH5Pjm/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786277382073938-888-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskgH5Pjm/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786277382073938-888-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskgH5Pjm/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786277382073938-888-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.222.62:37183
--webserver_interface=127.0.222.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskgH5Pjm/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786277382073938-888-0/krb5kdc/kudu_127.0.222.62.keytab
--principal=kudu/127.0.222.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.222.20:37035
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.222.62:37183
--ranger_config_path=/tmp/dist-test-taskgH5Pjm/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786277382073938-888-0/minicluster-data/ranger-client
--trusted_user_acl=test-admin
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.0.222.20:55861/kms
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskgH5Pjm/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786277382073938-888-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskgH5Pjm/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786277382073938-888-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskgH5Pjm/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786277382073938-888-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260809 12:13:06.282507 14208 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260809 12:13:06.282859 14208 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260809 12:13:06.283061 14208 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260809 12:13:06.290380 14208 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260809 12:13:06.290524 14208 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260809 12:13:06.290632 14208 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260809 12:13:06.290724 14208 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260809 12:13:06.290818 14208 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260809 12:13:06.301179 14208 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.222.20:37035
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskgH5Pjm/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786277382073938-888-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskgH5Pjm/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786277382073938-888-0/minicluster-data/master-0/wal
--ranger_kms_url=127.0.222.20:55861/kms
--trusted_user_acl=<redacted>
--ipki_ca_key_size=768
--master_addresses=127.0.222.62:37183
--ranger_config_path=/tmp/dist-test-taskgH5Pjm/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786277382073938-888-0/minicluster-data/ranger-client
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskgH5Pjm/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786277382073938-888-0/krb5kdc/kudu_127.0.222.62.keytab
--principal=kudu/127.0.222.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.222.62:37183
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskgH5Pjm/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786277382073938-888-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.222.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskgH5Pjm/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786277382073938-888-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 09 Aug 2026 11:06:41 UTC on ccycloud.quasar-bwoogy.root.comops.site
build id 4575991
ASAN enabled
I20260809 12:13:06.302506 14208 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260809 12:13:06.303905 14208 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260809 12:13:06.316052 14213 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 12:13:06.316088 14214 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 12:13:06.316009 14216 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260809 12:13:06.316435 14208 server_base.cc:1034] running on GCE node
I20260809 12:13:06.317420 14208 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260809 12:13:06.318830 14208 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260809 12:13:06.320104 14208 hybrid_clock.cc:648] HybridClock initialized: now 1786277586320060 us; error 50 us; skew 500 ppm
I20260809 12:13:06.320478 14208 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 09 12:13:06 dist-test-slave-2w3w krb5kdc[13742](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786277586, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.222.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260809 12:13:06.329505 14208 init.cc:377] Logged in from keytab as kudu/127.0.222.62@KRBTEST.COM (short username kudu)
I20260809 12:13:06.331212 14208 webserver.cc:460] Webserver started at http://127.0.222.62:45661/ using document root <none> and password file <none>
I20260809 12:13:06.331905 14208 fs_manager.cc:359] Metadata directory not provided
I20260809 12:13:06.332043 14208 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260809 12:13:06.332365 14208 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Aug 09 12:13:06 dist-test-slave-2w3w krb5kdc[13742](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786277586, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.222.62@KRBTEST.COM for HTTP/127.0.222.20@KRBTEST.COM
I20260809 12:13:06.396641 14208 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskgH5Pjm/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786277382073938-888-0/minicluster-data/master-0/data/instance:
uuid: "8751c8f15ead4a2a96a78af1bd301cc9"
format_stamp: "Formatted at 2026-08-09 12:13:06 on dist-test-slave-2w3w"
server_key: "55d2a29dc2c9ba6877af46e2fda3af0c"
server_key_iv: "27c59f6cb2df850e0fdf963d02600503"
server_key_version: "kuduclusterkey@0"
I20260809 12:13:06.397861 14208 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskgH5Pjm/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786277382073938-888-0/minicluster-data/master-0/wal/instance:
uuid: "8751c8f15ead4a2a96a78af1bd301cc9"
format_stamp: "Formatted at 2026-08-09 12:13:06 on dist-test-slave-2w3w"
server_key: "55d2a29dc2c9ba6877af46e2fda3af0c"
server_key_iv: "27c59f6cb2df850e0fdf963d02600503"
server_key_version: "kuduclusterkey@0"
I20260809 12:13:06.405400 14208 fs_manager.cc:691] Time spent creating directory manager: real 0.007s user 0.005s sys 0.004s
I20260809 12:13:06.432863 14223 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260809 12:13:06.435082 14208 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.005s sys 0.000s
I20260809 12:13:06.435421 14208 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskgH5Pjm/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786277382073938-888-0/minicluster-data/master-0/data,/tmp/dist-test-taskgH5Pjm/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786277382073938-888-0/minicluster-data/master-0/wal
uuid: "8751c8f15ead4a2a96a78af1bd301cc9"
format_stamp: "Formatted at 2026-08-09 12:13:06 on dist-test-slave-2w3w"
server_key: "55d2a29dc2c9ba6877af46e2fda3af0c"
server_key_iv: "27c59f6cb2df850e0fdf963d02600503"
server_key_version: "kuduclusterkey@0"
I20260809 12:13:06.435730 14208 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskgH5Pjm/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786277382073938-888-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskgH5Pjm/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786277382073938-888-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskgH5Pjm/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786277382073938-888-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260809 12:13:06.473603 14208 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260809 12:13:06.477437 14208 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260809 12:13:06.477772 14208 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260809 12:13:06.496563 14208 rpc_server.cc:307] RPC server started. Bound to: 127.0.222.62:37183
I20260809 12:13:06.496600 14277 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.222.62:37183 every 8 connection(s)
I20260809 12:13:06.497951 14208 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskgH5Pjm/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786277382073938-888-0/minicluster-data/master-0/data/info.pb
I20260809 12:13:06.500793 888 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskgH5Pjm/build/asan/bin/kudu as pid 14208
I20260809 12:13:06.501008 888 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskgH5Pjm/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786277382073938-888-0/minicluster-data/master-0/wal/instance
I20260809 12:13:06.501741 888 external_mini_cluster.cc:1428] Setting key 7ff888b7e8e390425d856cc8d7898526
I20260809 12:13:06.501940 14278 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260809 12:13:06.516588 14278 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 8751c8f15ead4a2a96a78af1bd301cc9: Bootstrap starting.
Aug 09 12:13:06 dist-test-slave-2w3w krb5kdc[13742](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786277585, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.222.62@KRBTEST.COM
I20260809 12:13:06.520284 14278 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 8751c8f15ead4a2a96a78af1bd301cc9: Neither blocks nor log segments found. Creating new log.
I20260809 12:13:06.521363 14278 log.cc:824] T 00000000000000000000000000000000 P 8751c8f15ead4a2a96a78af1bd301cc9: Log is configured to *not* fsync() on all Append() calls
I20260809 12:13:06.524317 14278 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 8751c8f15ead4a2a96a78af1bd301cc9: No bootstrap required, opened a new log
I20260809 12:13:06.525457 14281 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 12:13:06.503803 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43758 (local address 127.0.222.62:37183)
0809 12:13:06.504708 (+ 905us) server_negotiation.cc:206] Beginning negotiation
0809 12:13:06.504723 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0809 12:13:06.504779 (+ 56us) server_negotiation.cc:407] Connection header received
0809 12:13:06.506937 (+ 2158us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 12:13:06.506996 (+ 59us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 12:13:06.507518 (+ 522us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 12:13:06.507974 (+ 456us) server_negotiation.cc:226] Negotiated authn=SASL
0809 12:13:06.509170 (+ 1196us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 12:13:06.511325 (+ 2155us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 12:13:06.512608 (+ 1283us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 12:13:06.513715 (+ 1107us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 12:13:06.513917 (+ 202us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 12:13:06.518238 (+ 4321us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 12:13:06.518246 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 12:13:06.518260 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0809 12:13:06.518299 (+ 39us) server_negotiation.cc:922] calling sasl_server_start()
0809 12:13:06.521412 (+ 3113us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 12:13:06.522130 (+ 718us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 12:13:06.522137 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 12:13:06.522146 (+ 9us) server_negotiation.cc:967] calling sasl_server_step()
0809 12:13:06.522299 (+ 153us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 12:13:06.522695 (+ 396us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 12:13:06.522701 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 12:13:06.522704 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0809 12:13:06.523300 (+ 596us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 12:13:06.523609 (+ 309us) server_negotiation.cc:1035] Waiting for connection context
0809 12:13:06.524052 (+ 443us) server_negotiation.cc:299] Negotiation successful
0809 12:13:06.524466 (+ 414us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":625,"thread_start_us":392,"threads_started":1}
I20260809 12:13:06.532716 14278 raft_consensus.cc:348] T 00000000000000000000000000000000 P 8751c8f15ead4a2a96a78af1bd301cc9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8751c8f15ead4a2a96a78af1bd301cc9" member_type: VOTER last_known_addr { host: "127.0.222.62" port: 37183 } }
I20260809 12:13:06.533216 14278 raft_consensus.cc:374] T 00000000000000000000000000000000 P 8751c8f15ead4a2a96a78af1bd301cc9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260809 12:13:06.533322 14278 raft_consensus.cc:729] T 00000000000000000000000000000000 P 8751c8f15ead4a2a96a78af1bd301cc9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8751c8f15ead4a2a96a78af1bd301cc9, State: Initialized, Role: FOLLOWER
I20260809 12:13:06.533929 14278 consensus_queue.cc:260] T 00000000000000000000000000000000 P 8751c8f15ead4a2a96a78af1bd301cc9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8751c8f15ead4a2a96a78af1bd301cc9" member_type: VOTER last_known_addr { host: "127.0.222.62" port: 37183 } }
I20260809 12:13:06.534250 14278 raft_consensus.cc:388] T 00000000000000000000000000000000 P 8751c8f15ead4a2a96a78af1bd301cc9 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260809 12:13:06.534403 14278 raft_consensus.cc:482] T 00000000000000000000000000000000 P 8751c8f15ead4a2a96a78af1bd301cc9 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260809 12:13:06.534610 14278 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 8751c8f15ead4a2a96a78af1bd301cc9 [term 0 FOLLOWER]: Advancing to term 1
I20260809 12:13:06.537395 14278 raft_consensus.cc:504] T 00000000000000000000000000000000 P 8751c8f15ead4a2a96a78af1bd301cc9 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8751c8f15ead4a2a96a78af1bd301cc9" member_type: VOTER last_known_addr { host: "127.0.222.62" port: 37183 } }
I20260809 12:13:06.537855 14278 leader_election.cc:302] T 00000000000000000000000000000000 P 8751c8f15ead4a2a96a78af1bd301cc9 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 8751c8f15ead4a2a96a78af1bd301cc9; no voters:
I20260809 12:13:06.538383 14278 leader_election.cc:288] T 00000000000000000000000000000000 P 8751c8f15ead4a2a96a78af1bd301cc9 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260809 12:13:06.538662 14283 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 8751c8f15ead4a2a96a78af1bd301cc9 [term 1 FOLLOWER]: Leader election won for term 1
I20260809 12:13:06.539426 14283 raft_consensus.cc:686] T 00000000000000000000000000000000 P 8751c8f15ead4a2a96a78af1bd301cc9 [term 1 LEADER]: Becoming Leader. State: Replica: 8751c8f15ead4a2a96a78af1bd301cc9, State: Running, Role: LEADER
I20260809 12:13:06.539961 14283 consensus_queue.cc:237] T 00000000000000000000000000000000 P 8751c8f15ead4a2a96a78af1bd301cc9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8751c8f15ead4a2a96a78af1bd301cc9" member_type: VOTER last_known_addr { host: "127.0.222.62" port: 37183 } }
I20260809 12:13:06.540555 14278 sys_catalog.cc:564] T 00000000000000000000000000000000 P 8751c8f15ead4a2a96a78af1bd301cc9 [sys.catalog]: configured and running, proceeding with master startup.
I20260809 12:13:06.542837 14285 sys_catalog.cc:455] T 00000000000000000000000000000000 P 8751c8f15ead4a2a96a78af1bd301cc9 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 8751c8f15ead4a2a96a78af1bd301cc9. Latest consensus state: current_term: 1 leader_uuid: "8751c8f15ead4a2a96a78af1bd301cc9" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8751c8f15ead4a2a96a78af1bd301cc9" member_type: VOTER last_known_addr { host: "127.0.222.62" port: 37183 } } }
I20260809 12:13:06.543210 14284 sys_catalog.cc:455] T 00000000000000000000000000000000 P 8751c8f15ead4a2a96a78af1bd301cc9 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "8751c8f15ead4a2a96a78af1bd301cc9" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8751c8f15ead4a2a96a78af1bd301cc9" member_type: VOTER last_known_addr { host: "127.0.222.62" port: 37183 } } }
I20260809 12:13:06.543596 14285 sys_catalog.cc:458] T 00000000000000000000000000000000 P 8751c8f15ead4a2a96a78af1bd301cc9 [sys.catalog]: This master's current role is: LEADER
I20260809 12:13:06.543596 14284 sys_catalog.cc:458] T 00000000000000000000000000000000 P 8751c8f15ead4a2a96a78af1bd301cc9 [sys.catalog]: This master's current role is: LEADER
I20260809 12:13:06.544502 14288 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260809 12:13:06.550346 14288 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260809 12:13:06.558640 14288 catalog_manager.cc:1347] Generated new cluster ID: 07ff07683467491ea6b949512cbd09b9
I20260809 12:13:06.558744 14288 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260809 12:13:06.563083 14278 ranger_client.cc:318] Using new properties file: /tmp/dist-test-taskgH5Pjm/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786277382073938-888-0/minicluster-data/master-0/logs/kudu-ranger-subprocess-log4j2.properties
I20260809 12:13:06.580194 14288 catalog_manager.cc:1370] Generated new certificate authority record
I20260809 12:13:06.581751 14288 catalog_manager.cc:1504] Loading token signing keys...
I20260809 12:13:06.596905 14288 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 8751c8f15ead4a2a96a78af1bd301cc9: Generated new TSK 0
I20260809 12:13:06.598521 14288 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.0.222.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.222.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.222.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskgH5Pjm/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786277382073938-888-0/krb5kdc/kudu_127.0.222.1.keytab.
Entry for principal kudu/127.0.222.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskgH5Pjm/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786277382073938-888-0/krb5kdc/kudu_127.0.222.1.keytab.
I20260809 12:13:07.375083 888 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskgH5Pjm/build/asan/bin/kudu
/tmp/dist-test-taskgH5Pjm/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskgH5Pjm/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786277382073938-888-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskgH5Pjm/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786277382073938-888-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskgH5Pjm/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786277382073938-888-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskgH5Pjm/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786277382073938-888-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.0.222.20:55861/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.222.1:0
--local_ip_for_outbound_sockets=127.0.222.1
--webserver_interface=127.0.222.1
--webserver_port=0
--tserver_master_addrs=127.0.222.62:37183
--keytab_file=/tmp/dist-test-taskgH5Pjm/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786277382073938-888-0/krb5kdc/kudu_127.0.222.1.keytab
--principal=kudu/127.0.222.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.222.20:37035
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskgH5Pjm/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786277382073938-888-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskgH5Pjm/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786277382073938-888-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskgH5Pjm/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786277382073938-888-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Aug 09 12:13:07 dist-test-slave-2w3w krb5kdc[13742](info): AS_REQ (1 etypes {17}) 127.0.0.1: ISSUE: authtime 1786277587, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.222.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
W20260809 12:13:07.856395 14323 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260809 12:13:07.856856 14323 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260809 12:13:07.857045 14323 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260809 12:13:07.868891 14323 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260809 12:13:07.869118 14323 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260809 12:13:07.869333 14323 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.222.1
I20260809 12:13:07.886008 14323 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.222.20:37035
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskgH5Pjm/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786277382073938-888-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskgH5Pjm/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786277382073938-888-0/minicluster-data/ts-0/wal
--ranger_kms_url=127.0.222.20:55861/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskgH5Pjm/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786277382073938-888-0/krb5kdc/kudu_127.0.222.1.keytab
--principal=kudu/127.0.222.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.222.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskgH5Pjm/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786277382073938-888-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.222.1
--webserver_port=0
--tserver_master_addrs=127.0.222.62:37183
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.222.1
--log_dir=/tmp/dist-test-taskgH5Pjm/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786277382073938-888-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 09 Aug 2026 11:06:41 UTC on ccycloud.quasar-bwoogy.root.comops.site
build id 4575991
ASAN enabled
I20260809 12:13:07.887859 14323 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260809 12:13:07.889954 14323 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260809 12:13:07.908041 14334 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 12:13:07.908063 14335 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 12:13:07.909771 14337 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260809 12:13:07.910517 14323 server_base.cc:1034] running on GCE node
I20260809 12:13:07.911309 14323 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260809 12:13:07.912349 14323 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260809 12:13:07.913648 14323 hybrid_clock.cc:648] HybridClock initialized: now 1786277587913587 us; error 57 us; skew 500 ppm
I20260809 12:13:07.914012 14323 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 09 12:13:07 dist-test-slave-2w3w krb5kdc[13742](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786277587, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.222.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260809 12:13:07.917793 14323 init.cc:377] Logged in from keytab as kudu/127.0.222.1@KRBTEST.COM (short username kudu)
I20260809 12:13:07.919792 14323 webserver.cc:460] Webserver started at http://127.0.222.1:37073/ using document root <none> and password file <none>
I20260809 12:13:07.920724 14323 fs_manager.cc:359] Metadata directory not provided
I20260809 12:13:07.920868 14323 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260809 12:13:07.921250 14323 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Aug 09 12:13:07 dist-test-slave-2w3w krb5kdc[13742](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786277587, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.222.1@KRBTEST.COM for HTTP/127.0.222.20@KRBTEST.COM
I20260809 12:13:07.943049 14323 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskgH5Pjm/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786277382073938-888-0/minicluster-data/ts-0/data/instance:
uuid: "0d897ca0b7594fb6952f1d1f90301da6"
format_stamp: "Formatted at 2026-08-09 12:13:07 on dist-test-slave-2w3w"
server_key: "6b00af395e58ea1df1eceba6d00d86db"
server_key_iv: "2291c26a330deec0de2c8eeb5d6c429b"
server_key_version: "kuduclusterkey@0"
I20260809 12:13:07.944257 14323 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskgH5Pjm/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786277382073938-888-0/minicluster-data/ts-0/wal/instance:
uuid: "0d897ca0b7594fb6952f1d1f90301da6"
format_stamp: "Formatted at 2026-08-09 12:13:07 on dist-test-slave-2w3w"
server_key: "6b00af395e58ea1df1eceba6d00d86db"
server_key_iv: "2291c26a330deec0de2c8eeb5d6c429b"
server_key_version: "kuduclusterkey@0"
I20260809 12:13:07.957891 14323 fs_manager.cc:691] Time spent creating directory manager: real 0.013s user 0.005s sys 0.007s
I20260809 12:13:07.986032 14346 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260809 12:13:07.988044 14323 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.002s sys 0.000s
I20260809 12:13:07.988248 14323 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskgH5Pjm/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786277382073938-888-0/minicluster-data/ts-0/data,/tmp/dist-test-taskgH5Pjm/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786277382073938-888-0/minicluster-data/ts-0/wal
uuid: "0d897ca0b7594fb6952f1d1f90301da6"
format_stamp: "Formatted at 2026-08-09 12:13:07 on dist-test-slave-2w3w"
server_key: "6b00af395e58ea1df1eceba6d00d86db"
server_key_iv: "2291c26a330deec0de2c8eeb5d6c429b"
server_key_version: "kuduclusterkey@0"
I20260809 12:13:07.988518 14323 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskgH5Pjm/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786277382073938-888-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskgH5Pjm/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786277382073938-888-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskgH5Pjm/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786277382073938-888-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260809 12:13:08.010481 14323 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260809 12:13:08.016948 14323 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260809 12:13:08.017398 14323 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260809 12:13:08.018877 14323 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260809 12:13:08.027293 14323 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260809 12:13:08.027473 14323 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260809 12:13:08.027680 14323 ts_tablet_manager.cc:610] Registered 0 tablets
I20260809 12:13:08.027792 14323 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260809 12:13:08.120414 14323 rpc_server.cc:307] RPC server started. Bound to: 127.0.222.1:38517
I20260809 12:13:08.120471 14461 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.222.1:38517 every 8 connection(s)
I20260809 12:13:08.122190 14323 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskgH5Pjm/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786277382073938-888-0/minicluster-data/ts-0/data/info.pb
I20260809 12:13:08.128041 888 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskgH5Pjm/build/asan/bin/kudu as pid 14323
I20260809 12:13:08.128270 888 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskgH5Pjm/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786277382073938-888-0/minicluster-data/ts-0/wal/instance
I20260809 12:13:08.129151 888 external_mini_cluster.cc:1428] Setting key 412a85137472c037dbc6c18cfa27acf1
Aug 09 12:13:08 dist-test-slave-2w3w krb5kdc[13742](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786277587, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.222.1@KRBTEST.COM for kudu/127.0.222.62@KRBTEST.COM
I20260809 12:13:08.167340 14465 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 12:13:08.126574 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.222.1:51713 (local address 127.0.222.62:37183)
0809 12:13:08.146422 (+ 19848us) server_negotiation.cc:206] Beginning negotiation
0809 12:13:08.146429 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0809 12:13:08.146454 (+ 25us) server_negotiation.cc:407] Connection header received
0809 12:13:08.146524 (+ 70us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 12:13:08.146530 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 12:13:08.147700 (+ 1170us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 12:13:08.147837 (+ 137us) server_negotiation.cc:226] Negotiated authn=SASL
0809 12:13:08.150443 (+ 2606us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 12:13:08.151628 (+ 1185us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 12:13:08.154190 (+ 2562us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 12:13:08.154914 (+ 724us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 12:13:08.155037 (+ 123us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 12:13:08.159774 (+ 4737us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 12:13:08.159780 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 12:13:08.159785 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0809 12:13:08.159827 (+ 42us) server_negotiation.cc:922] calling sasl_server_start()
0809 12:13:08.163618 (+ 3791us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 12:13:08.164596 (+ 978us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 12:13:08.164601 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 12:13:08.164605 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0809 12:13:08.164765 (+ 160us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 12:13:08.165307 (+ 542us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 12:13:08.165312 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 12:13:08.165315 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0809 12:13:08.165643 (+ 328us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 12:13:08.165761 (+ 118us) server_negotiation.cc:1035] Waiting for connection context
0809 12:13:08.166708 (+ 947us) server_negotiation.cc:299] Negotiation successful
0809 12:13:08.166936 (+ 228us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":19684,"thread_start_us":19350,"threads_started":1}
I20260809 12:13:08.169595 14466 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 12:13:08.127163 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.222.62:37183 (local address 127.0.222.1:51713)
0809 12:13:08.138066 (+ 10903us) negotiation.cc:107] Waiting for socket to connect
0809 12:13:08.138169 (+ 103us) client_negotiation.cc:174] Beginning negotiation
0809 12:13:08.144842 (+ 6673us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 12:13:08.148350 (+ 3508us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 12:13:08.148362 (+ 12us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 12:13:08.148991 (+ 629us) client_negotiation.cc:189] Negotiated authn=SASL
0809 12:13:08.149820 (+ 829us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 12:13:08.149845 (+ 25us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 12:13:08.151813 (+ 1968us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 12:13:08.151819 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 12:13:08.154017 (+ 2198us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 12:13:08.154033 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 12:13:08.155400 (+ 1367us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 12:13:08.155408 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 12:13:08.155697 (+ 289us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 12:13:08.156537 (+ 840us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0809 12:13:08.156571 (+ 34us) client_negotiation.cc:656] calling sasl_client_start()
0809 12:13:08.159495 (+ 2924us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0809 12:13:08.163804 (+ 4309us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0809 12:13:08.163813 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0809 12:13:08.163834 (+ 21us) client_negotiation.cc:760] calling sasl_client_step()
0809 12:13:08.164446 (+ 612us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0809 12:13:08.164898 (+ 452us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0809 12:13:08.164903 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0809 12:13:08.164906 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0809 12:13:08.165164 (+ 258us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0809 12:13:08.165844 (+ 680us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0809 12:13:08.165852 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0809 12:13:08.166291 (+ 439us) client_negotiation.cc:769] Sending connection context
0809 12:13:08.167660 (+ 1369us) client_negotiation.cc:240] Negotiation successful
0809 12:13:08.168251 (+ 591us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":9810,"thread_start_us":11706,"threads_started":1}
I20260809 12:13:08.171550 14462 heartbeater.cc:347] Connected to a master server at 127.0.222.62:37183
I20260809 12:13:08.171905 14462 heartbeater.cc:464] Registering TS with master...
I20260809 12:13:08.172871 14462 heartbeater.cc:511] Master 127.0.222.62:37183 requested a full tablet report, sending...
WARNING: no policy specified for kudu/127.0.222.2@KRBTEST.COM; defaulting to no policy
I20260809 12:13:08.175755 14241 ts_manager.cc:194] Registered new tserver with Master: 0d897ca0b7594fb6952f1d1f90301da6 (127.0.222.1:38517)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.222.2@KRBTEST.COM" created.
I20260809 12:13:08.177877 14241 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.222.1@KRBTEST.COM'} at 127.0.222.1:51713
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.222.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskgH5Pjm/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786277382073938-888-0/krb5kdc/kudu_127.0.222.2.keytab.
Entry for principal kudu/127.0.222.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskgH5Pjm/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786277382073938-888-0/krb5kdc/kudu_127.0.222.2.keytab.
I20260809 12:13:08.224023 888 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskgH5Pjm/build/asan/bin/kudu
/tmp/dist-test-taskgH5Pjm/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskgH5Pjm/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786277382073938-888-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskgH5Pjm/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786277382073938-888-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskgH5Pjm/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786277382073938-888-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskgH5Pjm/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786277382073938-888-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.0.222.20:55861/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.222.2:0
--local_ip_for_outbound_sockets=127.0.222.2
--webserver_interface=127.0.222.2
--webserver_port=0
--tserver_master_addrs=127.0.222.62:37183
--keytab_file=/tmp/dist-test-taskgH5Pjm/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786277382073938-888-0/krb5kdc/kudu_127.0.222.2.keytab
--principal=kudu/127.0.222.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.222.20:37035
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskgH5Pjm/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786277382073938-888-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskgH5Pjm/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786277382073938-888-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskgH5Pjm/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786277382073938-888-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Aug 09 12:13:08 dist-test-slave-2w3w krb5kdc[13742](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786277587, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.222.62@KRBTEST.COM for HTTP/127.0.222.20@KRBTEST.COM
Aug 09 12:13:08 dist-test-slave-2w3w krb5kdc[13742](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786277587, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.222.62@KRBTEST.COM for HTTP/127.0.222.20@KRBTEST.COM
W20260809 12:13:08.727201 14469 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260809 12:13:08.727536 14469 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260809 12:13:08.727708 14469 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260809 12:13:08.737294 14469 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260809 12:13:08.737452 14469 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260809 12:13:08.737644 14469 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.222.2
I20260809 12:13:08.748387 14469 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.222.20:37035
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskgH5Pjm/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786277382073938-888-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskgH5Pjm/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786277382073938-888-0/minicluster-data/ts-1/wal
--ranger_kms_url=127.0.222.20:55861/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskgH5Pjm/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786277382073938-888-0/krb5kdc/kudu_127.0.222.2.keytab
--principal=kudu/127.0.222.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.222.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskgH5Pjm/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786277382073938-888-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.222.2
--webserver_port=0
--tserver_master_addrs=127.0.222.62:37183
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.222.2
--log_dir=/tmp/dist-test-taskgH5Pjm/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786277382073938-888-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 09 Aug 2026 11:06:41 UTC on ccycloud.quasar-bwoogy.root.comops.site
build id 4575991
ASAN enabled
I20260809 12:13:08.749782 14469 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260809 12:13:08.751469 14469 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260809 12:13:08.769039 14476 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 12:13:08.769038 14477 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 12:13:08.776788 14479 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260809 12:13:08.782039 14469 server_base.cc:1034] running on GCE node
I20260809 12:13:08.783018 14469 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260809 12:13:08.790015 14469 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260809 12:13:08.791352 14469 hybrid_clock.cc:648] HybridClock initialized: now 1786277588791231 us; error 122 us; skew 500 ppm
I20260809 12:13:08.791746 14469 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 09 12:13:08 dist-test-slave-2w3w krb5kdc[13742](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786277588, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.222.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260809 12:13:08.795578 14469 init.cc:377] Logged in from keytab as kudu/127.0.222.2@KRBTEST.COM (short username kudu)
I20260809 12:13:08.797622 14469 webserver.cc:460] Webserver started at http://127.0.222.2:34383/ using document root <none> and password file <none>
I20260809 12:13:08.798382 14469 fs_manager.cc:359] Metadata directory not provided
I20260809 12:13:08.798529 14469 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260809 12:13:08.798951 14469 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Aug 09 12:13:08 dist-test-slave-2w3w krb5kdc[13742](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786277588, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.222.2@KRBTEST.COM for HTTP/127.0.222.20@KRBTEST.COM
I20260809 12:13:08.818451 14469 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskgH5Pjm/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786277382073938-888-0/minicluster-data/ts-1/data/instance:
uuid: "b13c52d3d307484390935460b384345d"
format_stamp: "Formatted at 2026-08-09 12:13:08 on dist-test-slave-2w3w"
server_key: "7e29fdcfbd1fb501b2b588115913157d"
server_key_iv: "60cbf025fdf5581953cd0aa0c785e438"
server_key_version: "kuduclusterkey@0"
I20260809 12:13:08.819417 14469 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskgH5Pjm/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786277382073938-888-0/minicluster-data/ts-1/wal/instance:
uuid: "b13c52d3d307484390935460b384345d"
format_stamp: "Formatted at 2026-08-09 12:13:08 on dist-test-slave-2w3w"
server_key: "7e29fdcfbd1fb501b2b588115913157d"
server_key_iv: "60cbf025fdf5581953cd0aa0c785e438"
server_key_version: "kuduclusterkey@0"
I20260809 12:13:08.828835 14469 fs_manager.cc:691] Time spent creating directory manager: real 0.009s user 0.004s sys 0.004s
I20260809 12:13:08.850042 14494 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260809 12:13:08.851686 14469 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.003s sys 0.000s
I20260809 12:13:08.851884 14469 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskgH5Pjm/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786277382073938-888-0/minicluster-data/ts-1/data,/tmp/dist-test-taskgH5Pjm/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786277382073938-888-0/minicluster-data/ts-1/wal
uuid: "b13c52d3d307484390935460b384345d"
format_stamp: "Formatted at 2026-08-09 12:13:08 on dist-test-slave-2w3w"
server_key: "7e29fdcfbd1fb501b2b588115913157d"
server_key_iv: "60cbf025fdf5581953cd0aa0c785e438"
server_key_version: "kuduclusterkey@0"
I20260809 12:13:08.852154 14469 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskgH5Pjm/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786277382073938-888-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskgH5Pjm/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786277382073938-888-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskgH5Pjm/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786277382073938-888-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260809 12:13:08.869092 14469 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260809 12:13:08.875219 14469 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260809 12:13:08.875553 14469 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260809 12:13:08.876731 14469 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260809 12:13:08.878299 14469 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260809 12:13:08.878405 14469 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260809 12:13:08.878543 14469 ts_tablet_manager.cc:610] Registered 0 tablets
I20260809 12:13:08.878621 14469 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260809 12:13:08.934791 14469 rpc_server.cc:307] RPC server started. Bound to: 127.0.222.2:33317
I20260809 12:13:08.934811 14613 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.222.2:33317 every 8 connection(s)
I20260809 12:13:08.936215 14469 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskgH5Pjm/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786277382073938-888-0/minicluster-data/ts-1/data/info.pb
I20260809 12:13:08.941404 888 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskgH5Pjm/build/asan/bin/kudu as pid 14469
I20260809 12:13:08.941625 888 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskgH5Pjm/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786277382073938-888-0/minicluster-data/ts-1/wal/instance
I20260809 12:13:08.942376 888 external_mini_cluster.cc:1428] Setting key 5403d7e597359f2b989fa23b73393f57
Aug 09 12:13:08 dist-test-slave-2w3w krb5kdc[13742](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786277588, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.222.2@KRBTEST.COM for kudu/127.0.222.62@KRBTEST.COM
I20260809 12:13:08.980834 14616 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 12:13:08.939545 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.222.2:53725 (local address 127.0.222.62:37183)
0809 12:13:08.961100 (+ 21555us) server_negotiation.cc:206] Beginning negotiation
0809 12:13:08.961105 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0809 12:13:08.961126 (+ 21us) server_negotiation.cc:407] Connection header received
0809 12:13:08.961191 (+ 65us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 12:13:08.961194 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 12:13:08.961259 (+ 65us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 12:13:08.961408 (+ 149us) server_negotiation.cc:226] Negotiated authn=SASL
0809 12:13:08.963629 (+ 2221us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 12:13:08.964425 (+ 796us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 12:13:08.967318 (+ 2893us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 12:13:08.967731 (+ 413us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 12:13:08.967910 (+ 179us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 12:13:08.972695 (+ 4785us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 12:13:08.972700 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 12:13:08.972704 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0809 12:13:08.972739 (+ 35us) server_negotiation.cc:922] calling sasl_server_start()
0809 12:13:08.976541 (+ 3802us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 12:13:08.977554 (+ 1013us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 12:13:08.977562 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 12:13:08.977569 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0809 12:13:08.977753 (+ 184us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 12:13:08.978345 (+ 592us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 12:13:08.978353 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 12:13:08.978359 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0809 12:13:08.978771 (+ 412us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 12:13:08.978925 (+ 154us) server_negotiation.cc:1035] Waiting for connection context
0809 12:13:08.980034 (+ 1109us) server_negotiation.cc:299] Negotiation successful
0809 12:13:08.980315 (+ 281us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":21430,"thread_start_us":19297,"threads_started":1}
I20260809 12:13:08.983703 14617 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 12:13:08.940389 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.222.62:37183 (local address 127.0.222.2:53725)
0809 12:13:08.955009 (+ 14620us) negotiation.cc:107] Waiting for socket to connect
0809 12:13:08.955124 (+ 115us) client_negotiation.cc:174] Beginning negotiation
0809 12:13:08.958310 (+ 3186us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 12:13:08.961716 (+ 3406us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 12:13:08.961733 (+ 17us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 12:13:08.962454 (+ 721us) client_negotiation.cc:189] Negotiated authn=SASL
0809 12:13:08.963424 (+ 970us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 12:13:08.963456 (+ 32us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 12:13:08.964590 (+ 1134us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 12:13:08.964598 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 12:13:08.966899 (+ 2301us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 12:13:08.966920 (+ 21us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 12:13:08.968009 (+ 1089us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 12:13:08.968017 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 12:13:08.968349 (+ 332us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 12:13:08.969184 (+ 835us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0809 12:13:08.969219 (+ 35us) client_negotiation.cc:656] calling sasl_client_start()
0809 12:13:08.972424 (+ 3205us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0809 12:13:08.976719 (+ 4295us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0809 12:13:08.976731 (+ 12us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0809 12:13:08.976753 (+ 22us) client_negotiation.cc:760] calling sasl_client_step()
0809 12:13:08.977411 (+ 658us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0809 12:13:08.977927 (+ 516us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0809 12:13:08.977935 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0809 12:13:08.977941 (+ 6us) client_negotiation.cc:760] calling sasl_client_step()
0809 12:13:08.978200 (+ 259us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0809 12:13:08.979044 (+ 844us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0809 12:13:08.979055 (+ 11us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0809 12:13:08.979544 (+ 489us) client_negotiation.cc:769] Sending connection context
0809 12:13:08.981388 (+ 1844us) client_negotiation.cc:240] Negotiation successful
0809 12:13:08.982046 (+ 658us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":14229,"thread_start_us":12993,"threads_started":1}
WARNING: no policy specified for kudu/127.0.222.3@KRBTEST.COM; defaulting to no policy
I20260809 12:13:08.985937 14614 heartbeater.cc:347] Connected to a master server at 127.0.222.62:37183
I20260809 12:13:08.986251 14614 heartbeater.cc:464] Registering TS with master...
I20260809 12:13:08.987177 14614 heartbeater.cc:511] Master 127.0.222.62:37183 requested a full tablet report, sending...
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.222.3@KRBTEST.COM" created.
I20260809 12:13:08.988884 14241 ts_manager.cc:194] Registered new tserver with Master: b13c52d3d307484390935460b384345d (127.0.222.2:33317)
I20260809 12:13:08.989707 14241 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.222.2@KRBTEST.COM'} at 127.0.222.2:53725
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.222.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskgH5Pjm/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786277382073938-888-0/krb5kdc/kudu_127.0.222.3.keytab.
Entry for principal kudu/127.0.222.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskgH5Pjm/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786277382073938-888-0/krb5kdc/kudu_127.0.222.3.keytab.
I20260809 12:13:09.024698 888 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskgH5Pjm/build/asan/bin/kudu
/tmp/dist-test-taskgH5Pjm/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskgH5Pjm/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786277382073938-888-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskgH5Pjm/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786277382073938-888-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskgH5Pjm/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786277382073938-888-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskgH5Pjm/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786277382073938-888-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.0.222.20:55861/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.222.3:0
--local_ip_for_outbound_sockets=127.0.222.3
--webserver_interface=127.0.222.3
--webserver_port=0
--tserver_master_addrs=127.0.222.62:37183
--keytab_file=/tmp/dist-test-taskgH5Pjm/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786277382073938-888-0/krb5kdc/kudu_127.0.222.3.keytab
--principal=kudu/127.0.222.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.222.20:37035
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskgH5Pjm/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786277382073938-888-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskgH5Pjm/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786277382073938-888-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskgH5Pjm/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786277382073938-888-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260809 12:13:09.181945 14462 heartbeater.cc:503] Master 127.0.222.62:37183 was elected leader, sending a full tablet report...
W20260809 12:13:09.381830 14620 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260809 12:13:09.382181 14620 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260809 12:13:09.382346 14620 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260809 12:13:09.389864 14620 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260809 12:13:09.390013 14620 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260809 12:13:09.390203 14620 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.222.3
I20260809 12:13:09.400278 14620 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.222.20:37035
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskgH5Pjm/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786277382073938-888-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskgH5Pjm/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786277382073938-888-0/minicluster-data/ts-2/wal
--ranger_kms_url=127.0.222.20:55861/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskgH5Pjm/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786277382073938-888-0/krb5kdc/kudu_127.0.222.3.keytab
--principal=kudu/127.0.222.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.222.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskgH5Pjm/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786277382073938-888-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.222.3
--webserver_port=0
--tserver_master_addrs=127.0.222.62:37183
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.222.3
--log_dir=/tmp/dist-test-taskgH5Pjm/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786277382073938-888-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 09 Aug 2026 11:06:41 UTC on ccycloud.quasar-bwoogy.root.comops.site
build id 4575991
ASAN enabled
I20260809 12:13:09.401691 14620 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260809 12:13:09.403039 14620 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260809 12:13:09.416474 14625 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 12:13:09.416440 14626 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 12:13:09.416531 14628 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260809 12:13:09.417261 14620 server_base.cc:1034] running on GCE node
I20260809 12:13:09.417909 14620 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260809 12:13:09.418866 14620 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260809 12:13:09.420112 14620 hybrid_clock.cc:648] HybridClock initialized: now 1786277589420087 us; error 56 us; skew 500 ppm
I20260809 12:13:09.420408 14620 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 09 12:13:09 dist-test-slave-2w3w krb5kdc[13742](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786277589, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.222.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260809 12:13:09.423687 14620 init.cc:377] Logged in from keytab as kudu/127.0.222.3@KRBTEST.COM (short username kudu)
I20260809 12:13:09.425163 14620 webserver.cc:460] Webserver started at http://127.0.222.3:42365/ using document root <none> and password file <none>
I20260809 12:13:09.425868 14620 fs_manager.cc:359] Metadata directory not provided
I20260809 12:13:09.426017 14620 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260809 12:13:09.426328 14620 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Aug 09 12:13:09 dist-test-slave-2w3w krb5kdc[13742](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786277589, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.222.3@KRBTEST.COM for HTTP/127.0.222.20@KRBTEST.COM
I20260809 12:13:09.441254 14620 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskgH5Pjm/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786277382073938-888-0/minicluster-data/ts-2/data/instance:
uuid: "22bd9601d42e40a880eec51934dfeb23"
format_stamp: "Formatted at 2026-08-09 12:13:09 on dist-test-slave-2w3w"
server_key: "c6d89da093626487e1d1aba1775aee2b"
server_key_iv: "506feaf4020590927820e9aa2a2e738f"
server_key_version: "kuduclusterkey@0"
I20260809 12:13:09.442224 14620 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskgH5Pjm/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786277382073938-888-0/minicluster-data/ts-2/wal/instance:
uuid: "22bd9601d42e40a880eec51934dfeb23"
format_stamp: "Formatted at 2026-08-09 12:13:09 on dist-test-slave-2w3w"
server_key: "c6d89da093626487e1d1aba1775aee2b"
server_key_iv: "506feaf4020590927820e9aa2a2e738f"
server_key_version: "kuduclusterkey@0"
I20260809 12:13:09.448354 14620 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.005s sys 0.001s
I20260809 12:13:09.463719 14636 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260809 12:13:09.465166 14620 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260809 12:13:09.465369 14620 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskgH5Pjm/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786277382073938-888-0/minicluster-data/ts-2/data,/tmp/dist-test-taskgH5Pjm/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786277382073938-888-0/minicluster-data/ts-2/wal
uuid: "22bd9601d42e40a880eec51934dfeb23"
format_stamp: "Formatted at 2026-08-09 12:13:09 on dist-test-slave-2w3w"
server_key: "c6d89da093626487e1d1aba1775aee2b"
server_key_iv: "506feaf4020590927820e9aa2a2e738f"
server_key_version: "kuduclusterkey@0"
I20260809 12:13:09.465621 14620 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskgH5Pjm/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786277382073938-888-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskgH5Pjm/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786277382073938-888-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskgH5Pjm/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786277382073938-888-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260809 12:13:09.479171 14620 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260809 12:13:09.483530 14620 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260809 12:13:09.483865 14620 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260809 12:13:09.484845 14620 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260809 12:13:09.486505 14620 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260809 12:13:09.486581 14620 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260809 12:13:09.486670 14620 ts_tablet_manager.cc:610] Registered 0 tablets
I20260809 12:13:09.486714 14620 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260809 12:13:09.525744 14620 rpc_server.cc:307] RPC server started. Bound to: 127.0.222.3:43891
I20260809 12:13:09.525787 14753 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.222.3:43891 every 8 connection(s)
I20260809 12:13:09.527112 14620 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskgH5Pjm/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786277382073938-888-0/minicluster-data/ts-2/data/info.pb
I20260809 12:13:09.532374 888 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskgH5Pjm/build/asan/bin/kudu as pid 14620
I20260809 12:13:09.532574 888 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskgH5Pjm/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786277382073938-888-0/minicluster-data/ts-2/wal/instance
I20260809 12:13:09.533141 888 external_mini_cluster.cc:1428] Setting key ecf2b78ab9484eadcbfb818b5d70c401
Aug 09 12:13:09 dist-test-slave-2w3w krb5kdc[13742](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786277589, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.222.3@KRBTEST.COM for kudu/127.0.222.62@KRBTEST.COM
I20260809 12:13:09.547885 14756 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 12:13:09.529733 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.222.3:55419 (local address 127.0.222.62:37183)
0809 12:13:09.530195 (+ 462us) server_negotiation.cc:206] Beginning negotiation
0809 12:13:09.530200 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0809 12:13:09.531179 (+ 979us) server_negotiation.cc:407] Connection header received
0809 12:13:09.533055 (+ 1876us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 12:13:09.533062 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 12:13:09.533166 (+ 104us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 12:13:09.533313 (+ 147us) server_negotiation.cc:226] Negotiated authn=SASL
0809 12:13:09.534989 (+ 1676us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 12:13:09.535925 (+ 936us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 12:13:09.537657 (+ 1732us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 12:13:09.538080 (+ 423us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 12:13:09.538178 (+ 98us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 12:13:09.541915 (+ 3737us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 12:13:09.541920 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 12:13:09.541923 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0809 12:13:09.541989 (+ 66us) server_negotiation.cc:922] calling sasl_server_start()
0809 12:13:09.544611 (+ 2622us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 12:13:09.545370 (+ 759us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 12:13:09.545377 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 12:13:09.545380 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0809 12:13:09.545523 (+ 143us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 12:13:09.545985 (+ 462us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 12:13:09.545993 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 12:13:09.545997 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0809 12:13:09.546276 (+ 279us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 12:13:09.546452 (+ 176us) server_negotiation.cc:1035] Waiting for connection context
0809 12:13:09.547195 (+ 743us) server_negotiation.cc:299] Negotiation successful
0809 12:13:09.547424 (+ 229us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":350,"thread_start_us":213,"threads_started":1}
I20260809 12:13:09.548964 14757 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 12:13:09.530268 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.222.62:37183 (local address 127.0.222.3:55419)
0809 12:13:09.531011 (+ 743us) negotiation.cc:107] Waiting for socket to connect
0809 12:13:09.531058 (+ 47us) client_negotiation.cc:174] Beginning negotiation
0809 12:13:09.532775 (+ 1717us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 12:13:09.533514 (+ 739us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 12:13:09.533522 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 12:13:09.534100 (+ 578us) client_negotiation.cc:189] Negotiated authn=SASL
0809 12:13:09.534797 (+ 697us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 12:13:09.534818 (+ 21us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 12:13:09.536082 (+ 1264us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 12:13:09.536088 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 12:13:09.537526 (+ 1438us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 12:13:09.537538 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 12:13:09.538323 (+ 785us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 12:13:09.538329 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 12:13:09.538543 (+ 214us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 12:13:09.539352 (+ 809us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0809 12:13:09.539375 (+ 23us) client_negotiation.cc:656] calling sasl_client_start()
0809 12:13:09.541703 (+ 2328us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0809 12:13:09.544756 (+ 3053us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0809 12:13:09.544765 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0809 12:13:09.544783 (+ 18us) client_negotiation.cc:760] calling sasl_client_step()
0809 12:13:09.545243 (+ 460us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0809 12:13:09.545642 (+ 399us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0809 12:13:09.545647 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0809 12:13:09.545651 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0809 12:13:09.545867 (+ 216us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0809 12:13:09.546413 (+ 546us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0809 12:13:09.546421 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0809 12:13:09.546786 (+ 365us) client_negotiation.cc:769] Sending connection context
0809 12:13:09.547120 (+ 334us) client_negotiation.cc:240] Negotiation successful
0809 12:13:09.547671 (+ 551us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":478,"thread_start_us":281,"threads_started":1}
I20260809 12:13:09.550324 14754 heartbeater.cc:347] Connected to a master server at 127.0.222.62:37183
I20260809 12:13:09.550592 14754 heartbeater.cc:464] Registering TS with master...
I20260809 12:13:09.551339 14754 heartbeater.cc:511] Master 127.0.222.62:37183 requested a full tablet report, sending...
I20260809 12:13:09.552719 14241 ts_manager.cc:194] Registered new tserver with Master: 22bd9601d42e40a880eec51934dfeb23 (127.0.222.3:43891)
I20260809 12:13:09.553695 14241 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.222.3@KRBTEST.COM'} at 127.0.222.3:55419
I20260809 12:13:09.561295 888 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260809 12:13:09.563217 888 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskgH5Pjm/build/asan/bin/kudu pid 14323
Aug 09 12:13:09 dist-test-slave-2w3w krb5kdc[13742](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786277585, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.222.1@KRBTEST.COM
I20260809 12:13:09.580736 14759 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 12:13:09.564150 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59342 (local address 127.0.222.1:38517)
0809 12:13:09.564786 (+ 636us) server_negotiation.cc:206] Beginning negotiation
0809 12:13:09.564793 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0809 12:13:09.564818 (+ 25us) server_negotiation.cc:407] Connection header received
0809 12:13:09.564881 (+ 63us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 12:13:09.564887 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 12:13:09.565081 (+ 194us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 12:13:09.565221 (+ 140us) server_negotiation.cc:226] Negotiated authn=SASL
0809 12:13:09.566621 (+ 1400us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 12:13:09.567755 (+ 1134us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 12:13:09.569283 (+ 1528us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 12:13:09.569945 (+ 662us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 12:13:09.570048 (+ 103us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 12:13:09.574628 (+ 4580us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 12:13:09.574640 (+ 12us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 12:13:09.574653 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0809 12:13:09.574687 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0809 12:13:09.577781 (+ 3094us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 12:13:09.578506 (+ 725us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 12:13:09.578514 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 12:13:09.578516 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0809 12:13:09.578662 (+ 146us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 12:13:09.579088 (+ 426us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 12:13:09.579093 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 12:13:09.579098 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0809 12:13:09.579436 (+ 338us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 12:13:09.579777 (+ 341us) server_negotiation.cc:1035] Waiting for connection context
0809 12:13:09.580093 (+ 316us) server_negotiation.cc:299] Negotiation successful
0809 12:13:09.580365 (+ 272us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":485,"thread_start_us":330,"threads_started":1}
I20260809 12:13:09.582072 14457 generic_service.cc:196] Checking for leaks (request via RPC)
I20260809 12:13:09.665633 888 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskgH5Pjm/build/asan/bin/kudu with pid 14323
I20260809 12:13:09.686087 888 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskgH5Pjm/build/asan/bin/kudu pid 14469
Aug 09 12:13:09 dist-test-slave-2w3w krb5kdc[13742](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786277585, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.222.2@KRBTEST.COM
I20260809 12:13:09.707471 14761 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 12:13:09.687016 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40454 (local address 127.0.222.2:33317)
0809 12:13:09.687594 (+ 578us) server_negotiation.cc:206] Beginning negotiation
0809 12:13:09.687600 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0809 12:13:09.687628 (+ 28us) server_negotiation.cc:407] Connection header received
0809 12:13:09.687699 (+ 71us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 12:13:09.687705 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 12:13:09.687977 (+ 272us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 12:13:09.688144 (+ 167us) server_negotiation.cc:226] Negotiated authn=SASL
0809 12:13:09.689349 (+ 1205us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 12:13:09.690335 (+ 986us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 12:13:09.691742 (+ 1407us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 12:13:09.692321 (+ 579us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 12:13:09.692429 (+ 108us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 12:13:09.701531 (+ 9102us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 12:13:09.701547 (+ 16us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 12:13:09.701563 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0809 12:13:09.701603 (+ 40us) server_negotiation.cc:922] calling sasl_server_start()
0809 12:13:09.704718 (+ 3115us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 12:13:09.705557 (+ 839us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 12:13:09.705565 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 12:13:09.705567 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0809 12:13:09.705712 (+ 145us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 12:13:09.706042 (+ 330us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 12:13:09.706046 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 12:13:09.706047 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0809 12:13:09.706411 (+ 364us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 12:13:09.706644 (+ 233us) server_negotiation.cc:1035] Waiting for connection context
0809 12:13:09.706884 (+ 240us) server_negotiation.cc:299] Negotiation successful
0809 12:13:09.707150 (+ 266us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":422,"thread_start_us":308,"threads_started":1}
I20260809 12:13:09.708777 14609 generic_service.cc:196] Checking for leaks (request via RPC)
I20260809 12:13:09.793040 888 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskgH5Pjm/build/asan/bin/kudu with pid 14469
I20260809 12:13:09.812592 888 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskgH5Pjm/build/asan/bin/kudu pid 14620
Aug 09 12:13:09 dist-test-slave-2w3w krb5kdc[13742](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786277585, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.222.3@KRBTEST.COM
I20260809 12:13:09.828816 14763 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 12:13:09.813809 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35494 (local address 127.0.222.3:43891)
0809 12:13:09.814511 (+ 702us) server_negotiation.cc:206] Beginning negotiation
0809 12:13:09.814518 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0809 12:13:09.814546 (+ 28us) server_negotiation.cc:407] Connection header received
0809 12:13:09.814608 (+ 62us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 12:13:09.814614 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 12:13:09.814835 (+ 221us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 12:13:09.815039 (+ 204us) server_negotiation.cc:226] Negotiated authn=SASL
0809 12:13:09.816268 (+ 1229us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 12:13:09.817274 (+ 1006us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 12:13:09.818440 (+ 1166us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 12:13:09.819059 (+ 619us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 12:13:09.819155 (+ 96us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 12:13:09.823373 (+ 4218us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 12:13:09.823384 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 12:13:09.823399 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0809 12:13:09.823435 (+ 36us) server_negotiation.cc:922] calling sasl_server_start()
0809 12:13:09.826189 (+ 2754us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 12:13:09.826808 (+ 619us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 12:13:09.826814 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 12:13:09.826817 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0809 12:13:09.826946 (+ 129us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 12:13:09.827366 (+ 420us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 12:13:09.827369 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 12:13:09.827371 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0809 12:13:09.827706 (+ 335us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 12:13:09.827962 (+ 256us) server_negotiation.cc:1035] Waiting for connection context
0809 12:13:09.828209 (+ 247us) server_negotiation.cc:299] Negotiation successful
0809 12:13:09.828410 (+ 201us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":550,"thread_start_us":386,"threads_started":1}
I20260809 12:13:09.830098 14749 generic_service.cc:196] Checking for leaks (request via RPC)
I20260809 12:13:09.916312 888 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskgH5Pjm/build/asan/bin/kudu with pid 14620
I20260809 12:13:09.934427 888 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskgH5Pjm/build/asan/bin/kudu pid 14208
I20260809 12:13:09.935734 14273 generic_service.cc:196] Checking for leaks (request via RPC)
I20260809 12:13:10.043962 888 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskgH5Pjm/build/asan/bin/kudu with pid 14208
I20260809 12:13:10.069294 888 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskgH5Pjm/build/asan/bin/kudu
/tmp/dist-test-taskgH5Pjm/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskgH5Pjm/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786277382073938-888-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskgH5Pjm/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786277382073938-888-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskgH5Pjm/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786277382073938-888-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskgH5Pjm/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786277382073938-888-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.222.62:37183
--webserver_interface=127.0.222.62
--webserver_port=45661
--keytab_file=/tmp/dist-test-taskgH5Pjm/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786277382073938-888-0/krb5kdc/kudu_127.0.222.62.keytab
--principal=kudu/127.0.222.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.222.20:37035
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.222.62:37183
--ranger_config_path=/tmp/dist-test-taskgH5Pjm/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786277382073938-888-0/minicluster-data/ranger-client
--trusted_user_acl=test-admin
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.0.222.20:55861/kms
--rpc_trace_negotiation
--ranger_kms_url=invalid.host:1234/kms,127.0.222.20:55861/kms with env {KRB5CCNAME=/tmp/dist-test-taskgH5Pjm/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786277382073938-888-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskgH5Pjm/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786277382073938-888-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskgH5Pjm/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786277382073938-888-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260809 12:13:10.402108 14766 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260809 12:13:10.402415 14766 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260809 12:13:10.402509 14766 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260809 12:13:10.409878 14766 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260809 12:13:10.410015 14766 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260809 12:13:10.410063 14766 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260809 12:13:10.410120 14766 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260809 12:13:10.410166 14766 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260809 12:13:10.420547 14766 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.222.20:37035
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskgH5Pjm/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786277382073938-888-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskgH5Pjm/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786277382073938-888-0/minicluster-data/master-0/wal
--ranger_kms_url=invalid.host:1234/kms,127.0.222.20:55861/kms
--trusted_user_acl=<redacted>
--ipki_ca_key_size=768
--master_addresses=127.0.222.62:37183
--ranger_config_path=/tmp/dist-test-taskgH5Pjm/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786277382073938-888-0/minicluster-data/ranger-client
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskgH5Pjm/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786277382073938-888-0/krb5kdc/kudu_127.0.222.62.keytab
--principal=kudu/127.0.222.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.222.62:37183
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskgH5Pjm/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786277382073938-888-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.222.62
--webserver_port=45661
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskgH5Pjm/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786277382073938-888-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 09 Aug 2026 11:06:41 UTC on ccycloud.quasar-bwoogy.root.comops.site
build id 4575991
ASAN enabled
I20260809 12:13:10.421815 14766 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260809 12:13:10.423125 14766 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260809 12:13:10.435175 14771 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 12:13:10.435212 14775 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 12:13:10.435249 14772 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260809 12:13:10.436082 14766 server_base.cc:1034] running on GCE node
I20260809 12:13:10.436905 14766 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260809 12:13:10.438392 14766 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260809 12:13:10.439589 14766 hybrid_clock.cc:648] HybridClock initialized: now 1786277590439525 us; error 59 us; skew 500 ppm
I20260809 12:13:10.439978 14766 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 09 12:13:10 dist-test-slave-2w3w krb5kdc[13742](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786277590, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.222.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260809 12:13:10.443229 14766 init.cc:377] Logged in from keytab as kudu/127.0.222.62@KRBTEST.COM (short username kudu)
I20260809 12:13:10.444768 14766 webserver.cc:460] Webserver started at http://127.0.222.62:45661/ using document root <none> and password file <none>
I20260809 12:13:10.445456 14766 fs_manager.cc:359] Metadata directory not provided
I20260809 12:13:10.445603 14766 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
Aug 09 12:13:10 dist-test-slave-2w3w krb5kdc[13742](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786277590, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.222.62@KRBTEST.COM for HTTP/127.0.222.20@KRBTEST.COM
I20260809 12:13:10.525383 14766 fs_manager.cc:709] Time spent opening directory manager: real 0.004s user 0.001s sys 0.004s
I20260809 12:13:10.528687 14783 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260809 12:13:10.530141 14766 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260809 12:13:10.530368 14766 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskgH5Pjm/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786277382073938-888-0/minicluster-data/master-0/data,/tmp/dist-test-taskgH5Pjm/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786277382073938-888-0/minicluster-data/master-0/wal
uuid: "8751c8f15ead4a2a96a78af1bd301cc9"
format_stamp: "Formatted at 2026-08-09 12:13:06 on dist-test-slave-2w3w"
server_key: "55d2a29dc2c9ba6877af46e2fda3af0c"
server_key_iv: "27c59f6cb2df850e0fdf963d02600503"
server_key_version: "kuduclusterkey@0"
I20260809 12:13:10.531397 14766 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskgH5Pjm/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786277382073938-888-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskgH5Pjm/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786277382073938-888-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskgH5Pjm/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786277382073938-888-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260809 12:13:10.564443 14766 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260809 12:13:10.568400 14766 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260809 12:13:10.568706 14766 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260809 12:13:10.587815 14766 rpc_server.cc:307] RPC server started. Bound to: 127.0.222.62:37183
I20260809 12:13:10.587844 14839 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.222.62:37183 every 8 connection(s)
I20260809 12:13:10.589396 14766 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskgH5Pjm/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786277382073938-888-0/minicluster-data/master-0/data/info.pb
I20260809 12:13:10.597080 14840 sys_catalog.cc:263] Verifying existing consensus state
I20260809 12:13:10.597918 888 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskgH5Pjm/build/asan/bin/kudu as pid 14766
I20260809 12:13:10.598790 888 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskgH5Pjm/build/asan/bin/kudu
/tmp/dist-test-taskgH5Pjm/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskgH5Pjm/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786277382073938-888-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskgH5Pjm/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786277382073938-888-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskgH5Pjm/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786277382073938-888-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskgH5Pjm/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786277382073938-888-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.0.222.20:55861/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.222.1:38517
--local_ip_for_outbound_sockets=127.0.222.1
--tserver_master_addrs=127.0.222.62:37183
--webserver_port=37073
--webserver_interface=127.0.222.1
--keytab_file=/tmp/dist-test-taskgH5Pjm/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786277382073938-888-0/krb5kdc/kudu_127.0.222.1.keytab
--principal=kudu/127.0.222.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.222.20:37035
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--ranger_kms_url=invalid.host:1234/kms,127.0.222.20:55861/kms with env {KRB5CCNAME=/tmp/dist-test-taskgH5Pjm/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786277382073938-888-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskgH5Pjm/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786277382073938-888-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskgH5Pjm/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786277382073938-888-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260809 12:13:10.600917 14840 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 8751c8f15ead4a2a96a78af1bd301cc9: Bootstrap starting.
I20260809 12:13:10.625937 14840 log.cc:824] T 00000000000000000000000000000000 P 8751c8f15ead4a2a96a78af1bd301cc9: Log is configured to *not* fsync() on all Append() calls
I20260809 12:13:10.638258 14840 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 8751c8f15ead4a2a96a78af1bd301cc9: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=3 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260809 12:13:10.638947 14840 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 8751c8f15ead4a2a96a78af1bd301cc9: Bootstrap complete.
I20260809 12:13:10.652678 14840 raft_consensus.cc:348] T 00000000000000000000000000000000 P 8751c8f15ead4a2a96a78af1bd301cc9 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8751c8f15ead4a2a96a78af1bd301cc9" member_type: VOTER last_known_addr { host: "127.0.222.62" port: 37183 } }
I20260809 12:13:10.653914 14840 raft_consensus.cc:729] T 00000000000000000000000000000000 P 8751c8f15ead4a2a96a78af1bd301cc9 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8751c8f15ead4a2a96a78af1bd301cc9, State: Initialized, Role: FOLLOWER
I20260809 12:13:10.654619 14840 consensus_queue.cc:260] T 00000000000000000000000000000000 P 8751c8f15ead4a2a96a78af1bd301cc9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8751c8f15ead4a2a96a78af1bd301cc9" member_type: VOTER last_known_addr { host: "127.0.222.62" port: 37183 } }
I20260809 12:13:10.654933 14840 raft_consensus.cc:388] T 00000000000000000000000000000000 P 8751c8f15ead4a2a96a78af1bd301cc9 [term 1 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260809 12:13:10.655079 14840 raft_consensus.cc:482] T 00000000000000000000000000000000 P 8751c8f15ead4a2a96a78af1bd301cc9 [term 1 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260809 12:13:10.655265 14840 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 8751c8f15ead4a2a96a78af1bd301cc9 [term 1 FOLLOWER]: Advancing to term 2
I20260809 12:13:10.660610 14840 raft_consensus.cc:504] T 00000000000000000000000000000000 P 8751c8f15ead4a2a96a78af1bd301cc9 [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8751c8f15ead4a2a96a78af1bd301cc9" member_type: VOTER last_known_addr { host: "127.0.222.62" port: 37183 } }
I20260809 12:13:10.661266 14840 leader_election.cc:302] T 00000000000000000000000000000000 P 8751c8f15ead4a2a96a78af1bd301cc9 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 8751c8f15ead4a2a96a78af1bd301cc9; no voters:
I20260809 12:13:10.662127 14840 leader_election.cc:288] T 00000000000000000000000000000000 P 8751c8f15ead4a2a96a78af1bd301cc9 [CANDIDATE]: Term 2 election: Requested vote from peers
I20260809 12:13:10.662266 14844 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 8751c8f15ead4a2a96a78af1bd301cc9 [term 2 FOLLOWER]: Leader election won for term 2
I20260809 12:13:10.663569 14844 raft_consensus.cc:686] T 00000000000000000000000000000000 P 8751c8f15ead4a2a96a78af1bd301cc9 [term 2 LEADER]: Becoming Leader. State: Replica: 8751c8f15ead4a2a96a78af1bd301cc9, State: Running, Role: LEADER
I20260809 12:13:10.664144 14844 consensus_queue.cc:237] T 00000000000000000000000000000000 P 8751c8f15ead4a2a96a78af1bd301cc9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 2, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8751c8f15ead4a2a96a78af1bd301cc9" member_type: VOTER last_known_addr { host: "127.0.222.62" port: 37183 } }
I20260809 12:13:10.664476 14840 sys_catalog.cc:564] T 00000000000000000000000000000000 P 8751c8f15ead4a2a96a78af1bd301cc9 [sys.catalog]: configured and running, proceeding with master startup.
I20260809 12:13:10.666522 14846 sys_catalog.cc:455] T 00000000000000000000000000000000 P 8751c8f15ead4a2a96a78af1bd301cc9 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 8751c8f15ead4a2a96a78af1bd301cc9. Latest consensus state: current_term: 2 leader_uuid: "8751c8f15ead4a2a96a78af1bd301cc9" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8751c8f15ead4a2a96a78af1bd301cc9" member_type: VOTER last_known_addr { host: "127.0.222.62" port: 37183 } } }
I20260809 12:13:10.666927 14846 sys_catalog.cc:458] T 00000000000000000000000000000000 P 8751c8f15ead4a2a96a78af1bd301cc9 [sys.catalog]: This master's current role is: LEADER
I20260809 12:13:10.668184 14845 sys_catalog.cc:455] T 00000000000000000000000000000000 P 8751c8f15ead4a2a96a78af1bd301cc9 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 2 leader_uuid: "8751c8f15ead4a2a96a78af1bd301cc9" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8751c8f15ead4a2a96a78af1bd301cc9" member_type: VOTER last_known_addr { host: "127.0.222.62" port: 37183 } } }
I20260809 12:13:10.668718 14845 sys_catalog.cc:458] T 00000000000000000000000000000000 P 8751c8f15ead4a2a96a78af1bd301cc9 [sys.catalog]: This master's current role is: LEADER
I20260809 12:13:10.675738 14850 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260809 12:13:10.684371 14850 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260809 12:13:10.686223 14850 catalog_manager.cc:1259] Loaded cluster ID: 07ff07683467491ea6b949512cbd09b9
I20260809 12:13:10.686370 14850 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260809 12:13:10.692019 14850 catalog_manager.cc:1504] Loading token signing keys...
I20260809 12:13:10.693969 14850 catalog_manager.cc:5879] T 00000000000000000000000000000000 P 8751c8f15ead4a2a96a78af1bd301cc9: Loaded TSK: 0
I20260809 12:13:10.695963 14850 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260809 12:13:10.701378 14840 ranger_client.cc:318] Using overwritten properties file: /tmp/dist-test-taskgH5Pjm/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786277382073938-888-0/minicluster-data/master-0/logs/kudu-ranger-subprocess-log4j2.properties
W20260809 12:13:11.091231 14842 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260809 12:13:11.091648 14842 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260809 12:13:11.091822 14842 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260809 12:13:11.103792 14842 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260809 12:13:11.103955 14842 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260809 12:13:11.104148 14842 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.222.1
I20260809 12:13:11.120529 14842 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.222.20:37035
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskgH5Pjm/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786277382073938-888-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskgH5Pjm/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786277382073938-888-0/minicluster-data/ts-0/wal
--ranger_kms_url=invalid.host:1234/kms,127.0.222.20:55861/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskgH5Pjm/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786277382073938-888-0/krb5kdc/kudu_127.0.222.1.keytab
--principal=kudu/127.0.222.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.222.1:38517
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskgH5Pjm/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786277382073938-888-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.222.1
--webserver_port=37073
--tserver_master_addrs=127.0.222.62:37183
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.222.1
--log_dir=/tmp/dist-test-taskgH5Pjm/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786277382073938-888-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 09 Aug 2026 11:06:41 UTC on ccycloud.quasar-bwoogy.root.comops.site
build id 4575991
ASAN enabled
I20260809 12:13:11.122416 14842 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260809 12:13:11.124341 14842 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260809 12:13:11.160481 14878 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 12:13:11.160533 14879 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 12:13:11.160899 14881 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260809 12:13:11.169261 14842 server_base.cc:1034] running on GCE node
I20260809 12:13:11.170192 14842 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260809 12:13:11.181737 14842 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260809 12:13:11.183071 14842 hybrid_clock.cc:648] HybridClock initialized: now 1786277591182980 us; error 87 us; skew 500 ppm
I20260809 12:13:11.183552 14842 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 09 12:13:11 dist-test-slave-2w3w krb5kdc[13742](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786277591, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.222.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260809 12:13:11.189585 14842 init.cc:377] Logged in from keytab as kudu/127.0.222.1@KRBTEST.COM (short username kudu)
I20260809 12:13:11.200783 14842 webserver.cc:460] Webserver started at http://127.0.222.1:37073/ using document root <none> and password file <none>
I20260809 12:13:11.201606 14842 fs_manager.cc:359] Metadata directory not provided
I20260809 12:13:11.201789 14842 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
Aug 09 12:13:11 dist-test-slave-2w3w krb5kdc[13742](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786277591, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.222.1@KRBTEST.COM for HTTP/127.0.222.20@KRBTEST.COM
I20260809 12:13:11.240823 14842 fs_manager.cc:709] Time spent opening directory manager: real 0.013s user 0.006s sys 0.005s
I20260809 12:13:11.259745 14891 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260809 12:13:11.262007 14842 fs_manager.cc:725] Time spent opening block manager: real 0.019s user 0.007s sys 0.011s
I20260809 12:13:11.262226 14842 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskgH5Pjm/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786277382073938-888-0/minicluster-data/ts-0/data,/tmp/dist-test-taskgH5Pjm/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786277382073938-888-0/minicluster-data/ts-0/wal
uuid: "0d897ca0b7594fb6952f1d1f90301da6"
format_stamp: "Formatted at 2026-08-09 12:13:07 on dist-test-slave-2w3w"
server_key: "6b00af395e58ea1df1eceba6d00d86db"
server_key_iv: "2291c26a330deec0de2c8eeb5d6c429b"
server_key_version: "kuduclusterkey@0"
I20260809 12:13:11.263782 14842 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskgH5Pjm/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786277382073938-888-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskgH5Pjm/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786277382073938-888-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskgH5Pjm/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786277382073938-888-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260809 12:13:11.321146 14842 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260809 12:13:11.325817 14842 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260809 12:13:11.326122 14842 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260809 12:13:11.327276 14842 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260809 12:13:11.328790 14842 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260809 12:13:11.328964 14842 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260809 12:13:11.329093 14842 ts_tablet_manager.cc:610] Registered 0 tablets
I20260809 12:13:11.329180 14842 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260809 12:13:11.414253 14842 rpc_server.cc:307] RPC server started. Bound to: 127.0.222.1:38517
I20260809 12:13:11.414305 15012 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.222.1:38517 every 8 connection(s)
I20260809 12:13:11.415992 14842 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskgH5Pjm/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786277382073938-888-0/minicluster-data/ts-0/data/info.pb
I20260809 12:13:11.423305 888 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskgH5Pjm/build/asan/bin/kudu as pid 14842
I20260809 12:13:11.424176 888 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskgH5Pjm/build/asan/bin/kudu
/tmp/dist-test-taskgH5Pjm/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskgH5Pjm/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786277382073938-888-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskgH5Pjm/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786277382073938-888-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskgH5Pjm/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786277382073938-888-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskgH5Pjm/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786277382073938-888-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.0.222.20:55861/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.222.2:33317
--local_ip_for_outbound_sockets=127.0.222.2
--tserver_master_addrs=127.0.222.62:37183
--webserver_port=34383
--webserver_interface=127.0.222.2
--keytab_file=/tmp/dist-test-taskgH5Pjm/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786277382073938-888-0/krb5kdc/kudu_127.0.222.2.keytab
--principal=kudu/127.0.222.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.222.20:37035
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--ranger_kms_url=invalid.host:1234/kms,127.0.222.20:55861/kms with env {KRB5CCNAME=/tmp/dist-test-taskgH5Pjm/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786277382073938-888-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskgH5Pjm/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786277382073938-888-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskgH5Pjm/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786277382073938-888-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Aug 09 12:13:11 dist-test-slave-2w3w krb5kdc[13742](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786277591, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.222.1@KRBTEST.COM for kudu/127.0.222.62@KRBTEST.COM
I20260809 12:13:11.449893 15016 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 12:13:11.419495 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.222.62:37183 (local address 127.0.222.1:58103)
0809 12:13:11.420535 (+ 1040us) negotiation.cc:107] Waiting for socket to connect
0809 12:13:11.420572 (+ 37us) client_negotiation.cc:174] Beginning negotiation
0809 12:13:11.422511 (+ 1939us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 12:13:11.424764 (+ 2253us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 12:13:11.424775 (+ 11us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 12:13:11.425254 (+ 479us) client_negotiation.cc:189] Negotiated authn=SASL
0809 12:13:11.425829 (+ 575us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 12:13:11.425849 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 12:13:11.427546 (+ 1697us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 12:13:11.427552 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 12:13:11.428937 (+ 1385us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 12:13:11.428954 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 12:13:11.430806 (+ 1852us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 12:13:11.430814 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 12:13:11.431137 (+ 323us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 12:13:11.435785 (+ 4648us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0809 12:13:11.435823 (+ 38us) client_negotiation.cc:656] calling sasl_client_start()
0809 12:13:11.438611 (+ 2788us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0809 12:13:11.444188 (+ 5577us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0809 12:13:11.444200 (+ 12us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0809 12:13:11.444219 (+ 19us) client_negotiation.cc:760] calling sasl_client_step()
0809 12:13:11.444890 (+ 671us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0809 12:13:11.445533 (+ 643us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0809 12:13:11.445538 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0809 12:13:11.445542 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0809 12:13:11.445822 (+ 280us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0809 12:13:11.446925 (+ 1103us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0809 12:13:11.446937 (+ 12us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0809 12:13:11.447396 (+ 459us) client_negotiation.cc:769] Sending connection context
0809 12:13:11.447769 (+ 373us) client_negotiation.cc:240] Negotiation successful
0809 12:13:11.448435 (+ 666us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":790,"thread_start_us":587,"threads_started":1}
I20260809 12:13:11.449890 15015 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 12:13:11.419295 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.222.1:58103 (local address 127.0.222.62:37183)
0809 12:13:11.420041 (+ 746us) server_negotiation.cc:206] Beginning negotiation
0809 12:13:11.420063 (+ 22us) server_negotiation.cc:399] Waiting for connection header
0809 12:13:11.420743 (+ 680us) server_negotiation.cc:407] Connection header received
0809 12:13:11.423274 (+ 2531us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 12:13:11.423288 (+ 14us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 12:13:11.423788 (+ 500us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 12:13:11.424299 (+ 511us) server_negotiation.cc:226] Negotiated authn=SASL
0809 12:13:11.426034 (+ 1735us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 12:13:11.427379 (+ 1345us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 12:13:11.429148 (+ 1769us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 12:13:11.430185 (+ 1037us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 12:13:11.430380 (+ 195us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 12:13:11.439053 (+ 8673us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 12:13:11.439067 (+ 14us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 12:13:11.439090 (+ 23us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0809 12:13:11.439145 (+ 55us) server_negotiation.cc:922] calling sasl_server_start()
0809 12:13:11.443971 (+ 4826us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 12:13:11.445119 (+ 1148us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 12:13:11.445129 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 12:13:11.445137 (+ 8us) server_negotiation.cc:967] calling sasl_server_step()
0809 12:13:11.445377 (+ 240us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 12:13:11.445994 (+ 617us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 12:13:11.446000 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 12:13:11.446003 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0809 12:13:11.446729 (+ 726us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 12:13:11.446948 (+ 219us) server_negotiation.cc:1035] Waiting for connection context
0809 12:13:11.448012 (+ 1064us) server_negotiation.cc:299] Negotiation successful
0809 12:13:11.448438 (+ 426us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":476,"spinlock_wait_cycles":28544,"thread_start_us":388,"threads_started":1}
I20260809 12:13:11.454766 15013 heartbeater.cc:347] Connected to a master server at 127.0.222.62:37183
I20260809 12:13:11.455144 15013 heartbeater.cc:464] Registering TS with master...
I20260809 12:13:11.456107 15013 heartbeater.cc:511] Master 127.0.222.62:37183 requested a full tablet report, sending...
W20260809 12:13:11.459972 15013 heartbeater.cc:644] Failed to heartbeat to 127.0.222.62:37183 (0 consecutive failures): Service unavailable: Catalog manager is not initialized. State: Starting
I20260809 12:13:11.460229 15013 heartbeater.cc:464] Registering TS with master...
I20260809 12:13:11.460526 15013 heartbeater.cc:511] Master 127.0.222.62:37183 requested a full tablet report, sending...
I20260809 12:13:11.461824 15013 heartbeater.cc:464] Registering TS with master...
I20260809 12:13:11.462142 15013 heartbeater.cc:511] Master 127.0.222.62:37183 requested a full tablet report, sending...
W20260809 12:13:11.463328 15013 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
W20260809 12:13:11.868225 15017 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260809 12:13:11.868628 15017 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260809 12:13:11.868848 15017 flags.cc:425] Enabled unsafe flag: --never_fsync=true
Aug 09 12:13:11 dist-test-slave-2w3w krb5kdc[13742](info): AS_REQ (1 etypes {17}) 127.0.0.1: ISSUE: authtime 1786277591, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.222.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
W20260809 12:13:11.880607 15017 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260809 12:13:11.880829 15017 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260809 12:13:11.881076 15017 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.222.2
I20260809 12:13:11.897110 15017 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.222.20:37035
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskgH5Pjm/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786277382073938-888-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskgH5Pjm/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786277382073938-888-0/minicluster-data/ts-1/wal
--ranger_kms_url=invalid.host:1234/kms,127.0.222.20:55861/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskgH5Pjm/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786277382073938-888-0/krb5kdc/kudu_127.0.222.2.keytab
--principal=kudu/127.0.222.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.222.2:33317
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskgH5Pjm/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786277382073938-888-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.222.2
--webserver_port=34383
--tserver_master_addrs=127.0.222.62:37183
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.222.2
--log_dir=/tmp/dist-test-taskgH5Pjm/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786277382073938-888-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 09 Aug 2026 11:06:41 UTC on ccycloud.quasar-bwoogy.root.comops.site
build id 4575991
ASAN enabled
I20260809 12:13:11.899295 15017 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260809 12:13:11.901129 15017 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260809 12:13:11.923128 15034 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 12:13:11.924557 15035 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 12:13:11.934834 15037 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260809 12:13:11.936002 15017 server_base.cc:1034] running on GCE node
I20260809 12:13:11.936857 15017 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260809 12:13:11.938158 15017 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260809 12:13:11.939507 15017 hybrid_clock.cc:648] HybridClock initialized: now 1786277591939425 us; error 63 us; skew 500 ppm
I20260809 12:13:11.939914 15017 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 09 12:13:11 dist-test-slave-2w3w krb5kdc[13742](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786277591, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.222.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260809 12:13:11.944236 15017 init.cc:377] Logged in from keytab as kudu/127.0.222.2@KRBTEST.COM (short username kudu)
I20260809 12:13:11.946316 15017 webserver.cc:460] Webserver started at http://127.0.222.2:34383/ using document root <none> and password file <none>
I20260809 12:13:11.947309 15017 fs_manager.cc:359] Metadata directory not provided
I20260809 12:13:11.947472 15017 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
Aug 09 12:13:11 dist-test-slave-2w3w krb5kdc[13742](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786277591, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.222.2@KRBTEST.COM for HTTP/127.0.222.20@KRBTEST.COM
I20260809 12:13:11.979645 15017 fs_manager.cc:709] Time spent opening directory manager: real 0.012s user 0.004s sys 0.007s
I20260809 12:13:11.986094 15046 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260809 12:13:11.988430 15017 fs_manager.cc:725] Time spent opening block manager: real 0.007s user 0.004s sys 0.000s
I20260809 12:13:11.988647 15017 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskgH5Pjm/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786277382073938-888-0/minicluster-data/ts-1/data,/tmp/dist-test-taskgH5Pjm/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786277382073938-888-0/minicluster-data/ts-1/wal
uuid: "b13c52d3d307484390935460b384345d"
format_stamp: "Formatted at 2026-08-09 12:13:08 on dist-test-slave-2w3w"
server_key: "7e29fdcfbd1fb501b2b588115913157d"
server_key_iv: "60cbf025fdf5581953cd0aa0c785e438"
server_key_version: "kuduclusterkey@0"
I20260809 12:13:11.989820 15017 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskgH5Pjm/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786277382073938-888-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskgH5Pjm/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786277382073938-888-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskgH5Pjm/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786277382073938-888-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260809 12:13:12.028017 15017 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260809 12:13:12.032871 15017 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260809 12:13:12.033223 15017 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260809 12:13:12.034613 15017 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260809 12:13:12.036561 15017 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260809 12:13:12.036737 15017 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.001s sys 0.000s
I20260809 12:13:12.036917 15017 ts_tablet_manager.cc:610] Registered 0 tablets
I20260809 12:13:12.037046 15017 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260809 12:13:12.095927 15017 rpc_server.cc:307] RPC server started. Bound to: 127.0.222.2:33317
I20260809 12:13:12.096012 15169 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.222.2:33317 every 8 connection(s)
I20260809 12:13:12.097955 15017 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskgH5Pjm/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786277382073938-888-0/minicluster-data/ts-1/data/info.pb
I20260809 12:13:12.104486 888 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskgH5Pjm/build/asan/bin/kudu as pid 15017
I20260809 12:13:12.105283 888 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskgH5Pjm/build/asan/bin/kudu
/tmp/dist-test-taskgH5Pjm/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskgH5Pjm/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786277382073938-888-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskgH5Pjm/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786277382073938-888-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskgH5Pjm/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786277382073938-888-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskgH5Pjm/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786277382073938-888-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.0.222.20:55861/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.222.3:43891
--local_ip_for_outbound_sockets=127.0.222.3
--tserver_master_addrs=127.0.222.62:37183
--webserver_port=42365
--webserver_interface=127.0.222.3
--keytab_file=/tmp/dist-test-taskgH5Pjm/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786277382073938-888-0/krb5kdc/kudu_127.0.222.3.keytab
--principal=kudu/127.0.222.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.222.20:37035
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--ranger_kms_url=invalid.host:1234/kms,127.0.222.20:55861/kms with env {KRB5CCNAME=/tmp/dist-test-taskgH5Pjm/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786277382073938-888-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskgH5Pjm/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786277382073938-888-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskgH5Pjm/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786277382073938-888-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Aug 09 12:13:12 dist-test-slave-2w3w krb5kdc[13742](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786277591, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.222.2@KRBTEST.COM for kudu/127.0.222.62@KRBTEST.COM
I20260809 12:13:12.179687 15173 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 12:13:12.102687 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.222.2:48627 (local address 127.0.222.62:37183)
0809 12:13:12.144937 (+ 42250us) server_negotiation.cc:206] Beginning negotiation
0809 12:13:12.144944 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0809 12:13:12.149172 (+ 4228us) server_negotiation.cc:407] Connection header received
0809 12:13:12.149251 (+ 79us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 12:13:12.149256 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 12:13:12.149345 (+ 89us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 12:13:12.149473 (+ 128us) server_negotiation.cc:226] Negotiated authn=SASL
0809 12:13:12.162108 (+ 12635us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 12:13:12.163350 (+ 1242us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 12:13:12.165926 (+ 2576us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 12:13:12.166614 (+ 688us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 12:13:12.166769 (+ 155us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 12:13:12.173434 (+ 6665us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 12:13:12.173439 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 12:13:12.173442 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0809 12:13:12.173481 (+ 39us) server_negotiation.cc:922] calling sasl_server_start()
0809 12:13:12.176365 (+ 2884us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 12:13:12.177194 (+ 829us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 12:13:12.177198 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 12:13:12.177200 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0809 12:13:12.177333 (+ 133us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 12:13:12.177826 (+ 493us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 12:13:12.177830 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 12:13:12.177832 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0809 12:13:12.178106 (+ 274us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 12:13:12.178275 (+ 169us) server_negotiation.cc:1035] Waiting for connection context
0809 12:13:12.179038 (+ 763us) server_negotiation.cc:299] Negotiation successful
0809 12:13:12.179272 (+ 234us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":42085,"thread_start_us":42272,"threads_started":1}
I20260809 12:13:12.180760 15172 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 12:13:12.102085 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.222.62:37183 (local address 127.0.222.2:48627)
0809 12:13:12.145747 (+ 43662us) negotiation.cc:107] Waiting for socket to connect
0809 12:13:12.145824 (+ 77us) client_negotiation.cc:174] Beginning negotiation
0809 12:13:12.148612 (+ 2788us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 12:13:12.160065 (+ 11453us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 12:13:12.160080 (+ 15us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 12:13:12.160854 (+ 774us) client_negotiation.cc:189] Negotiated authn=SASL
0809 12:13:12.161814 (+ 960us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 12:13:12.161844 (+ 30us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 12:13:12.163603 (+ 1759us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 12:13:12.163609 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 12:13:12.165729 (+ 2120us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 12:13:12.165746 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 12:13:12.167154 (+ 1408us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 12:13:12.167165 (+ 11us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 12:13:12.167395 (+ 230us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 12:13:12.168199 (+ 804us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0809 12:13:12.168231 (+ 32us) client_negotiation.cc:656] calling sasl_client_start()
0809 12:13:12.173147 (+ 4916us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0809 12:13:12.176539 (+ 3392us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0809 12:13:12.176550 (+ 11us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0809 12:13:12.176568 (+ 18us) client_negotiation.cc:760] calling sasl_client_step()
0809 12:13:12.177042 (+ 474us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0809 12:13:12.177450 (+ 408us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0809 12:13:12.177457 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0809 12:13:12.177461 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0809 12:13:12.177689 (+ 228us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0809 12:13:12.178243 (+ 554us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0809 12:13:12.178251 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0809 12:13:12.178621 (+ 370us) client_negotiation.cc:769] Sending connection context
0809 12:13:12.178897 (+ 276us) client_negotiation.cc:240] Negotiation successful
0809 12:13:12.179512 (+ 615us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":43261,"thread_start_us":47440,"threads_started":1}
I20260809 12:13:12.182407 15170 heartbeater.cc:347] Connected to a master server at 127.0.222.62:37183
I20260809 12:13:12.182741 15170 heartbeater.cc:464] Registering TS with master...
I20260809 12:13:12.183645 15170 heartbeater.cc:511] Master 127.0.222.62:37183 requested a full tablet report, sending...
I20260809 12:13:12.185473 14802 ts_manager.cc:194] Registered new tserver with Master: b13c52d3d307484390935460b384345d (127.0.222.2:33317)
I20260809 12:13:12.187146 14802 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.222.2@KRBTEST.COM'} at 127.0.222.2:48627
Aug 09 12:13:12 dist-test-slave-2w3w krb5kdc[13742](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786277591, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.222.62@KRBTEST.COM for HTTP/127.0.222.20@KRBTEST.COM
I20260809 12:13:12.463511 15013 heartbeater.cc:464] Registering TS with master...
I20260809 12:13:12.464037 15013 heartbeater.cc:511] Master 127.0.222.62:37183 requested a full tablet report, sending...
I20260809 12:13:12.465420 14802 ts_manager.cc:194] Registered new tserver with Master: 0d897ca0b7594fb6952f1d1f90301da6 (127.0.222.1:38517)
I20260809 12:13:12.466640 14802 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.222.1@KRBTEST.COM'} at 127.0.222.1:58103
W20260809 12:13:12.616995 15174 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260809 12:13:12.617491 15174 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260809 12:13:12.617677 15174 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260809 12:13:12.629052 15174 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260809 12:13:12.629204 15174 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260809 12:13:12.629434 15174 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.222.3
I20260809 12:13:12.645038 15174 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.222.20:37035
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskgH5Pjm/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786277382073938-888-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskgH5Pjm/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786277382073938-888-0/minicluster-data/ts-2/wal
--ranger_kms_url=invalid.host:1234/kms,127.0.222.20:55861/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskgH5Pjm/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786277382073938-888-0/krb5kdc/kudu_127.0.222.3.keytab
--principal=kudu/127.0.222.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.222.3:43891
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskgH5Pjm/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786277382073938-888-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.222.3
--webserver_port=42365
--tserver_master_addrs=127.0.222.62:37183
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.222.3
--log_dir=/tmp/dist-test-taskgH5Pjm/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786277382073938-888-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 09 Aug 2026 11:06:41 UTC on ccycloud.quasar-bwoogy.root.comops.site
build id 4575991
ASAN enabled
I20260809 12:13:12.647022 15174 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260809 12:13:12.648877 15174 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260809 12:13:12.670080 15180 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 12:13:12.681954 15181 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260809 12:13:12.691337 15174 server_base.cc:1034] running on GCE node
W20260809 12:13:12.692452 15184 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260809 12:13:12.693534 15174 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
Aug 09 12:13:12 dist-test-slave-2w3w krb5kdc[13742](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786277591, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.222.62@KRBTEST.COM for HTTP/127.0.222.20@KRBTEST.COM
I20260809 12:13:12.701300 15174 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260809 12:13:12.702654 15174 hybrid_clock.cc:648] HybridClock initialized: now 1786277592702535 us; error 114 us; skew 500 ppm
I20260809 12:13:12.703130 15174 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 09 12:13:12 dist-test-slave-2w3w krb5kdc[13742](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786277592, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.222.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260809 12:13:12.707612 15174 init.cc:377] Logged in from keytab as kudu/127.0.222.3@KRBTEST.COM (short username kudu)
I20260809 12:13:12.711375 15174 webserver.cc:460] Webserver started at http://127.0.222.3:42365/ using document root <none> and password file <none>
I20260809 12:13:12.712437 15174 fs_manager.cc:359] Metadata directory not provided
I20260809 12:13:12.712623 15174 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
Aug 09 12:13:12 dist-test-slave-2w3w krb5kdc[13742](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786277592, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.222.3@KRBTEST.COM for HTTP/127.0.222.20@KRBTEST.COM
I20260809 12:13:12.764011 15174 fs_manager.cc:709] Time spent opening directory manager: real 0.012s user 0.001s sys 0.009s
I20260809 12:13:12.769312 15192 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260809 12:13:12.771229 15174 fs_manager.cc:725] Time spent opening block manager: real 0.005s user 0.005s sys 0.000s
I20260809 12:13:12.771410 15174 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskgH5Pjm/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786277382073938-888-0/minicluster-data/ts-2/data,/tmp/dist-test-taskgH5Pjm/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786277382073938-888-0/minicluster-data/ts-2/wal
uuid: "22bd9601d42e40a880eec51934dfeb23"
format_stamp: "Formatted at 2026-08-09 12:13:09 on dist-test-slave-2w3w"
server_key: "c6d89da093626487e1d1aba1775aee2b"
server_key_iv: "506feaf4020590927820e9aa2a2e738f"
server_key_version: "kuduclusterkey@0"
I20260809 12:13:12.772847 15174 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskgH5Pjm/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786277382073938-888-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskgH5Pjm/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786277382073938-888-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskgH5Pjm/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786277382073938-888-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260809 12:13:12.826062 15174 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260809 12:13:12.831818 15174 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260809 12:13:12.832142 15174 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260809 12:13:12.843439 15174 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260809 12:13:12.847975 15174 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260809 12:13:12.848109 15174 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260809 12:13:12.848229 15174 ts_tablet_manager.cc:610] Registered 0 tablets
I20260809 12:13:12.848315 15174 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260809 12:13:12.958436 15174 rpc_server.cc:307] RPC server started. Bound to: 127.0.222.3:43891
I20260809 12:13:12.958473 15317 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.222.3:43891 every 8 connection(s)
I20260809 12:13:12.960311 15174 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskgH5Pjm/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786277382073938-888-0/minicluster-data/ts-2/data/info.pb
I20260809 12:13:12.970161 888 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskgH5Pjm/build/asan/bin/kudu as pid 15174
Aug 09 12:13:12 dist-test-slave-2w3w krb5kdc[13742](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786277592, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.222.3@KRBTEST.COM for kudu/127.0.222.62@KRBTEST.COM
I20260809 12:13:12.985309 15320 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 12:13:12.963061 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.222.3:48583 (local address 127.0.222.62:37183)
0809 12:13:12.963590 (+ 529us) server_negotiation.cc:206] Beginning negotiation
0809 12:13:12.963594 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0809 12:13:12.964902 (+ 1308us) server_negotiation.cc:407] Connection header received
0809 12:13:12.966924 (+ 2022us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 12:13:12.966930 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 12:13:12.967049 (+ 119us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 12:13:12.967184 (+ 135us) server_negotiation.cc:226] Negotiated authn=SASL
0809 12:13:12.968896 (+ 1712us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 12:13:12.969779 (+ 883us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 12:13:12.972534 (+ 2755us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 12:13:12.973246 (+ 712us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 12:13:12.973383 (+ 137us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 12:13:12.977490 (+ 4107us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 12:13:12.977495 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 12:13:12.977498 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0809 12:13:12.977543 (+ 45us) server_negotiation.cc:922] calling sasl_server_start()
0809 12:13:12.981256 (+ 3713us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 12:13:12.982260 (+ 1004us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 12:13:12.982266 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 12:13:12.982269 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0809 12:13:12.982445 (+ 176us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 12:13:12.983234 (+ 789us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 12:13:12.983241 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 12:13:12.983249 (+ 8us) server_negotiation.cc:967] calling sasl_server_step()
0809 12:13:12.983548 (+ 299us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 12:13:12.983716 (+ 168us) server_negotiation.cc:1035] Waiting for connection context
0809 12:13:12.984606 (+ 890us) server_negotiation.cc:299] Negotiation successful
0809 12:13:12.984876 (+ 270us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":397,"thread_start_us":268,"threads_started":1}
I20260809 12:13:12.986477 15321 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 12:13:12.963853 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.222.62:37183 (local address 127.0.222.3:48583)
0809 12:13:12.964715 (+ 862us) negotiation.cc:107] Waiting for socket to connect
0809 12:13:12.964761 (+ 46us) client_negotiation.cc:174] Beginning negotiation
0809 12:13:12.966613 (+ 1852us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 12:13:12.967495 (+ 882us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 12:13:12.967503 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 12:13:12.967969 (+ 466us) client_negotiation.cc:189] Negotiated authn=SASL
0809 12:13:12.968631 (+ 662us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 12:13:12.968694 (+ 63us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 12:13:12.969950 (+ 1256us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 12:13:12.969955 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 12:13:12.972362 (+ 2407us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 12:13:12.972377 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 12:13:12.973520 (+ 1143us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 12:13:12.973527 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 12:13:12.973887 (+ 360us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 12:13:12.974735 (+ 848us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0809 12:13:12.974786 (+ 51us) client_negotiation.cc:656] calling sasl_client_start()
0809 12:13:12.977254 (+ 2468us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0809 12:13:12.981437 (+ 4183us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0809 12:13:12.981450 (+ 13us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0809 12:13:12.981471 (+ 21us) client_negotiation.cc:760] calling sasl_client_step()
0809 12:13:12.982123 (+ 652us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0809 12:13:12.982585 (+ 462us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0809 12:13:12.982591 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0809 12:13:12.982595 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0809 12:13:12.982815 (+ 220us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0809 12:13:12.983697 (+ 882us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0809 12:13:12.983706 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0809 12:13:12.984151 (+ 445us) client_negotiation.cc:769] Sending connection context
0809 12:13:12.984486 (+ 335us) client_negotiation.cc:240] Negotiation successful
0809 12:13:12.985072 (+ 586us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":519,"thread_start_us":313,"threads_started":1}
I20260809 12:13:12.988209 15318 heartbeater.cc:347] Connected to a master server at 127.0.222.62:37183
I20260809 12:13:12.988538 15318 heartbeater.cc:464] Registering TS with master...
I20260809 12:13:12.989274 15318 heartbeater.cc:511] Master 127.0.222.62:37183 requested a full tablet report, sending...
I20260809 12:13:12.990869 14802 ts_manager.cc:194] Registered new tserver with Master: 22bd9601d42e40a880eec51934dfeb23 (127.0.222.3:43891)
I20260809 12:13:12.991314 15324 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 12:13:12.971693 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52870 (local address 127.0.222.62:37183)
0809 12:13:12.972226 (+ 533us) server_negotiation.cc:206] Beginning negotiation
0809 12:13:12.972233 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0809 12:13:12.972251 (+ 18us) server_negotiation.cc:407] Connection header received
0809 12:13:12.972311 (+ 60us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 12:13:12.972316 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 12:13:12.972424 (+ 108us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 12:13:12.972526 (+ 102us) server_negotiation.cc:226] Negotiated authn=SASL
0809 12:13:12.974760 (+ 2234us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 12:13:12.976042 (+ 1282us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 12:13:12.978233 (+ 2191us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 12:13:12.979044 (+ 811us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 12:13:12.979196 (+ 152us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 12:13:12.982167 (+ 2971us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 12:13:12.982174 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 12:13:12.982179 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0809 12:13:12.982231 (+ 52us) server_negotiation.cc:922] calling sasl_server_start()
0809 12:13:12.986147 (+ 3916us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 12:13:12.987334 (+ 1187us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 12:13:12.987340 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 12:13:12.987343 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0809 12:13:12.987510 (+ 167us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 12:13:12.987976 (+ 466us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 12:13:12.987981 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 12:13:12.987985 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0809 12:13:12.988309 (+ 324us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 12:13:12.988557 (+ 248us) server_negotiation.cc:1035] Waiting for connection context
0809 12:13:12.990608 (+ 2051us) server_negotiation.cc:299] Negotiation successful
0809 12:13:12.990845 (+ 237us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":419,"thread_start_us":275,"threads_started":1}
I20260809 12:13:12.992105 14802 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.222.3@KRBTEST.COM'} at 127.0.222.3:48583
I20260809 12:13:12.997381 888 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260809 12:13:13.016175 15324 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 12:13:13.001289 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52876 (local address 127.0.222.62:37183)
0809 12:13:13.001471 (+ 182us) server_negotiation.cc:206] Beginning negotiation
0809 12:13:13.001477 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0809 12:13:13.001993 (+ 516us) server_negotiation.cc:407] Connection header received
0809 12:13:13.002145 (+ 152us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 12:13:13.002151 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 12:13:13.002245 (+ 94us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 12:13:13.002349 (+ 104us) server_negotiation.cc:226] Negotiated authn=SASL
0809 12:13:13.004023 (+ 1674us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 12:13:13.005129 (+ 1106us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 12:13:13.006493 (+ 1364us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 12:13:13.007212 (+ 719us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 12:13:13.007345 (+ 133us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 12:13:13.009918 (+ 2573us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 12:13:13.009925 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 12:13:13.009930 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0809 12:13:13.009971 (+ 41us) server_negotiation.cc:922] calling sasl_server_start()
0809 12:13:13.013408 (+ 3437us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 12:13:13.014066 (+ 658us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 12:13:13.014071 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 12:13:13.014075 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0809 12:13:13.014234 (+ 159us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 12:13:13.014607 (+ 373us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 12:13:13.014611 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 12:13:13.014614 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0809 12:13:13.015060 (+ 446us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 12:13:13.015306 (+ 246us) server_negotiation.cc:1035] Waiting for connection context
0809 12:13:13.015581 (+ 275us) server_negotiation.cc:299] Negotiation successful
0809 12:13:13.015824 (+ 243us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":66}
I20260809 12:13:13.021404 14802 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:52876:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260809 12:13:13.028872 14802 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260809 12:13:13.091763 15337 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 12:13:13.043996 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.222.3:43891 (local address 127.0.0.1:35498)
0809 12:13:13.067478 (+ 23482us) negotiation.cc:107] Waiting for socket to connect
0809 12:13:13.067554 (+ 76us) client_negotiation.cc:174] Beginning negotiation
0809 12:13:13.067768 (+ 214us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 12:13:13.081309 (+ 13541us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 12:13:13.081317 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 12:13:13.081374 (+ 57us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0809 12:13:13.081873 (+ 499us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 12:13:13.081890 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 12:13:13.084665 (+ 2775us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 12:13:13.084674 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 12:13:13.088012 (+ 3338us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 12:13:13.088035 (+ 23us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 12:13:13.090740 (+ 2705us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 12:13:13.090746 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 12:13:13.090992 (+ 246us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 12:13:13.091046 (+ 54us) client_negotiation.cc:769] Sending connection context
0809 12:13:13.091178 (+ 132us) client_negotiation.cc:240] Negotiation successful
0809 12:13:13.091384 (+ 206us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":23115,"thread_start_us":24144,"threads_started":1}
I20260809 12:13:13.092202 15334 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 12:13:13.044813 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.222.1:38517 (local address 127.0.0.1:59352)
0809 12:13:13.068382 (+ 23569us) negotiation.cc:107] Waiting for socket to connect
0809 12:13:13.068399 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0809 12:13:13.068522 (+ 123us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 12:13:13.080918 (+ 12396us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 12:13:13.080926 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 12:13:13.080970 (+ 44us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0809 12:13:13.082113 (+ 1143us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 12:13:13.082127 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 12:13:13.084199 (+ 2072us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 12:13:13.084204 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 12:13:13.087301 (+ 3097us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 12:13:13.087317 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 12:13:13.091411 (+ 4094us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 12:13:13.091417 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 12:13:13.091606 (+ 189us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 12:13:13.091636 (+ 30us) client_negotiation.cc:769] Sending connection context
0809 12:13:13.091705 (+ 69us) client_negotiation.cc:240] Negotiation successful
0809 12:13:13.091833 (+ 128us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":23436,"thread_start_us":30433,"threads_started":1}
I20260809 12:13:13.092885 15338 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 12:13:13.049635 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59352 (local address 127.0.222.1:38517)
0809 12:13:13.066515 (+ 16880us) server_negotiation.cc:206] Beginning negotiation
0809 12:13:13.066529 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0809 12:13:13.068657 (+ 2128us) server_negotiation.cc:407] Connection header received
0809 12:13:13.068763 (+ 106us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 12:13:13.068772 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 12:13:13.069095 (+ 323us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 12:13:13.077181 (+ 8086us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0809 12:13:13.082362 (+ 5181us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 12:13:13.084011 (+ 1649us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 12:13:13.088640 (+ 4629us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 12:13:13.090136 (+ 1496us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 12:13:13.090237 (+ 101us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 12:13:13.090488 (+ 251us) server_negotiation.cc:1035] Waiting for connection context
0809 12:13:13.092366 (+ 1878us) server_negotiation.cc:299] Negotiation successful
0809 12:13:13.092546 (+ 180us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":16615,"thread_start_us":16875,"threads_started":1}
I20260809 12:13:13.094245 15335 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 12:13:13.044596 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35498 (local address 127.0.222.3:43891)
0809 12:13:13.080102 (+ 35506us) server_negotiation.cc:206] Beginning negotiation
0809 12:13:13.080115 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0809 12:13:13.080161 (+ 46us) server_negotiation.cc:407] Connection header received
0809 12:13:13.080296 (+ 135us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 12:13:13.080308 (+ 12us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 12:13:13.080661 (+ 353us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 12:13:13.080913 (+ 252us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0809 12:13:13.082205 (+ 1292us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 12:13:13.084453 (+ 2248us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 12:13:13.088308 (+ 3855us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 12:13:13.090388 (+ 2080us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 12:13:13.093293 (+ 2905us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 12:13:13.093628 (+ 335us) server_negotiation.cc:1035] Waiting for connection context
0809 12:13:13.093702 (+ 74us) server_negotiation.cc:299] Negotiation successful
0809 12:13:13.093926 (+ 224us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":35234,"thread_start_us":35352,"threads_started":1}
I20260809 12:13:13.100282 15340 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 12:13:13.069375 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40466 (local address 127.0.222.2:33317)
0809 12:13:13.081695 (+ 12320us) server_negotiation.cc:206] Beginning negotiation
0809 12:13:13.081704 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0809 12:13:13.091237 (+ 9533us) server_negotiation.cc:407] Connection header received
0809 12:13:13.091357 (+ 120us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 12:13:13.091365 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 12:13:13.091639 (+ 274us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 12:13:13.091874 (+ 235us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0809 12:13:13.092787 (+ 913us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 12:13:13.094429 (+ 1642us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 12:13:13.096970 (+ 2541us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 12:13:13.098671 (+ 1701us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 12:13:13.098842 (+ 171us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 12:13:13.099160 (+ 318us) server_negotiation.cc:1035] Waiting for connection context
0809 12:13:13.099745 (+ 585us) server_negotiation.cc:299] Negotiation successful
0809 12:13:13.099939 (+ 194us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":12151,"thread_start_us":12318,"threads_started":1}
I20260809 12:13:13.101006 15339 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 12:13:13.069304 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.222.2:33317 (local address 127.0.0.1:40466)
0809 12:13:13.091075 (+ 21771us) negotiation.cc:107] Waiting for socket to connect
0809 12:13:13.091093 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0809 12:13:13.091224 (+ 131us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 12:13:13.092155 (+ 931us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 12:13:13.092160 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 12:13:13.092186 (+ 26us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0809 12:13:13.092557 (+ 371us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 12:13:13.092570 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 12:13:13.094624 (+ 2054us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 12:13:13.094627 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 12:13:13.096773 (+ 2146us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 12:13:13.096788 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 12:13:13.099358 (+ 2570us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 12:13:13.099365 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 12:13:13.099539 (+ 174us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 12:13:13.099572 (+ 33us) client_negotiation.cc:769] Sending connection context
0809 12:13:13.100538 (+ 966us) client_negotiation.cc:240] Negotiation successful
0809 12:13:13.100682 (+ 144us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":21618,"thread_start_us":21423,"threads_started":1}
I20260809 12:13:13.101980 14944 tablet_service.cc:1467] Processing CreateTablet for tablet 2838dc8950034297ad4c3e10ca27bcaa (DEFAULT_TABLE table=test-table [id=32ad5abe0ef341f09cac21b315b9e821]), partition=RANGE (key) PARTITION UNBOUNDED
I20260809 12:13:13.103170 15243 tablet_service.cc:1467] Processing CreateTablet for tablet 2838dc8950034297ad4c3e10ca27bcaa (DEFAULT_TABLE table=test-table [id=32ad5abe0ef341f09cac21b315b9e821]), partition=RANGE (key) PARTITION UNBOUNDED
I20260809 12:13:13.104400 15243 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 2838dc8950034297ad4c3e10ca27bcaa. 1 dirs total, 0 dirs full, 0 dirs failed
I20260809 12:13:13.106146 15099 tablet_service.cc:1467] Processing CreateTablet for tablet 2838dc8950034297ad4c3e10ca27bcaa (DEFAULT_TABLE table=test-table [id=32ad5abe0ef341f09cac21b315b9e821]), partition=RANGE (key) PARTITION UNBOUNDED
I20260809 12:13:13.107510 15099 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 2838dc8950034297ad4c3e10ca27bcaa. 1 dirs total, 0 dirs full, 0 dirs failed
I20260809 12:13:13.107679 14944 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 2838dc8950034297ad4c3e10ca27bcaa. 1 dirs total, 0 dirs full, 0 dirs failed
I20260809 12:13:13.125933 15341 tablet_bootstrap.cc:492] T 2838dc8950034297ad4c3e10ca27bcaa P 22bd9601d42e40a880eec51934dfeb23: Bootstrap starting.
I20260809 12:13:13.130324 15343 tablet_bootstrap.cc:492] T 2838dc8950034297ad4c3e10ca27bcaa P 0d897ca0b7594fb6952f1d1f90301da6: Bootstrap starting.
I20260809 12:13:13.130823 15341 tablet_bootstrap.cc:654] T 2838dc8950034297ad4c3e10ca27bcaa P 22bd9601d42e40a880eec51934dfeb23: Neither blocks nor log segments found. Creating new log.
I20260809 12:13:13.133131 15341 log.cc:824] T 2838dc8950034297ad4c3e10ca27bcaa P 22bd9601d42e40a880eec51934dfeb23: Log is configured to *not* fsync() on all Append() calls
I20260809 12:13:13.131629 15342 tablet_bootstrap.cc:492] T 2838dc8950034297ad4c3e10ca27bcaa P b13c52d3d307484390935460b384345d: Bootstrap starting.
I20260809 12:13:13.137084 15343 tablet_bootstrap.cc:654] T 2838dc8950034297ad4c3e10ca27bcaa P 0d897ca0b7594fb6952f1d1f90301da6: Neither blocks nor log segments found. Creating new log.
I20260809 12:13:13.137830 15342 tablet_bootstrap.cc:654] T 2838dc8950034297ad4c3e10ca27bcaa P b13c52d3d307484390935460b384345d: Neither blocks nor log segments found. Creating new log.
I20260809 12:13:13.138939 15343 log.cc:824] T 2838dc8950034297ad4c3e10ca27bcaa P 0d897ca0b7594fb6952f1d1f90301da6: Log is configured to *not* fsync() on all Append() calls
I20260809 12:13:13.139213 15342 log.cc:824] T 2838dc8950034297ad4c3e10ca27bcaa P b13c52d3d307484390935460b384345d: Log is configured to *not* fsync() on all Append() calls
I20260809 12:13:13.143621 15341 tablet_bootstrap.cc:492] T 2838dc8950034297ad4c3e10ca27bcaa P 22bd9601d42e40a880eec51934dfeb23: No bootstrap required, opened a new log
I20260809 12:13:13.144075 15341 ts_tablet_manager.cc:1397] T 2838dc8950034297ad4c3e10ca27bcaa P 22bd9601d42e40a880eec51934dfeb23: Time spent bootstrapping tablet: real 0.018s user 0.011s sys 0.005s
I20260809 12:13:13.156647 15342 tablet_bootstrap.cc:492] T 2838dc8950034297ad4c3e10ca27bcaa P b13c52d3d307484390935460b384345d: No bootstrap required, opened a new log
I20260809 12:13:13.156952 15342 ts_tablet_manager.cc:1397] T 2838dc8950034297ad4c3e10ca27bcaa P b13c52d3d307484390935460b384345d: Time spent bootstrapping tablet: real 0.026s user 0.006s sys 0.010s
I20260809 12:13:13.157480 15343 tablet_bootstrap.cc:492] T 2838dc8950034297ad4c3e10ca27bcaa P 0d897ca0b7594fb6952f1d1f90301da6: No bootstrap required, opened a new log
I20260809 12:13:13.157820 15343 ts_tablet_manager.cc:1397] T 2838dc8950034297ad4c3e10ca27bcaa P 0d897ca0b7594fb6952f1d1f90301da6: Time spent bootstrapping tablet: real 0.028s user 0.008s sys 0.013s
I20260809 12:13:13.157774 15341 raft_consensus.cc:348] T 2838dc8950034297ad4c3e10ca27bcaa P 22bd9601d42e40a880eec51934dfeb23 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "22bd9601d42e40a880eec51934dfeb23" member_type: VOTER last_known_addr { host: "127.0.222.3" port: 43891 } } peers { permanent_uuid: "b13c52d3d307484390935460b384345d" member_type: VOTER last_known_addr { host: "127.0.222.2" port: 33317 } } peers { permanent_uuid: "0d897ca0b7594fb6952f1d1f90301da6" member_type: VOTER last_known_addr { host: "127.0.222.1" port: 38517 } }
I20260809 12:13:13.158398 15341 raft_consensus.cc:374] T 2838dc8950034297ad4c3e10ca27bcaa P 22bd9601d42e40a880eec51934dfeb23 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260809 12:13:13.158567 15341 raft_consensus.cc:729] T 2838dc8950034297ad4c3e10ca27bcaa P 22bd9601d42e40a880eec51934dfeb23 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 22bd9601d42e40a880eec51934dfeb23, State: Initialized, Role: FOLLOWER
I20260809 12:13:13.159437 15341 consensus_queue.cc:260] T 2838dc8950034297ad4c3e10ca27bcaa P 22bd9601d42e40a880eec51934dfeb23 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "22bd9601d42e40a880eec51934dfeb23" member_type: VOTER last_known_addr { host: "127.0.222.3" port: 43891 } } peers { permanent_uuid: "b13c52d3d307484390935460b384345d" member_type: VOTER last_known_addr { host: "127.0.222.2" port: 33317 } } peers { permanent_uuid: "0d897ca0b7594fb6952f1d1f90301da6" member_type: VOTER last_known_addr { host: "127.0.222.1" port: 38517 } }
I20260809 12:13:13.167689 15342 raft_consensus.cc:348] T 2838dc8950034297ad4c3e10ca27bcaa P b13c52d3d307484390935460b384345d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "22bd9601d42e40a880eec51934dfeb23" member_type: VOTER last_known_addr { host: "127.0.222.3" port: 43891 } } peers { permanent_uuid: "b13c52d3d307484390935460b384345d" member_type: VOTER last_known_addr { host: "127.0.222.2" port: 33317 } } peers { permanent_uuid: "0d897ca0b7594fb6952f1d1f90301da6" member_type: VOTER last_known_addr { host: "127.0.222.1" port: 38517 } }
I20260809 12:13:13.168342 15342 raft_consensus.cc:374] T 2838dc8950034297ad4c3e10ca27bcaa P b13c52d3d307484390935460b384345d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260809 12:13:13.168515 15342 raft_consensus.cc:729] T 2838dc8950034297ad4c3e10ca27bcaa P b13c52d3d307484390935460b384345d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b13c52d3d307484390935460b384345d, State: Initialized, Role: FOLLOWER
I20260809 12:13:13.169387 15342 consensus_queue.cc:260] T 2838dc8950034297ad4c3e10ca27bcaa P b13c52d3d307484390935460b384345d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "22bd9601d42e40a880eec51934dfeb23" member_type: VOTER last_known_addr { host: "127.0.222.3" port: 43891 } } peers { permanent_uuid: "b13c52d3d307484390935460b384345d" member_type: VOTER last_known_addr { host: "127.0.222.2" port: 33317 } } peers { permanent_uuid: "0d897ca0b7594fb6952f1d1f90301da6" member_type: VOTER last_known_addr { host: "127.0.222.1" port: 38517 } }
I20260809 12:13:13.170397 15341 ts_tablet_manager.cc:1428] T 2838dc8950034297ad4c3e10ca27bcaa P 22bd9601d42e40a880eec51934dfeb23: Time spent starting tablet: real 0.026s user 0.023s sys 0.000s
I20260809 12:13:13.171155 15318 heartbeater.cc:503] Master 127.0.222.62:37183 was elected leader, sending a full tablet report...
I20260809 12:13:13.175173 15343 raft_consensus.cc:348] T 2838dc8950034297ad4c3e10ca27bcaa P 0d897ca0b7594fb6952f1d1f90301da6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "22bd9601d42e40a880eec51934dfeb23" member_type: VOTER last_known_addr { host: "127.0.222.3" port: 43891 } } peers { permanent_uuid: "b13c52d3d307484390935460b384345d" member_type: VOTER last_known_addr { host: "127.0.222.2" port: 33317 } } peers { permanent_uuid: "0d897ca0b7594fb6952f1d1f90301da6" member_type: VOTER last_known_addr { host: "127.0.222.1" port: 38517 } }
I20260809 12:13:13.175746 15343 raft_consensus.cc:374] T 2838dc8950034297ad4c3e10ca27bcaa P 0d897ca0b7594fb6952f1d1f90301da6 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260809 12:13:13.175902 15343 raft_consensus.cc:729] T 2838dc8950034297ad4c3e10ca27bcaa P 0d897ca0b7594fb6952f1d1f90301da6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0d897ca0b7594fb6952f1d1f90301da6, State: Initialized, Role: FOLLOWER
I20260809 12:13:13.176602 15343 consensus_queue.cc:260] T 2838dc8950034297ad4c3e10ca27bcaa P 0d897ca0b7594fb6952f1d1f90301da6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "22bd9601d42e40a880eec51934dfeb23" member_type: VOTER last_known_addr { host: "127.0.222.3" port: 43891 } } peers { permanent_uuid: "b13c52d3d307484390935460b384345d" member_type: VOTER last_known_addr { host: "127.0.222.2" port: 33317 } } peers { permanent_uuid: "0d897ca0b7594fb6952f1d1f90301da6" member_type: VOTER last_known_addr { host: "127.0.222.1" port: 38517 } }
I20260809 12:13:13.179768 15013 heartbeater.cc:503] Master 127.0.222.62:37183 was elected leader, sending a full tablet report...
I20260809 12:13:13.181734 15343 ts_tablet_manager.cc:1428] T 2838dc8950034297ad4c3e10ca27bcaa P 0d897ca0b7594fb6952f1d1f90301da6: Time spent starting tablet: real 0.024s user 0.016s sys 0.000s
I20260809 12:13:13.183032 15170 heartbeater.cc:503] Master 127.0.222.62:37183 was elected leader, sending a full tablet report...
I20260809 12:13:13.183656 15342 ts_tablet_manager.cc:1428] T 2838dc8950034297ad4c3e10ca27bcaa P b13c52d3d307484390935460b384345d: Time spent starting tablet: real 0.027s user 0.016s sys 0.008s
W20260809 12:13:13.212013 15319 tablet.cc:2369] T 2838dc8950034297ad4c3e10ca27bcaa P 22bd9601d42e40a880eec51934dfeb23: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260809 12:13:13.298312 15347 raft_consensus.cc:482] T 2838dc8950034297ad4c3e10ca27bcaa P 22bd9601d42e40a880eec51934dfeb23 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260809 12:13:13.298645 15347 raft_consensus.cc:504] T 2838dc8950034297ad4c3e10ca27bcaa P 22bd9601d42e40a880eec51934dfeb23 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "22bd9601d42e40a880eec51934dfeb23" member_type: VOTER last_known_addr { host: "127.0.222.3" port: 43891 } } peers { permanent_uuid: "b13c52d3d307484390935460b384345d" member_type: VOTER last_known_addr { host: "127.0.222.2" port: 33317 } } peers { permanent_uuid: "0d897ca0b7594fb6952f1d1f90301da6" member_type: VOTER last_known_addr { host: "127.0.222.1" port: 38517 } }
I20260809 12:13:13.300781 15347 leader_election.cc:288] T 2838dc8950034297ad4c3e10ca27bcaa P 22bd9601d42e40a880eec51934dfeb23 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers b13c52d3d307484390935460b384345d (127.0.222.2:33317), 0d897ca0b7594fb6952f1d1f90301da6 (127.0.222.1:38517)
I20260809 12:13:13.308362 15338 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 12:13:13.301775 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.222.3:47731 (local address 127.0.222.1:38517)
0809 12:13:13.302353 (+ 578us) server_negotiation.cc:206] Beginning negotiation
0809 12:13:13.302360 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0809 12:13:13.302381 (+ 21us) server_negotiation.cc:407] Connection header received
0809 12:13:13.302582 (+ 201us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 12:13:13.302587 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 12:13:13.302670 (+ 83us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 12:13:13.302773 (+ 103us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0809 12:13:13.303360 (+ 587us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 12:13:13.304356 (+ 996us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 12:13:13.306120 (+ 1764us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 12:13:13.307362 (+ 1242us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 12:13:13.307507 (+ 145us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 12:13:13.307620 (+ 113us) server_negotiation.cc:1035] Waiting for connection context
0809 12:13:13.307959 (+ 339us) server_negotiation.cc:299] Negotiation successful
0809 12:13:13.308097 (+ 138us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":402}
I20260809 12:13:13.308573 15340 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 12:13:13.301701 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.222.3:42855 (local address 127.0.222.2:33317)
0809 12:13:13.302014 (+ 313us) server_negotiation.cc:206] Beginning negotiation
0809 12:13:13.302019 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0809 12:13:13.302041 (+ 22us) server_negotiation.cc:407] Connection header received
0809 12:13:13.302123 (+ 82us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 12:13:13.302127 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 12:13:13.302209 (+ 82us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 12:13:13.302309 (+ 100us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0809 12:13:13.302903 (+ 594us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 12:13:13.304064 (+ 1161us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 12:13:13.306415 (+ 2351us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 12:13:13.307694 (+ 1279us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 12:13:13.307830 (+ 136us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 12:13:13.307931 (+ 101us) server_negotiation.cc:1035] Waiting for connection context
0809 12:13:13.308192 (+ 261us) server_negotiation.cc:299] Negotiation successful
0809 12:13:13.308328 (+ 136us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":79}
I20260809 12:13:13.308573 15321 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 12:13:13.301071 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.222.2:33317 (local address 127.0.222.3:42855)
0809 12:13:13.301314 (+ 243us) negotiation.cc:107] Waiting for socket to connect
0809 12:13:13.301336 (+ 22us) client_negotiation.cc:174] Beginning negotiation
0809 12:13:13.301475 (+ 139us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 12:13:13.302479 (+ 1004us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 12:13:13.302482 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 12:13:13.302526 (+ 44us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0809 12:13:13.302720 (+ 194us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 12:13:13.302736 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 12:13:13.304227 (+ 1491us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 12:13:13.304232 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 12:13:13.306223 (+ 1991us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 12:13:13.306238 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 12:13:13.307850 (+ 1612us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 12:13:13.307857 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 12:13:13.308033 (+ 176us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 12:13:13.308058 (+ 25us) client_negotiation.cc:769] Sending connection context
0809 12:13:13.308162 (+ 104us) client_negotiation.cc:240] Negotiation successful
0809 12:13:13.308306 (+ 144us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":84}
I20260809 12:13:13.309264 15350 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 12:13:13.301166 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.222.1:38517 (local address 127.0.222.3:47731)
0809 12:13:13.302244 (+ 1078us) negotiation.cc:107] Waiting for socket to connect
0809 12:13:13.302261 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0809 12:13:13.302386 (+ 125us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 12:13:13.302805 (+ 419us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 12:13:13.302810 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 12:13:13.302835 (+ 25us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0809 12:13:13.303191 (+ 356us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 12:13:13.303203 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 12:13:13.304516 (+ 1313us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 12:13:13.304520 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 12:13:13.305941 (+ 1421us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 12:13:13.305953 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 12:13:13.307583 (+ 1630us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 12:13:13.307588 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 12:13:13.307767 (+ 179us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 12:13:13.307800 (+ 33us) client_negotiation.cc:769] Sending connection context
0809 12:13:13.308789 (+ 989us) client_negotiation.cc:240] Negotiation successful
0809 12:13:13.308937 (+ 148us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":971,"thread_start_us":797,"threads_started":1}
I20260809 12:13:13.309799 15119 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2838dc8950034297ad4c3e10ca27bcaa" candidate_uuid: "22bd9601d42e40a880eec51934dfeb23" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b13c52d3d307484390935460b384345d" is_pre_election: true
I20260809 12:13:13.310251 14965 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2838dc8950034297ad4c3e10ca27bcaa" candidate_uuid: "22bd9601d42e40a880eec51934dfeb23" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0d897ca0b7594fb6952f1d1f90301da6" is_pre_election: true
I20260809 12:13:13.310413 15119 raft_consensus.cc:2454] T 2838dc8950034297ad4c3e10ca27bcaa P b13c52d3d307484390935460b384345d [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 22bd9601d42e40a880eec51934dfeb23 in term 0.
I20260809 12:13:13.310755 14965 raft_consensus.cc:2454] T 2838dc8950034297ad4c3e10ca27bcaa P 0d897ca0b7594fb6952f1d1f90301da6 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 22bd9601d42e40a880eec51934dfeb23 in term 0.
I20260809 12:13:13.311200 15193 leader_election.cc:302] T 2838dc8950034297ad4c3e10ca27bcaa P 22bd9601d42e40a880eec51934dfeb23 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 22bd9601d42e40a880eec51934dfeb23, b13c52d3d307484390935460b384345d; no voters:
I20260809 12:13:13.311635 15347 raft_consensus.cc:2781] T 2838dc8950034297ad4c3e10ca27bcaa P 22bd9601d42e40a880eec51934dfeb23 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260809 12:13:13.311794 15347 raft_consensus.cc:482] T 2838dc8950034297ad4c3e10ca27bcaa P 22bd9601d42e40a880eec51934dfeb23 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260809 12:13:13.311952 15347 raft_consensus.cc:3037] T 2838dc8950034297ad4c3e10ca27bcaa P 22bd9601d42e40a880eec51934dfeb23 [term 0 FOLLOWER]: Advancing to term 1
I20260809 12:13:13.314118 15347 raft_consensus.cc:504] T 2838dc8950034297ad4c3e10ca27bcaa P 22bd9601d42e40a880eec51934dfeb23 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "22bd9601d42e40a880eec51934dfeb23" member_type: VOTER last_known_addr { host: "127.0.222.3" port: 43891 } } peers { permanent_uuid: "b13c52d3d307484390935460b384345d" member_type: VOTER last_known_addr { host: "127.0.222.2" port: 33317 } } peers { permanent_uuid: "0d897ca0b7594fb6952f1d1f90301da6" member_type: VOTER last_known_addr { host: "127.0.222.1" port: 38517 } }
I20260809 12:13:13.314726 15347 leader_election.cc:288] T 2838dc8950034297ad4c3e10ca27bcaa P 22bd9601d42e40a880eec51934dfeb23 [CANDIDATE]: Term 1 election: Requested vote from peers b13c52d3d307484390935460b384345d (127.0.222.2:33317), 0d897ca0b7594fb6952f1d1f90301da6 (127.0.222.1:38517)
I20260809 12:13:13.315213 15119 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2838dc8950034297ad4c3e10ca27bcaa" candidate_uuid: "22bd9601d42e40a880eec51934dfeb23" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b13c52d3d307484390935460b384345d"
I20260809 12:13:13.315236 14965 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2838dc8950034297ad4c3e10ca27bcaa" candidate_uuid: "22bd9601d42e40a880eec51934dfeb23" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0d897ca0b7594fb6952f1d1f90301da6"
I20260809 12:13:13.315455 14965 raft_consensus.cc:3037] T 2838dc8950034297ad4c3e10ca27bcaa P 0d897ca0b7594fb6952f1d1f90301da6 [term 0 FOLLOWER]: Advancing to term 1
I20260809 12:13:13.315443 15119 raft_consensus.cc:3037] T 2838dc8950034297ad4c3e10ca27bcaa P b13c52d3d307484390935460b384345d [term 0 FOLLOWER]: Advancing to term 1
I20260809 12:13:13.318815 15119 raft_consensus.cc:2454] T 2838dc8950034297ad4c3e10ca27bcaa P b13c52d3d307484390935460b384345d [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 22bd9601d42e40a880eec51934dfeb23 in term 1.
I20260809 12:13:13.318804 14965 raft_consensus.cc:2454] T 2838dc8950034297ad4c3e10ca27bcaa P 0d897ca0b7594fb6952f1d1f90301da6 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 22bd9601d42e40a880eec51934dfeb23 in term 1.
I20260809 12:13:13.319263 15196 leader_election.cc:302] T 2838dc8950034297ad4c3e10ca27bcaa P 22bd9601d42e40a880eec51934dfeb23 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0d897ca0b7594fb6952f1d1f90301da6, 22bd9601d42e40a880eec51934dfeb23; no voters:
I20260809 12:13:13.319610 15347 raft_consensus.cc:2781] T 2838dc8950034297ad4c3e10ca27bcaa P 22bd9601d42e40a880eec51934dfeb23 [term 1 FOLLOWER]: Leader election won for term 1
I20260809 12:13:13.320101 15347 raft_consensus.cc:686] T 2838dc8950034297ad4c3e10ca27bcaa P 22bd9601d42e40a880eec51934dfeb23 [term 1 LEADER]: Becoming Leader. State: Replica: 22bd9601d42e40a880eec51934dfeb23, State: Running, Role: LEADER
I20260809 12:13:13.320643 15347 consensus_queue.cc:237] T 2838dc8950034297ad4c3e10ca27bcaa P 22bd9601d42e40a880eec51934dfeb23 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "22bd9601d42e40a880eec51934dfeb23" member_type: VOTER last_known_addr { host: "127.0.222.3" port: 43891 } } peers { permanent_uuid: "b13c52d3d307484390935460b384345d" member_type: VOTER last_known_addr { host: "127.0.222.2" port: 33317 } } peers { permanent_uuid: "0d897ca0b7594fb6952f1d1f90301da6" member_type: VOTER last_known_addr { host: "127.0.222.1" port: 38517 } }
I20260809 12:13:13.326190 14802 catalog_manager.cc:5495] T 2838dc8950034297ad4c3e10ca27bcaa P 22bd9601d42e40a880eec51934dfeb23 reported cstate change: term changed from 0 to 1, leader changed from <none> to 22bd9601d42e40a880eec51934dfeb23 (127.0.222.3). New cstate: current_term: 1 leader_uuid: "22bd9601d42e40a880eec51934dfeb23" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "22bd9601d42e40a880eec51934dfeb23" member_type: VOTER last_known_addr { host: "127.0.222.3" port: 43891 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "b13c52d3d307484390935460b384345d" member_type: VOTER last_known_addr { host: "127.0.222.2" port: 33317 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "0d897ca0b7594fb6952f1d1f90301da6" member_type: VOTER last_known_addr { host: "127.0.222.1" port: 38517 } health_report { overall_health: UNKNOWN } } }
I20260809 12:13:13.358043 15335 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 12:13:13.352637 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35510 (local address 127.0.222.3:43891)
0809 12:13:13.352873 (+ 236us) server_negotiation.cc:206] Beginning negotiation
0809 12:13:13.352881 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0809 12:13:13.352900 (+ 19us) server_negotiation.cc:407] Connection header received
0809 12:13:13.353000 (+ 100us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 12:13:13.353003 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 12:13:13.353078 (+ 75us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 12:13:13.353237 (+ 159us) server_negotiation.cc:226] Negotiated authn=TOKEN
0809 12:13:13.353648 (+ 411us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 12:13:13.354382 (+ 734us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 12:13:13.355862 (+ 1480us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 12:13:13.356281 (+ 419us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 12:13:13.356431 (+ 150us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 12:13:13.356942 (+ 511us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0809 12:13:13.357458 (+ 516us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0809 12:13:13.357633 (+ 175us) server_negotiation.cc:1035] Waiting for connection context
0809 12:13:13.357692 (+ 59us) server_negotiation.cc:299] Negotiation successful
0809 12:13:13.357775 (+ 83us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":67}
I20260809 12:13:13.366524 14965 raft_consensus.cc:1261] T 2838dc8950034297ad4c3e10ca27bcaa P 0d897ca0b7594fb6952f1d1f90301da6 [term 1 FOLLOWER]: Refusing update from remote peer 22bd9601d42e40a880eec51934dfeb23: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260809 12:13:13.366524 15119 raft_consensus.cc:1261] T 2838dc8950034297ad4c3e10ca27bcaa P b13c52d3d307484390935460b384345d [term 1 FOLLOWER]: Refusing update from remote peer 22bd9601d42e40a880eec51934dfeb23: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260809 12:13:13.367588 15351 consensus_queue.cc:1035] T 2838dc8950034297ad4c3e10ca27bcaa P 22bd9601d42e40a880eec51934dfeb23 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b13c52d3d307484390935460b384345d" member_type: VOTER last_known_addr { host: "127.0.222.2" port: 33317 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260809 12:13:13.367930 15347 consensus_queue.cc:1035] T 2838dc8950034297ad4c3e10ca27bcaa P 22bd9601d42e40a880eec51934dfeb23 [LEADER]: Connected to new peer: Peer: permanent_uuid: "0d897ca0b7594fb6952f1d1f90301da6" member_type: VOTER last_known_addr { host: "127.0.222.1" port: 38517 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260809 12:13:13.380411 15354 mvcc.cc:205] Tried to move back new op lower bound from 7316593022418677760 to 7316593022246109184. Current Snapshot: MvccSnapshot[applied={T|T < 7316593022418677760}]
I20260809 12:13:13.382477 15355 mvcc.cc:205] Tried to move back new op lower bound from 7316593022418677760 to 7316593022246109184. Current Snapshot: MvccSnapshot[applied={T|T < 7316593022418677760}]
I20260809 12:13:13.393545 14801 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:52876:
table { table_name: "test-table" } modify_external_catalogs: true
I20260809 12:13:13.393913 14801 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:52876:
table { table_name: "test-table" } modify_external_catalogs: true
I20260809 12:13:13.398237 14801 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 8751c8f15ead4a2a96a78af1bd301cc9: Sending DeleteTablet for 3 replicas of tablet 2838dc8950034297ad4c3e10ca27bcaa
I20260809 12:13:13.399761 15099 tablet_service.cc:1514] Processing DeleteTablet for tablet 2838dc8950034297ad4c3e10ca27bcaa with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-09 12:13:13 UTC) from {username='kudu', principal='kudu/127.0.222.62@KRBTEST.COM'} at 127.0.0.1:40466
I20260809 12:13:13.399806 15243 tablet_service.cc:1514] Processing DeleteTablet for tablet 2838dc8950034297ad4c3e10ca27bcaa with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-09 12:13:13 UTC) from {username='kudu', principal='kudu/127.0.222.62@KRBTEST.COM'} at 127.0.0.1:35498
I20260809 12:13:13.400695 14944 tablet_service.cc:1514] Processing DeleteTablet for tablet 2838dc8950034297ad4c3e10ca27bcaa with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-09 12:13:13 UTC) from {username='kudu', principal='kudu/127.0.222.62@KRBTEST.COM'} at 127.0.0.1:59352
I20260809 12:13:13.409938 888 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskgH5Pjm/build/asan/bin/kudu pid 14842
I20260809 12:13:13.429152 15366 tablet_replica.cc:330] T 2838dc8950034297ad4c3e10ca27bcaa P 0d897ca0b7594fb6952f1d1f90301da6: stopping tablet replica
I20260809 12:13:13.429915 15365 tablet_replica.cc:330] T 2838dc8950034297ad4c3e10ca27bcaa P 22bd9601d42e40a880eec51934dfeb23: stopping tablet replica
I20260809 12:13:13.430541 15366 raft_consensus.cc:2229] T 2838dc8950034297ad4c3e10ca27bcaa P 0d897ca0b7594fb6952f1d1f90301da6 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260809 12:13:13.431263 15365 raft_consensus.cc:2229] T 2838dc8950034297ad4c3e10ca27bcaa P 22bd9601d42e40a880eec51934dfeb23 [term 1 LEADER]: Raft consensus shutting down.
I20260809 12:13:13.432348 15365 raft_consensus.cc:2258] T 2838dc8950034297ad4c3e10ca27bcaa P 22bd9601d42e40a880eec51934dfeb23 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260809 12:13:13.433355 15366 raft_consensus.cc:2258] T 2838dc8950034297ad4c3e10ca27bcaa P 0d897ca0b7594fb6952f1d1f90301da6 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260809 12:13:13.436636 15366 ts_tablet_manager.cc:1905] T 2838dc8950034297ad4c3e10ca27bcaa P 0d897ca0b7594fb6952f1d1f90301da6: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260809 12:13:13.437620 15338 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 12:13:13.411603 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59354 (local address 127.0.222.1:38517)
0809 12:13:13.411836 (+ 233us) server_negotiation.cc:206] Beginning negotiation
0809 12:13:13.411845 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0809 12:13:13.411874 (+ 29us) server_negotiation.cc:407] Connection header received
0809 12:13:13.411960 (+ 86us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 12:13:13.411968 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 12:13:13.412100 (+ 132us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 12:13:13.412257 (+ 157us) server_negotiation.cc:226] Negotiated authn=SASL
0809 12:13:13.414474 (+ 2217us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 12:13:13.416007 (+ 1533us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 12:13:13.417913 (+ 1906us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 12:13:13.418846 (+ 933us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 12:13:13.419023 (+ 177us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 12:13:13.423270 (+ 4247us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 12:13:13.423306 (+ 36us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 12:13:13.423334 (+ 28us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0809 12:13:13.423394 (+ 60us) server_negotiation.cc:922] calling sasl_server_start()
0809 12:13:13.428990 (+ 5596us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 12:13:13.430330 (+ 1340us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 12:13:13.430344 (+ 14us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 12:13:13.430350 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0809 12:13:13.430583 (+ 233us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 12:13:13.431254 (+ 671us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 12:13:13.431262 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 12:13:13.431267 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0809 12:13:13.431921 (+ 654us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 12:13:13.432326 (+ 405us) server_negotiation.cc:1035] Waiting for connection context
0809 12:13:13.434807 (+ 2481us) server_negotiation.cc:299] Negotiation successful
0809 12:13:13.435226 (+ 419us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":79}
I20260809 12:13:13.438668 15007 generic_service.cc:196] Checking for leaks (request via RPC)
I20260809 12:13:13.439432 15364 tablet_replica.cc:330] T 2838dc8950034297ad4c3e10ca27bcaa P b13c52d3d307484390935460b384345d: stopping tablet replica
I20260809 12:13:13.439858 15364 raft_consensus.cc:2229] T 2838dc8950034297ad4c3e10ca27bcaa P b13c52d3d307484390935460b384345d [term 1 FOLLOWER]: Raft consensus shutting down.
I20260809 12:13:13.440376 15364 raft_consensus.cc:2258] T 2838dc8950034297ad4c3e10ca27bcaa P b13c52d3d307484390935460b384345d [term 1 FOLLOWER]: Raft consensus is shut down!
I20260809 12:13:13.442895 15364 ts_tablet_manager.cc:1905] T 2838dc8950034297ad4c3e10ca27bcaa P b13c52d3d307484390935460b384345d: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260809 12:13:13.444511 15365 ts_tablet_manager.cc:1905] T 2838dc8950034297ad4c3e10ca27bcaa P 22bd9601d42e40a880eec51934dfeb23: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260809 12:13:13.450794 15364 ts_tablet_manager.cc:1918] T 2838dc8950034297ad4c3e10ca27bcaa P b13c52d3d307484390935460b384345d: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260809 12:13:13.451020 15364 log.cc:1196] T 2838dc8950034297ad4c3e10ca27bcaa P b13c52d3d307484390935460b384345d: Deleting WAL directory at /tmp/dist-test-taskgH5Pjm/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786277382073938-888-0/minicluster-data/ts-1/wal/wals/2838dc8950034297ad4c3e10ca27bcaa
I20260809 12:13:13.451643 15364 ts_tablet_manager.cc:1939] T 2838dc8950034297ad4c3e10ca27bcaa P b13c52d3d307484390935460b384345d: Deleting consensus metadata
I20260809 12:13:13.453876 14784 catalog_manager.cc:4841] TS b13c52d3d307484390935460b384345d (127.0.222.2:33317): tablet 2838dc8950034297ad4c3e10ca27bcaa (table test-table [id=32ad5abe0ef341f09cac21b315b9e821]) successfully deleted
I20260809 12:13:13.456796 15365 ts_tablet_manager.cc:1918] T 2838dc8950034297ad4c3e10ca27bcaa P 22bd9601d42e40a880eec51934dfeb23: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260809 12:13:13.457020 15365 log.cc:1196] T 2838dc8950034297ad4c3e10ca27bcaa P 22bd9601d42e40a880eec51934dfeb23: Deleting WAL directory at /tmp/dist-test-taskgH5Pjm/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786277382073938-888-0/minicluster-data/ts-2/wal/wals/2838dc8950034297ad4c3e10ca27bcaa
I20260809 12:13:13.457548 15365 ts_tablet_manager.cc:1939] T 2838dc8950034297ad4c3e10ca27bcaa P 22bd9601d42e40a880eec51934dfeb23: Deleting consensus metadata
I20260809 12:13:13.459195 14784 catalog_manager.cc:4841] TS 22bd9601d42e40a880eec51934dfeb23 (127.0.222.3:43891): tablet 2838dc8950034297ad4c3e10ca27bcaa (table test-table [id=32ad5abe0ef341f09cac21b315b9e821]) successfully deleted
I20260809 12:13:13.547937 888 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskgH5Pjm/build/asan/bin/kudu with pid 14842
I20260809 12:13:13.569188 888 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskgH5Pjm/build/asan/bin/kudu pid 15017
W20260809 12:13:13.569165 14788 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.0.222.1:38517 (error 108)
I20260809 12:13:13.571213 15339 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 12:13:13.570718 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.222.1:38517 (local address 127.0.0.1:59362)
0809 12:13:13.570914 (+ 196us) negotiation.cc:107] Waiting for socket to connect
0809 12:13:13.571073 (+ 159us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.222.1:38517: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":79}
W20260809 12:13:13.571493 14788 catalog_manager.cc:4584] TS 0d897ca0b7594fb6952f1d1f90301da6 (127.0.222.1:38517): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 2838dc8950034297ad4c3e10ca27bcaa: Network error: Client connection negotiation failed: client connection to 127.0.222.1:38517: connect: Connection refused (error 111)
I20260809 12:13:13.585624 15340 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 12:13:13.570376 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40476 (local address 127.0.222.2:33317)
0809 12:13:13.570576 (+ 200us) server_negotiation.cc:206] Beginning negotiation
0809 12:13:13.570581 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0809 12:13:13.570598 (+ 17us) server_negotiation.cc:407] Connection header received
0809 12:13:13.570654 (+ 56us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 12:13:13.570658 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 12:13:13.570717 (+ 59us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 12:13:13.570838 (+ 121us) server_negotiation.cc:226] Negotiated authn=SASL
0809 12:13:13.572777 (+ 1939us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 12:13:13.573499 (+ 722us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 12:13:13.575368 (+ 1869us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 12:13:13.575758 (+ 390us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 12:13:13.575858 (+ 100us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 12:13:13.579320 (+ 3462us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 12:13:13.579333 (+ 13us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 12:13:13.579348 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0809 12:13:13.579387 (+ 39us) server_negotiation.cc:922] calling sasl_server_start()
0809 12:13:13.582293 (+ 2906us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 12:13:13.583353 (+ 1060us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 12:13:13.583362 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 12:13:13.583364 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0809 12:13:13.583545 (+ 181us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 12:13:13.583986 (+ 441us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 12:13:13.583991 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 12:13:13.583994 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0809 12:13:13.584372 (+ 378us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 12:13:13.584644 (+ 272us) server_negotiation.cc:1035] Waiting for connection context
0809 12:13:13.585034 (+ 390us) server_negotiation.cc:299] Negotiation successful
0809 12:13:13.585218 (+ 184us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":95}
I20260809 12:13:13.586429 15163 generic_service.cc:196] Checking for leaks (request via RPC)
I20260809 12:13:13.625239 15339 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 12:13:13.624681 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.222.1:38517 (local address 127.0.0.1:59368)
0809 12:13:13.624937 (+ 256us) negotiation.cc:107] Waiting for socket to connect
0809 12:13:13.625067 (+ 130us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.222.1:38517: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":92}
I20260809 12:13:13.626470 15339 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 12:13:13.626028 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.222.1:38517 (local address 127.0.0.1:59380)
0809 12:13:13.626198 (+ 170us) negotiation.cc:107] Waiting for socket to connect
0809 12:13:13.626290 (+ 92us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.222.1:38517: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":56}
I20260809 12:13:13.686825 888 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskgH5Pjm/build/asan/bin/kudu with pid 15017
I20260809 12:13:13.687536 15339 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 12:13:13.686863 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.222.1:38517 (local address 127.0.0.1:59382)
0809 12:13:13.687211 (+ 348us) negotiation.cc:107] Waiting for socket to connect
0809 12:13:13.687330 (+ 119us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.222.1:38517: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":209}
I20260809 12:13:13.689548 15339 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 12:13:13.689084 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.222.1:38517 (local address 127.0.0.1:59390)
0809 12:13:13.689278 (+ 194us) negotiation.cc:107] Waiting for socket to connect
0809 12:13:13.689389 (+ 111us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.222.1:38517: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":64}
I20260809 12:13:13.708877 888 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskgH5Pjm/build/asan/bin/kudu pid 15174
I20260809 12:13:13.726115 15335 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 12:13:13.709946 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35524 (local address 127.0.222.3:43891)
0809 12:13:13.710150 (+ 204us) server_negotiation.cc:206] Beginning negotiation
0809 12:13:13.710161 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0809 12:13:13.710188 (+ 27us) server_negotiation.cc:407] Connection header received
0809 12:13:13.710348 (+ 160us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 12:13:13.710354 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 12:13:13.710473 (+ 119us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 12:13:13.710619 (+ 146us) server_negotiation.cc:226] Negotiated authn=SASL
0809 12:13:13.712038 (+ 1419us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 12:13:13.713324 (+ 1286us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 12:13:13.714638 (+ 1314us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 12:13:13.715466 (+ 828us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 12:13:13.715629 (+ 163us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 12:13:13.718342 (+ 2713us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 12:13:13.718359 (+ 17us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 12:13:13.718387 (+ 28us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0809 12:13:13.718444 (+ 57us) server_negotiation.cc:922] calling sasl_server_start()
0809 12:13:13.722624 (+ 4180us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 12:13:13.723711 (+ 1087us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 12:13:13.723727 (+ 16us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 12:13:13.723730 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0809 12:13:13.723979 (+ 249us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 12:13:13.724389 (+ 410us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 12:13:13.724394 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 12:13:13.724397 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0809 12:13:13.724883 (+ 486us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 12:13:13.725230 (+ 347us) server_negotiation.cc:1035] Waiting for connection context
0809 12:13:13.725406 (+ 176us) server_negotiation.cc:299] Negotiation successful
0809 12:13:13.725667 (+ 261us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":78}
I20260809 12:13:13.727031 15312 generic_service.cc:196] Checking for leaks (request via RPC)
I20260809 12:13:13.770455 15339 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 12:13:13.769891 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.222.1:38517 (local address 127.0.0.1:59406)
0809 12:13:13.770128 (+ 237us) negotiation.cc:107] Waiting for socket to connect
0809 12:13:13.770242 (+ 114us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.222.1:38517: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":97}
I20260809 12:13:13.771703 15339 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 12:13:13.771297 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.222.1:38517 (local address 127.0.0.1:59410)
0809 12:13:13.771513 (+ 216us) negotiation.cc:107] Waiting for socket to connect
0809 12:13:13.771594 (+ 81us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.222.1:38517: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":102}
I20260809 12:13:13.854466 888 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskgH5Pjm/build/asan/bin/kudu with pid 15174
I20260809 12:13:13.878067 888 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskgH5Pjm/build/asan/bin/kudu pid 14766
I20260809 12:13:13.879405 14835 generic_service.cc:196] Checking for leaks (request via RPC)
I20260809 12:13:13.998047 15339 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 12:13:13.997183 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.222.1:38517 (local address 127.0.0.1:59424)
0809 12:13:13.997400 (+ 217us) negotiation.cc:107] Waiting for socket to connect
0809 12:13:13.997565 (+ 165us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.222.1:38517: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":78}
I20260809 12:13:13.998391 888 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskgH5Pjm/build/asan/bin/kudu with pid 14766
I20260809 12:13:14.025928 888 mini_ranger_kms.cc:60] Stopping Ranger KMS...
I20260809 12:13:14.371547 888 mini_ranger_kms.cc:62] Stopped Ranger KMS
I20260809 12:13:14.371799 888 mini_ranger.cc:66] Stopping Ranger...
I20260809 12:13:14.718344 888 mini_ranger.cc:68] Stopped Ranger
2026-08-09 12:13:14.718 UTC [13775] LOG: received smart shutdown request
2026-08-09 12:13:14.722 UTC [13775] LOG: background worker "logical replication launcher" (PID 13784) exited with exit code 1
2026-08-09 12:13:14.722 UTC [13779] LOG: shutting down
2026-08-09 12:13:14.908 UTC [13775] LOG: database system is shut down
2026-08-09T12:13:14Z chronyd exiting
[ OK ] SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers (54326 ms)
[ RUN ] SecurityITest.IPKICACert
Loading random data
Initializing database '/tmp/dist-test-taskgH5Pjm/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786277382073938-888-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 09 12:13:15 dist-test-slave-2w3w krb5kdc[15382](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 09 12:13:15 dist-test-slave-2w3w krb5kdc[15382](info): set up 2 sockets
Aug 09 12:13:15 dist-test-slave-2w3w krb5kdc[15382](info): commencing operation
krb5kdc: starting...
W20260809 12:13:16.217034 888 mini_kdc.cc:121] Time spent starting KDC: real 1.129s user 0.000s sys 0.037s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskgH5Pjm/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786277382073938-888-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskgH5Pjm/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786277382073938-888-0/krb5kdc/test-user.keytab.
Aug 09 12:13:16 dist-test-slave-2w3w krb5kdc[15382](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786277596, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-09T12:13:16Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-09T12:13:16Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.222.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.222.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.222.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskgH5Pjm/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786277382073938-888-0/krb5kdc/kudu_127.0.222.62.keytab.
Entry for principal kudu/127.0.222.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskgH5Pjm/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786277382073938-888-0/krb5kdc/kudu_127.0.222.62.keytab.
I20260809 12:13:16.515103 888 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskgH5Pjm/build/asan/bin/kudu
/tmp/dist-test-taskgH5Pjm/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskgH5Pjm/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786277382073938-888-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskgH5Pjm/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786277382073938-888-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskgH5Pjm/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786277382073938-888-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskgH5Pjm/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786277382073938-888-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.222.62:36165
--webserver_interface=127.0.222.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskgH5Pjm/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786277382073938-888-0/krb5kdc/kudu_127.0.222.62.keytab
--principal=kudu/127.0.222.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.222.20:46875
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.222.62:36165,127.0.222.61:35925,127.0.222.60:45255
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskgH5Pjm/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786277382073938-888-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskgH5Pjm/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786277382073938-888-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskgH5Pjm/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786277382073938-888-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260809 12:13:16.858079 15396 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260809 12:13:16.858475 15396 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260809 12:13:16.858655 15396 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260809 12:13:16.866081 15396 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260809 12:13:16.866248 15396 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260809 12:13:16.866355 15396 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260809 12:13:16.866464 15396 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260809 12:13:16.866555 15396 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260809 12:13:16.876746 15396 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.222.20:46875
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskgH5Pjm/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786277382073938-888-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskgH5Pjm/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786277382073938-888-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.222.62:36165,127.0.222.61:35925,127.0.222.60:45255
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskgH5Pjm/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786277382073938-888-0/krb5kdc/kudu_127.0.222.62.keytab
--principal=kudu/127.0.222.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.222.62:36165
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskgH5Pjm/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786277382073938-888-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.222.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskgH5Pjm/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786277382073938-888-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 09 Aug 2026 11:06:41 UTC on ccycloud.quasar-bwoogy.root.comops.site
build id 4575991
ASAN enabled
I20260809 12:13:16.878216 15396 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260809 12:13:16.879628 15396 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260809 12:13:16.891824 15401 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 12:13:16.891853 15404 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 12:13:16.891876 15402 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260809 12:13:16.893204 15396 server_base.cc:1034] running on GCE node
I20260809 12:13:16.894167 15396 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260809 12:13:16.895700 15396 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260809 12:13:16.896948 15396 hybrid_clock.cc:648] HybridClock initialized: now 1786277596896903 us; error 42 us; skew 500 ppm
I20260809 12:13:16.897315 15396 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 09 12:13:16 dist-test-slave-2w3w krb5kdc[15382](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786277596, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.222.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260809 12:13:16.900743 15396 init.cc:377] Logged in from keytab as kudu/127.0.222.62@KRBTEST.COM (short username kudu)
I20260809 12:13:16.902848 15396 webserver.cc:460] Webserver started at http://127.0.222.62:46857/ using document root <none> and password file <none>
I20260809 12:13:16.903791 15396 fs_manager.cc:359] Metadata directory not provided
I20260809 12:13:16.903934 15396 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260809 12:13:16.904212 15396 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260809 12:13:16.907356 15396 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskgH5Pjm/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786277382073938-888-0/minicluster-data/master-0/data/instance:
uuid: "962f780b4e6e42209bae1266526c1e08"
format_stamp: "Formatted at 2026-08-09 12:13:16 on dist-test-slave-2w3w"
server_key: "77d9cd75c7f866afedf3d8d578d8d822"
server_key_iv: "85d58eb0daa8a2abe68e4e1b73a1d7a6"
server_key_version: "encryptionkey@0"
I20260809 12:13:16.908123 15396 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskgH5Pjm/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786277382073938-888-0/minicluster-data/master-0/wal/instance:
uuid: "962f780b4e6e42209bae1266526c1e08"
format_stamp: "Formatted at 2026-08-09 12:13:16 on dist-test-slave-2w3w"
server_key: "77d9cd75c7f866afedf3d8d578d8d822"
server_key_iv: "85d58eb0daa8a2abe68e4e1b73a1d7a6"
server_key_version: "encryptionkey@0"
I20260809 12:13:16.913264 15396 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.000s
I20260809 12:13:16.916661 15412 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260809 12:13:16.918005 15396 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.000s
I20260809 12:13:16.918190 15396 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskgH5Pjm/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786277382073938-888-0/minicluster-data/master-0/data,/tmp/dist-test-taskgH5Pjm/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786277382073938-888-0/minicluster-data/master-0/wal
uuid: "962f780b4e6e42209bae1266526c1e08"
format_stamp: "Formatted at 2026-08-09 12:13:16 on dist-test-slave-2w3w"
server_key: "77d9cd75c7f866afedf3d8d578d8d822"
server_key_iv: "85d58eb0daa8a2abe68e4e1b73a1d7a6"
server_key_version: "encryptionkey@0"
I20260809 12:13:16.918393 15396 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskgH5Pjm/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786277382073938-888-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskgH5Pjm/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786277382073938-888-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskgH5Pjm/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786277382073938-888-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260809 12:13:16.953339 15396 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260809 12:13:16.958024 15396 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260809 12:13:16.958348 15396 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260809 12:13:16.977597 15396 rpc_server.cc:307] RPC server started. Bound to: 127.0.222.62:36165
I20260809 12:13:16.977633 15467 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.222.62:36165 every 8 connection(s)
I20260809 12:13:16.979110 15396 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskgH5Pjm/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786277382073938-888-0/minicluster-data/master-0/data/info.pb
I20260809 12:13:16.982784 15468 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260809 12:13:16.984738 888 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskgH5Pjm/build/asan/bin/kudu as pid 15396
I20260809 12:13:16.984989 888 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskgH5Pjm/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786277382073938-888-0/minicluster-data/master-0/wal/instance
I20260809 12:13:16.985842 888 external_mini_cluster.cc:1428] Setting key 5df3e75fedd24c85c7d9f2ff52f2f208
I20260809 12:13:16.992555 15468 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.222.62" port: 36165 } has no permanent_uuid. Determining permanent_uuid...
Aug 09 12:13:17 dist-test-slave-2w3w krb5kdc[15382](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786277596, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.222.62@KRBTEST.COM for kudu/127.0.222.62@KRBTEST.COM
WARNING: no policy specified for kudu/127.0.222.61@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.222.61@KRBTEST.COM" created.
I20260809 12:13:17.017344 15472 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 12:13:16.997816 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40170 (local address 127.0.222.62:36165)
0809 12:13:16.998533 (+ 717us) server_negotiation.cc:206] Beginning negotiation
0809 12:13:16.998545 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0809 12:13:16.998598 (+ 53us) server_negotiation.cc:407] Connection header received
0809 12:13:16.999992 (+ 1394us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 12:13:16.999998 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 12:13:17.000219 (+ 221us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 12:13:17.000427 (+ 208us) server_negotiation.cc:226] Negotiated authn=SASL
0809 12:13:17.001913 (+ 1486us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 12:13:17.003209 (+ 1296us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 12:13:17.004695 (+ 1486us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 12:13:17.005270 (+ 575us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 12:13:17.005496 (+ 226us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 12:13:17.009536 (+ 4040us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 12:13:17.009545 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 12:13:17.009559 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0809 12:13:17.009590 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0809 12:13:17.012556 (+ 2966us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 12:13:17.013289 (+ 733us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 12:13:17.013296 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 12:13:17.013299 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0809 12:13:17.013499 (+ 200us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 12:13:17.013922 (+ 423us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 12:13:17.013930 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 12:13:17.013933 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0809 12:13:17.014477 (+ 544us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 12:13:17.014698 (+ 221us) server_negotiation.cc:1035] Waiting for connection context
0809 12:13:17.015365 (+ 667us) server_negotiation.cc:299] Negotiation successful
0809 12:13:17.015827 (+ 462us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":60,"server-negotiator.queue_time_us":505,"thread_start_us":311,"threads_started":1}
I20260809 12:13:17.017364 15470 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 12:13:16.996532 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.222.62:36165 (local address 127.0.0.1:40170)
0809 12:13:16.997465 (+ 933us) negotiation.cc:107] Waiting for socket to connect
0809 12:13:16.997527 (+ 62us) client_negotiation.cc:174] Beginning negotiation
0809 12:13:16.999619 (+ 2092us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 12:13:17.000452 (+ 833us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 12:13:17.000459 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 12:13:17.001327 (+ 868us) client_negotiation.cc:189] Negotiated authn=SASL
0809 12:13:17.001681 (+ 354us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 12:13:17.001699 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 12:13:17.003368 (+ 1669us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 12:13:17.003375 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 12:13:17.004553 (+ 1178us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 12:13:17.004568 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 12:13:17.005531 (+ 963us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 12:13:17.005538 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 12:13:17.005721 (+ 183us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 12:13:17.006451 (+ 730us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0809 12:13:17.006477 (+ 26us) client_negotiation.cc:656] calling sasl_client_start()
0809 12:13:17.009242 (+ 2765us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0809 12:13:17.012703 (+ 3461us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0809 12:13:17.012718 (+ 15us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0809 12:13:17.012731 (+ 13us) client_negotiation.cc:760] calling sasl_client_step()
0809 12:13:17.013134 (+ 403us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0809 12:13:17.013620 (+ 486us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0809 12:13:17.013626 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0809 12:13:17.013633 (+ 7us) client_negotiation.cc:760] calling sasl_client_step()
0809 12:13:17.013805 (+ 172us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0809 12:13:17.014653 (+ 848us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0809 12:13:17.014663 (+ 10us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0809 12:13:17.014931 (+ 268us) client_negotiation.cc:769] Sending connection context
0809 12:13:17.015264 (+ 333us) client_negotiation.cc:240] Negotiation successful
0809 12:13:17.015963 (+ 699us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":599,"mutex_wait_us":169,"thread_start_us":384,"threads_started":1}
I20260809 12:13:17.021345 15468 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.222.61" port: 35925 } has no permanent_uuid. Determining permanent_uuid...
I20260809 12:13:17.022598 15470 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 12:13:17.022154 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.222.61:35925 (local address 127.0.0.1:48180)
0809 12:13:17.022356 (+ 202us) negotiation.cc:107] Waiting for socket to connect
0809 12:13:17.022478 (+ 122us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.222.61:35925: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":66}
W20260809 12:13:17.022912 15416 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.222.61:35925: connect: Connection refused (error 111)
I20260809 12:13:17.024603 15470 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 12:13:17.024252 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.222.61:35925 (local address 127.0.0.1:48196)
0809 12:13:17.024412 (+ 160us) negotiation.cc:107] Waiting for socket to connect
0809 12:13:17.024476 (+ 64us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.222.61:35925: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":52}
W20260809 12:13:17.024791 15468 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.222.61:35925: Network error: Client connection negotiation failed: client connection to 127.0.222.61:35925: connect: Connection refused (error 111)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.222.61 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskgH5Pjm/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786277382073938-888-0/krb5kdc/kudu_127.0.222.61.keytab.
Entry for principal kudu/127.0.222.61 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskgH5Pjm/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786277382073938-888-0/krb5kdc/kudu_127.0.222.61.keytab.
I20260809 12:13:17.050707 888 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskgH5Pjm/build/asan/bin/kudu
/tmp/dist-test-taskgH5Pjm/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskgH5Pjm/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786277382073938-888-0/minicluster-data/master-1/wal
--fs_data_dirs=/tmp/dist-test-taskgH5Pjm/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786277382073938-888-0/minicluster-data/master-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskgH5Pjm/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786277382073938-888-0/minicluster-data/master-1/logs
--server_dump_info_path=/tmp/dist-test-taskgH5Pjm/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786277382073938-888-0/minicluster-data/master-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.222.61:35925
--webserver_interface=127.0.222.61
--webserver_port=0
--keytab_file=/tmp/dist-test-taskgH5Pjm/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786277382073938-888-0/krb5kdc/kudu_127.0.222.61.keytab
--principal=kudu/127.0.222.61
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.222.20:46875
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.222.62:36165,127.0.222.61:35925,127.0.222.60:45255
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskgH5Pjm/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786277382073938-888-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskgH5Pjm/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786277382073938-888-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskgH5Pjm/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786277382073938-888-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260809 12:13:17.074193 15468 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.222.61" port: 35925 } attempt: 1
I20260809 12:13:17.075523 15470 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 12:13:17.075042 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.222.61:35925 (local address 127.0.0.1:48212)
0809 12:13:17.075256 (+ 214us) negotiation.cc:107] Waiting for socket to connect
0809 12:13:17.075363 (+ 107us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.222.61:35925: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":79}
I20260809 12:13:17.076556 15470 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 12:13:17.076173 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.222.61:35925 (local address 127.0.0.1:48216)
0809 12:13:17.076342 (+ 169us) negotiation.cc:107] Waiting for socket to connect
0809 12:13:17.076413 (+ 71us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.222.61:35925: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":51}
W20260809 12:13:17.076752 15468 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.222.61:35925: Network error: Client connection negotiation failed: client connection to 127.0.222.61:35925: connect: Connection refused (error 111)
I20260809 12:13:17.145004 15468 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.222.61" port: 35925 } attempt: 2
I20260809 12:13:17.146199 15470 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 12:13:17.145682 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.222.61:35925 (local address 127.0.0.1:48224)
0809 12:13:17.145947 (+ 265us) negotiation.cc:107] Waiting for socket to connect
0809 12:13:17.146052 (+ 105us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.222.61:35925: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":75}
I20260809 12:13:17.147250 15470 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 12:13:17.146841 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.222.61:35925 (local address 127.0.0.1:48240)
0809 12:13:17.147030 (+ 189us) negotiation.cc:107] Waiting for socket to connect
0809 12:13:17.147110 (+ 80us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.222.61:35925: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":50}
W20260809 12:13:17.147428 15468 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.222.61:35925: Network error: Client connection negotiation failed: client connection to 127.0.222.61:35925: connect: Connection refused (error 111)
I20260809 12:13:17.238698 15468 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.222.61" port: 35925 } attempt: 3
I20260809 12:13:17.239915 15470 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 12:13:17.239446 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.222.61:35925 (local address 127.0.0.1:48246)
0809 12:13:17.239662 (+ 216us) negotiation.cc:107] Waiting for socket to connect
0809 12:13:17.239776 (+ 114us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.222.61:35925: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":80}
I20260809 12:13:17.240901 15470 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 12:13:17.240551 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.222.61:35925 (local address 127.0.0.1:48256)
0809 12:13:17.240695 (+ 144us) negotiation.cc:107] Waiting for socket to connect
0809 12:13:17.240767 (+ 72us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.222.61:35925: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":50}
W20260809 12:13:17.241099 15468 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.222.61:35925: Network error: Client connection negotiation failed: client connection to 127.0.222.61:35925: connect: Connection refused (error 111)
I20260809 12:13:17.384322 15468 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.222.61" port: 35925 } attempt: 4
I20260809 12:13:17.385579 15470 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 12:13:17.385008 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.222.61:35925 (local address 127.0.0.1:48258)
0809 12:13:17.385325 (+ 317us) negotiation.cc:107] Waiting for socket to connect
0809 12:13:17.385411 (+ 86us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.222.61:35925: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":152}
I20260809 12:13:17.386658 15470 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 12:13:17.386319 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.222.61:35925 (local address 127.0.0.1:48260)
0809 12:13:17.386454 (+ 135us) negotiation.cc:107] Waiting for socket to connect
0809 12:13:17.386544 (+ 90us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.222.61:35925: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":48}
W20260809 12:13:17.386878 15468 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.222.61:35925: Network error: Client connection negotiation failed: client connection to 127.0.222.61:35925: connect: Connection refused (error 111)
W20260809 12:13:17.389225 15475 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260809 12:13:17.389559 15475 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260809 12:13:17.389734 15475 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260809 12:13:17.397166 15475 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260809 12:13:17.397311 15475 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260809 12:13:17.397420 15475 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260809 12:13:17.397529 15475 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260809 12:13:17.397616 15475 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260809 12:13:17.407846 15475 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.222.20:46875
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskgH5Pjm/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786277382073938-888-0/minicluster-data/master-1/data
--fs_wal_dir=/tmp/dist-test-taskgH5Pjm/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786277382073938-888-0/minicluster-data/master-1/wal
--ipki_ca_key_size=768
--master_addresses=127.0.222.62:36165,127.0.222.61:35925,127.0.222.60:45255
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskgH5Pjm/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786277382073938-888-0/krb5kdc/kudu_127.0.222.61.keytab
--principal=kudu/127.0.222.61
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.222.61:35925
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskgH5Pjm/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786277382073938-888-0/minicluster-data/master-1/data/info.pb
--webserver_interface=127.0.222.61
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskgH5Pjm/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786277382073938-888-0/minicluster-data/master-1/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 09 Aug 2026 11:06:41 UTC on ccycloud.quasar-bwoogy.root.comops.site
build id 4575991
ASAN enabled
I20260809 12:13:17.409258 15475 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260809 12:13:17.410617 15475 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260809 12:13:17.422916 15480 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 12:13:17.423025 15481 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 12:13:17.423228 15483 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260809 12:13:17.424542 15475 server_base.cc:1034] running on GCE node
I20260809 12:13:17.425390 15475 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260809 12:13:17.426860 15475 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260809 12:13:17.428072 15475 hybrid_clock.cc:648] HybridClock initialized: now 1786277597428030 us; error 33 us; skew 500 ppm
I20260809 12:13:17.428469 15475 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 09 12:13:17 dist-test-slave-2w3w krb5kdc[15382](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786277597, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.222.61@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260809 12:13:17.431821 15475 init.cc:377] Logged in from keytab as kudu/127.0.222.61@KRBTEST.COM (short username kudu)
I20260809 12:13:17.433451 15475 webserver.cc:460] Webserver started at http://127.0.222.61:39009/ using document root <none> and password file <none>
I20260809 12:13:17.434194 15475 fs_manager.cc:359] Metadata directory not provided
I20260809 12:13:17.434340 15475 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260809 12:13:17.434648 15475 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260809 12:13:17.437779 15475 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskgH5Pjm/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786277382073938-888-0/minicluster-data/master-1/data/instance:
uuid: "2898ede5262c485c946a1aaa2d62a304"
format_stamp: "Formatted at 2026-08-09 12:13:17 on dist-test-slave-2w3w"
server_key: "00548aab4db4d0dac597d33bcca0a9ae"
server_key_iv: "8d83dbb50317bfd31fe5b899745003fc"
server_key_version: "encryptionkey@0"
I20260809 12:13:17.438632 15475 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskgH5Pjm/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786277382073938-888-0/minicluster-data/master-1/wal/instance:
uuid: "2898ede5262c485c946a1aaa2d62a304"
format_stamp: "Formatted at 2026-08-09 12:13:17 on dist-test-slave-2w3w"
server_key: "00548aab4db4d0dac597d33bcca0a9ae"
server_key_iv: "8d83dbb50317bfd31fe5b899745003fc"
server_key_version: "encryptionkey@0"
I20260809 12:13:17.444090 15475 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.003s sys 0.001s
I20260809 12:13:17.447582 15490 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260809 12:13:17.448899 15475 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260809 12:13:17.449086 15475 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskgH5Pjm/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786277382073938-888-0/minicluster-data/master-1/data,/tmp/dist-test-taskgH5Pjm/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786277382073938-888-0/minicluster-data/master-1/wal
uuid: "2898ede5262c485c946a1aaa2d62a304"
format_stamp: "Formatted at 2026-08-09 12:13:17 on dist-test-slave-2w3w"
server_key: "00548aab4db4d0dac597d33bcca0a9ae"
server_key_iv: "8d83dbb50317bfd31fe5b899745003fc"
server_key_version: "encryptionkey@0"
I20260809 12:13:17.449266 15475 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskgH5Pjm/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786277382073938-888-0/minicluster-data/master-1/wal
metadata directory: /tmp/dist-test-taskgH5Pjm/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786277382073938-888-0/minicluster-data/master-1/wal
1 data directories: /tmp/dist-test-taskgH5Pjm/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786277382073938-888-0/minicluster-data/master-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260809 12:13:17.475967 15475 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260809 12:13:17.480520 15475 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260809 12:13:17.480829 15475 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260809 12:13:17.499333 15475 rpc_server.cc:307] RPC server started. Bound to: 127.0.222.61:35925
I20260809 12:13:17.499352 15546 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.222.61:35925 every 8 connection(s)
I20260809 12:13:17.500753 15475 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskgH5Pjm/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786277382073938-888-0/minicluster-data/master-1/data/info.pb
I20260809 12:13:17.504438 15547 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260809 12:13:17.507974 888 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskgH5Pjm/build/asan/bin/kudu as pid 15475
I20260809 12:13:17.508179 888 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskgH5Pjm/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786277382073938-888-0/minicluster-data/master-1/wal/instance
I20260809 12:13:17.508754 888 external_mini_cluster.cc:1428] Setting key 2a7ea081679efaf0efbdf911e68a8384
I20260809 12:13:17.513254 15547 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.222.62" port: 36165 } has no permanent_uuid. Determining permanent_uuid...
Aug 09 12:13:17 dist-test-slave-2w3w krb5kdc[15382](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786277597, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.222.61@KRBTEST.COM for kudu/127.0.222.62@KRBTEST.COM
I20260809 12:13:17.537837 15472 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 12:13:17.516847 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40182 (local address 127.0.222.62:36165)
0809 12:13:17.517082 (+ 235us) server_negotiation.cc:206] Beginning negotiation
0809 12:13:17.517089 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0809 12:13:17.518422 (+ 1333us) server_negotiation.cc:407] Connection header received
0809 12:13:17.521267 (+ 2845us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 12:13:17.521275 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 12:13:17.521385 (+ 110us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 12:13:17.521517 (+ 132us) server_negotiation.cc:226] Negotiated authn=SASL
0809 12:13:17.523112 (+ 1595us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 12:13:17.524277 (+ 1165us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 12:13:17.525806 (+ 1529us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 12:13:17.526587 (+ 781us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 12:13:17.526734 (+ 147us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 12:13:17.530446 (+ 3712us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 12:13:17.530457 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 12:13:17.530466 (+ 9us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0809 12:13:17.530531 (+ 65us) server_negotiation.cc:922] calling sasl_server_start()
0809 12:13:17.534353 (+ 3822us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 12:13:17.535165 (+ 812us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 12:13:17.535173 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 12:13:17.535178 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0809 12:13:17.535344 (+ 166us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 12:13:17.535753 (+ 409us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 12:13:17.535761 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 12:13:17.535766 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0809 12:13:17.536204 (+ 438us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 12:13:17.536401 (+ 197us) server_negotiation.cc:1035] Waiting for connection context
0809 12:13:17.537128 (+ 727us) server_negotiation.cc:299] Negotiation successful
0809 12:13:17.537417 (+ 289us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":78}
I20260809 12:13:17.538468 15549 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 12:13:17.517349 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.222.62:36165 (local address 127.0.0.1:40182)
0809 12:13:17.518225 (+ 876us) negotiation.cc:107] Waiting for socket to connect
0809 12:13:17.518276 (+ 51us) client_negotiation.cc:174] Beginning negotiation
0809 12:13:17.520974 (+ 2698us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 12:13:17.521705 (+ 731us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 12:13:17.521713 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 12:13:17.522328 (+ 615us) client_negotiation.cc:189] Negotiated authn=SASL
0809 12:13:17.522904 (+ 576us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 12:13:17.522921 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 12:13:17.524448 (+ 1527us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 12:13:17.524451 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 12:13:17.525655 (+ 1204us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 12:13:17.525667 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 12:13:17.526797 (+ 1130us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 12:13:17.526802 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 12:13:17.527048 (+ 246us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 12:13:17.527767 (+ 719us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0809 12:13:17.527789 (+ 22us) client_negotiation.cc:656] calling sasl_client_start()
0809 12:13:17.530170 (+ 2381us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0809 12:13:17.534510 (+ 4340us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0809 12:13:17.534517 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0809 12:13:17.534530 (+ 13us) client_negotiation.cc:760] calling sasl_client_step()
0809 12:13:17.535025 (+ 495us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0809 12:13:17.535456 (+ 431us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0809 12:13:17.535459 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0809 12:13:17.535461 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0809 12:13:17.535625 (+ 164us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0809 12:13:17.536329 (+ 704us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0809 12:13:17.536334 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0809 12:13:17.536714 (+ 380us) client_negotiation.cc:769] Sending connection context
0809 12:13:17.536956 (+ 242us) client_negotiation.cc:240] Negotiation successful
0809 12:13:17.537330 (+ 374us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":581,"thread_start_us":349,"threads_started":1}
WARNING: no policy specified for kudu/127.0.222.60@KRBTEST.COM; defaulting to no policy
I20260809 12:13:17.540400 15547 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.222.61" port: 35925 } has no permanent_uuid. Determining permanent_uuid...
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.222.60@KRBTEST.COM" created.
Aug 09 12:13:17 dist-test-slave-2w3w krb5kdc[15382](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786277597, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.222.61@KRBTEST.COM for kudu/127.0.222.61@KRBTEST.COM
I20260809 12:13:17.555174 15549 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 12:13:17.541275 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.222.61:35925 (local address 127.0.0.1:48268)
0809 12:13:17.541474 (+ 199us) negotiation.cc:107] Waiting for socket to connect
0809 12:13:17.541494 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0809 12:13:17.541629 (+ 135us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 12:13:17.542596 (+ 967us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 12:13:17.542601 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 12:13:17.543118 (+ 517us) client_negotiation.cc:189] Negotiated authn=SASL
0809 12:13:17.543258 (+ 140us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 12:13:17.543267 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 12:13:17.544614 (+ 1347us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 12:13:17.544622 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 12:13:17.545457 (+ 835us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 12:13:17.545467 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 12:13:17.546278 (+ 811us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 12:13:17.546284 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 12:13:17.546503 (+ 219us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 12:13:17.547123 (+ 620us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0809 12:13:17.547141 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0809 12:13:17.549357 (+ 2216us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0809 12:13:17.552545 (+ 3188us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0809 12:13:17.552552 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0809 12:13:17.552559 (+ 7us) client_negotiation.cc:760] calling sasl_client_step()
0809 12:13:17.552938 (+ 379us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0809 12:13:17.553349 (+ 411us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0809 12:13:17.553357 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0809 12:13:17.553361 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0809 12:13:17.553557 (+ 196us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0809 12:13:17.554346 (+ 789us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0809 12:13:17.554350 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0809 12:13:17.554497 (+ 147us) client_negotiation.cc:769] Sending connection context
0809 12:13:17.554612 (+ 115us) client_negotiation.cc:240] Negotiation successful
0809 12:13:17.554825 (+ 213us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":56}
I20260809 12:13:17.555649 15551 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 12:13:17.541345 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48268 (local address 127.0.222.61:35925)
0809 12:13:17.542126 (+ 781us) server_negotiation.cc:206] Beginning negotiation
0809 12:13:17.542137 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0809 12:13:17.542171 (+ 34us) server_negotiation.cc:407] Connection header received
0809 12:13:17.542235 (+ 64us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 12:13:17.542244 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 12:13:17.542448 (+ 204us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 12:13:17.542607 (+ 159us) server_negotiation.cc:226] Negotiated authn=SASL
0809 12:13:17.543421 (+ 814us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 12:13:17.544476 (+ 1055us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 12:13:17.545596 (+ 1120us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 12:13:17.546129 (+ 533us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 12:13:17.546257 (+ 128us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 12:13:17.549559 (+ 3302us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 12:13:17.549571 (+ 12us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 12:13:17.549584 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0809 12:13:17.549612 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0809 12:13:17.552413 (+ 2801us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 12:13:17.553075 (+ 662us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 12:13:17.553081 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 12:13:17.553083 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0809 12:13:17.553196 (+ 113us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 12:13:17.553716 (+ 520us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 12:13:17.553724 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 12:13:17.553730 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0809 12:13:17.554214 (+ 484us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 12:13:17.554439 (+ 225us) server_negotiation.cc:1035] Waiting for connection context
0809 12:13:17.554812 (+ 373us) server_negotiation.cc:299] Negotiation successful
0809 12:13:17.555120 (+ 308us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":589,"thread_start_us":429,"threads_started":1}
I20260809 12:13:17.558035 15547 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.222.60" port: 45255 } has no permanent_uuid. Determining permanent_uuid...
I20260809 12:13:17.559171 15549 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 12:13:17.558652 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.222.60:45255 (local address 127.0.0.1:39754)
0809 12:13:17.558842 (+ 190us) negotiation.cc:107] Waiting for socket to connect
0809 12:13:17.559004 (+ 162us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.222.60:45255: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":59}
W20260809 12:13:17.559396 15494 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.222.60:45255: connect: Connection refused (error 111)
I20260809 12:13:17.561017 15549 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 12:13:17.560689 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.222.60:45255 (local address 127.0.0.1:39760)
0809 12:13:17.560831 (+ 142us) negotiation.cc:107] Waiting for socket to connect
0809 12:13:17.560916 (+ 85us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.222.60:45255: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":50}
W20260809 12:13:17.561231 15547 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.222.60:45255: Network error: Client connection negotiation failed: client connection to 127.0.222.60:45255: connect: Connection refused (error 111)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.222.60 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskgH5Pjm/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786277382073938-888-0/krb5kdc/kudu_127.0.222.60.keytab.
Entry for principal kudu/127.0.222.60 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskgH5Pjm/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786277382073938-888-0/krb5kdc/kudu_127.0.222.60.keytab.
I20260809 12:13:17.578121 888 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskgH5Pjm/build/asan/bin/kudu
/tmp/dist-test-taskgH5Pjm/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskgH5Pjm/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786277382073938-888-0/minicluster-data/master-2/wal
--fs_data_dirs=/tmp/dist-test-taskgH5Pjm/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786277382073938-888-0/minicluster-data/master-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskgH5Pjm/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786277382073938-888-0/minicluster-data/master-2/logs
--server_dump_info_path=/tmp/dist-test-taskgH5Pjm/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786277382073938-888-0/minicluster-data/master-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.222.60:45255
--webserver_interface=127.0.222.60
--webserver_port=0
--keytab_file=/tmp/dist-test-taskgH5Pjm/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786277382073938-888-0/krb5kdc/kudu_127.0.222.60.keytab
--principal=kudu/127.0.222.60
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.222.20:46875
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.222.62:36165,127.0.222.61:35925,127.0.222.60:45255
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskgH5Pjm/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786277382073938-888-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskgH5Pjm/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786277382073938-888-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskgH5Pjm/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786277382073938-888-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260809 12:13:17.610644 15547 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.222.60" port: 45255 } attempt: 1
I20260809 12:13:17.611948 15549 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 12:13:17.611439 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.222.60:45255 (local address 127.0.0.1:39772)
0809 12:13:17.611656 (+ 217us) negotiation.cc:107] Waiting for socket to connect
0809 12:13:17.611796 (+ 140us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.222.60:45255: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":71}
I20260809 12:13:17.612983 15549 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 12:13:17.612648 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.222.60:45255 (local address 127.0.0.1:39782)
0809 12:13:17.612796 (+ 148us) negotiation.cc:107] Waiting for socket to connect
0809 12:13:17.612886 (+ 90us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.222.60:45255: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":50}
W20260809 12:13:17.613240 15547 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.222.60:45255: Network error: Client connection negotiation failed: client connection to 127.0.222.60:45255: connect: Connection refused (error 111)
I20260809 12:13:17.681529 15547 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.222.60" port: 45255 } attempt: 2
I20260809 12:13:17.682756 15549 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 12:13:17.682275 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.222.60:45255 (local address 127.0.0.1:39788)
0809 12:13:17.682504 (+ 229us) negotiation.cc:107] Waiting for socket to connect
0809 12:13:17.682598 (+ 94us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.222.60:45255: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":79}
I20260809 12:13:17.683848 15549 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 12:13:17.683506 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.222.60:45255 (local address 127.0.0.1:39798)
0809 12:13:17.683670 (+ 164us) negotiation.cc:107] Waiting for socket to connect
0809 12:13:17.683737 (+ 67us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.222.60:45255: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":52}
W20260809 12:13:17.684118 15547 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.222.60:45255: Network error: Client connection negotiation failed: client connection to 127.0.222.60:45255: connect: Connection refused (error 111)
I20260809 12:13:17.686148 15468 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.222.61" port: 35925 } attempt: 5
Aug 09 12:13:17 dist-test-slave-2w3w krb5kdc[15382](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786277596, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.222.62@KRBTEST.COM for kudu/127.0.222.61@KRBTEST.COM
I20260809 12:13:17.701057 15470 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 12:13:17.686808 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.222.61:35925 (local address 127.0.0.1:48270)
0809 12:13:17.687211 (+ 403us) negotiation.cc:107] Waiting for socket to connect
0809 12:13:17.687227 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0809 12:13:17.687341 (+ 114us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 12:13:17.687718 (+ 377us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 12:13:17.687722 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 12:13:17.688127 (+ 405us) client_negotiation.cc:189] Negotiated authn=SASL
0809 12:13:17.688305 (+ 178us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 12:13:17.688313 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 12:13:17.689689 (+ 1376us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 12:13:17.689693 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 12:13:17.690566 (+ 873us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 12:13:17.690577 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 12:13:17.691810 (+ 1233us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 12:13:17.691814 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 12:13:17.691940 (+ 126us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 12:13:17.692574 (+ 634us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0809 12:13:17.692592 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0809 12:13:17.694630 (+ 2038us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0809 12:13:17.698611 (+ 3981us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0809 12:13:17.698615 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0809 12:13:17.698618 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0809 12:13:17.699058 (+ 440us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0809 12:13:17.699466 (+ 408us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0809 12:13:17.699469 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0809 12:13:17.699471 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0809 12:13:17.699604 (+ 133us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0809 12:13:17.700306 (+ 702us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0809 12:13:17.700309 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0809 12:13:17.700441 (+ 132us) client_negotiation.cc:769] Sending connection context
0809 12:13:17.700591 (+ 150us) client_negotiation.cc:240] Negotiation successful
0809 12:13:17.700798 (+ 207us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":285}
I20260809 12:13:17.701256 15551 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 12:13:17.686857 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48270 (local address 127.0.222.61:35925)
0809 12:13:17.687056 (+ 199us) server_negotiation.cc:206] Beginning negotiation
0809 12:13:17.687063 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0809 12:13:17.687322 (+ 259us) server_negotiation.cc:407] Connection header received
0809 12:13:17.687463 (+ 141us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 12:13:17.687468 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 12:13:17.687567 (+ 99us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 12:13:17.687682 (+ 115us) server_negotiation.cc:226] Negotiated authn=SASL
0809 12:13:17.688463 (+ 781us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 12:13:17.689537 (+ 1074us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 12:13:17.690730 (+ 1193us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 12:13:17.691659 (+ 929us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 12:13:17.691804 (+ 145us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 12:13:17.694848 (+ 3044us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 12:13:17.694854 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 12:13:17.694859 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0809 12:13:17.694901 (+ 42us) server_negotiation.cc:922] calling sasl_server_start()
0809 12:13:17.698468 (+ 3567us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 12:13:17.699172 (+ 704us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 12:13:17.699178 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 12:13:17.699181 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0809 12:13:17.699347 (+ 166us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 12:13:17.699723 (+ 376us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 12:13:17.699730 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 12:13:17.699736 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0809 12:13:17.700119 (+ 383us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 12:13:17.700299 (+ 180us) server_negotiation.cc:1035] Waiting for connection context
0809 12:13:17.700726 (+ 427us) server_negotiation.cc:299] Negotiation successful
0809 12:13:17.700951 (+ 225us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":62}
I20260809 12:13:17.702278 15468 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.222.60" port: 45255 } has no permanent_uuid. Determining permanent_uuid...
I20260809 12:13:17.703331 15470 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 12:13:17.702953 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.222.60:45255 (local address 127.0.0.1:39804)
0809 12:13:17.703130 (+ 177us) negotiation.cc:107] Waiting for socket to connect
0809 12:13:17.703203 (+ 73us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.222.60:45255: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":58}
I20260809 12:13:17.704285 15470 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 12:13:17.703975 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.222.60:45255 (local address 127.0.0.1:39808)
0809 12:13:17.704126 (+ 151us) negotiation.cc:107] Waiting for socket to connect
0809 12:13:17.704194 (+ 68us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.222.60:45255: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":46}
W20260809 12:13:17.704516 15468 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.222.60:45255: Network error: Client connection negotiation failed: client connection to 127.0.222.60:45255: connect: Connection refused (error 111)
I20260809 12:13:17.755750 15468 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.222.60" port: 45255 } attempt: 1
I20260809 12:13:17.757047 15470 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 12:13:17.756564 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.222.60:45255 (local address 127.0.0.1:39812)
0809 12:13:17.756763 (+ 199us) negotiation.cc:107] Waiting for socket to connect
0809 12:13:17.756882 (+ 119us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.222.60:45255: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":78}
I20260809 12:13:17.758107 15470 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 12:13:17.757763 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.222.60:45255 (local address 127.0.0.1:39818)
0809 12:13:17.757924 (+ 161us) negotiation.cc:107] Waiting for socket to connect
0809 12:13:17.757997 (+ 73us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.222.60:45255: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":57}
W20260809 12:13:17.758324 15468 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.222.60:45255: Network error: Client connection negotiation failed: client connection to 127.0.222.60:45255: connect: Connection refused (error 111)
I20260809 12:13:17.775345 15547 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.222.60" port: 45255 } attempt: 3
I20260809 12:13:17.776640 15549 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 12:13:17.776126 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.222.60:45255 (local address 127.0.0.1:39826)
0809 12:13:17.776376 (+ 250us) negotiation.cc:107] Waiting for socket to connect
0809 12:13:17.776473 (+ 97us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.222.60:45255: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":82}
I20260809 12:13:17.777731 15549 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 12:13:17.777296 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.222.60:45255 (local address 127.0.0.1:39836)
0809 12:13:17.777527 (+ 231us) negotiation.cc:107] Waiting for socket to connect
0809 12:13:17.777616 (+ 89us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.222.60:45255: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":130}
W20260809 12:13:17.777930 15547 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.222.60:45255: Network error: Client connection negotiation failed: client connection to 127.0.222.60:45255: connect: Connection refused (error 111)
I20260809 12:13:17.826546 15468 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.222.60" port: 45255 } attempt: 2
I20260809 12:13:17.827746 15470 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 12:13:17.827225 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.222.60:45255 (local address 127.0.0.1:39852)
0809 12:13:17.827485 (+ 260us) negotiation.cc:107] Waiting for socket to connect
0809 12:13:17.827579 (+ 94us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.222.60:45255: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":71}
I20260809 12:13:17.828896 15470 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 12:13:17.828560 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.222.60:45255 (local address 127.0.0.1:39856)
0809 12:13:17.828719 (+ 159us) negotiation.cc:107] Waiting for socket to connect
0809 12:13:17.828794 (+ 75us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.222.60:45255: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":54}
W20260809 12:13:17.829137 15468 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.222.60:45255: Network error: Client connection negotiation failed: client connection to 127.0.222.60:45255: connect: Connection refused (error 111)
I20260809 12:13:17.921149 15547 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.222.60" port: 45255 } attempt: 4
I20260809 12:13:17.922377 15549 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 12:13:17.921844 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.222.60:45255 (local address 127.0.0.1:39870)
0809 12:13:17.922099 (+ 255us) negotiation.cc:107] Waiting for socket to connect
0809 12:13:17.922223 (+ 124us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.222.60:45255: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":63}
I20260809 12:13:17.923527 15549 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 12:13:17.923204 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.222.60:45255 (local address 127.0.0.1:39884)
0809 12:13:17.923339 (+ 135us) negotiation.cc:107] Waiting for socket to connect
0809 12:13:17.923423 (+ 84us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.222.60:45255: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":32}
W20260809 12:13:17.923725 15547 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.222.60:45255: Network error: Client connection negotiation failed: client connection to 127.0.222.60:45255: connect: Connection refused (error 111)
W20260809 12:13:17.933456 15554 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260809 12:13:17.933800 15554 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260809 12:13:17.933972 15554 flags.cc:425] Enabled unsafe flag: --never_fsync=true
I20260809 12:13:17.935328 15468 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.222.60" port: 45255 } attempt: 3
I20260809 12:13:17.936393 15470 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 12:13:17.935962 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.222.60:45255 (local address 127.0.0.1:39892)
0809 12:13:17.936146 (+ 184us) negotiation.cc:107] Waiting for socket to connect
0809 12:13:17.936232 (+ 86us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.222.60:45255: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":67}
I20260809 12:13:17.937405 15470 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 12:13:17.937041 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.222.60:45255 (local address 127.0.0.1:39908)
0809 12:13:17.937230 (+ 189us) negotiation.cc:107] Waiting for socket to connect
0809 12:13:17.937300 (+ 70us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.222.60:45255: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":58}
W20260809 12:13:17.937626 15468 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.222.60:45255: Network error: Client connection negotiation failed: client connection to 127.0.222.60:45255: connect: Connection refused (error 111)
W20260809 12:13:17.941939 15554 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260809 12:13:17.942080 15554 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260809 12:13:17.942183 15554 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260809 12:13:17.942286 15554 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260809 12:13:17.942363 15554 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260809 12:13:17.952255 15554 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.222.20:46875
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskgH5Pjm/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786277382073938-888-0/minicluster-data/master-2/data
--fs_wal_dir=/tmp/dist-test-taskgH5Pjm/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786277382073938-888-0/minicluster-data/master-2/wal
--ipki_ca_key_size=768
--master_addresses=127.0.222.62:36165,127.0.222.61:35925,127.0.222.60:45255
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskgH5Pjm/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786277382073938-888-0/krb5kdc/kudu_127.0.222.60.keytab
--principal=kudu/127.0.222.60
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.222.60:45255
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskgH5Pjm/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786277382073938-888-0/minicluster-data/master-2/data/info.pb
--webserver_interface=127.0.222.60
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskgH5Pjm/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786277382073938-888-0/minicluster-data/master-2/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 09 Aug 2026 11:06:41 UTC on ccycloud.quasar-bwoogy.root.comops.site
build id 4575991
ASAN enabled
I20260809 12:13:17.953747 15554 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260809 12:13:17.955294 15554 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260809 12:13:17.967424 15559 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 12:13:17.967408 15560 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 12:13:17.967408 15562 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260809 12:13:17.968585 15554 server_base.cc:1034] running on GCE node
I20260809 12:13:17.969419 15554 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260809 12:13:17.970875 15554 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260809 12:13:17.972124 15554 hybrid_clock.cc:648] HybridClock initialized: now 1786277597972084 us; error 35 us; skew 500 ppm
I20260809 12:13:17.972487 15554 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 09 12:13:17 dist-test-slave-2w3w krb5kdc[15382](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786277597, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.222.60@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260809 12:13:17.975618 15554 init.cc:377] Logged in from keytab as kudu/127.0.222.60@KRBTEST.COM (short username kudu)
I20260809 12:13:17.977259 15554 webserver.cc:460] Webserver started at http://127.0.222.60:45953/ using document root <none> and password file <none>
I20260809 12:13:17.977959 15554 fs_manager.cc:359] Metadata directory not provided
I20260809 12:13:17.978102 15554 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260809 12:13:17.978447 15554 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260809 12:13:17.981468 15554 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskgH5Pjm/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786277382073938-888-0/minicluster-data/master-2/data/instance:
uuid: "b3359c45352b4ec9a218caec9a9a949f"
format_stamp: "Formatted at 2026-08-09 12:13:17 on dist-test-slave-2w3w"
server_key: "75c62032d42f96d5047bb1b845c3b7a9"
server_key_iv: "79db9fbf4aa77ce2dc725dae0643189a"
server_key_version: "encryptionkey@0"
I20260809 12:13:17.982263 15554 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskgH5Pjm/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786277382073938-888-0/minicluster-data/master-2/wal/instance:
uuid: "b3359c45352b4ec9a218caec9a9a949f"
format_stamp: "Formatted at 2026-08-09 12:13:17 on dist-test-slave-2w3w"
server_key: "75c62032d42f96d5047bb1b845c3b7a9"
server_key_iv: "79db9fbf4aa77ce2dc725dae0643189a"
server_key_version: "encryptionkey@0"
I20260809 12:13:17.987332 15554 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.002s
I20260809 12:13:17.990743 15569 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260809 12:13:17.992062 15554 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260809 12:13:17.992247 15554 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskgH5Pjm/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786277382073938-888-0/minicluster-data/master-2/data,/tmp/dist-test-taskgH5Pjm/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786277382073938-888-0/minicluster-data/master-2/wal
uuid: "b3359c45352b4ec9a218caec9a9a949f"
format_stamp: "Formatted at 2026-08-09 12:13:17 on dist-test-slave-2w3w"
server_key: "75c62032d42f96d5047bb1b845c3b7a9"
server_key_iv: "79db9fbf4aa77ce2dc725dae0643189a"
server_key_version: "encryptionkey@0"
I20260809 12:13:17.992455 15554 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskgH5Pjm/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786277382073938-888-0/minicluster-data/master-2/wal
metadata directory: /tmp/dist-test-taskgH5Pjm/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786277382073938-888-0/minicluster-data/master-2/wal
1 data directories: /tmp/dist-test-taskgH5Pjm/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786277382073938-888-0/minicluster-data/master-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260809 12:13:18.018523 15554 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260809 12:13:18.022742 15554 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260809 12:13:18.023098 15554 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260809 12:13:18.041831 15554 rpc_server.cc:307] RPC server started. Bound to: 127.0.222.60:45255
I20260809 12:13:18.041870 15622 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.222.60:45255 every 8 connection(s)
I20260809 12:13:18.043428 15554 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskgH5Pjm/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786277382073938-888-0/minicluster-data/master-2/data/info.pb
I20260809 12:13:18.047008 888 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskgH5Pjm/build/asan/bin/kudu as pid 15554
I20260809 12:13:18.047232 888 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskgH5Pjm/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786277382073938-888-0/minicluster-data/master-2/wal/instance
I20260809 12:13:18.047456 15623 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260809 12:13:18.047873 888 external_mini_cluster.cc:1428] Setting key 5fec0a18fe05bcff2e519b926fe99d83
I20260809 12:13:18.057677 15623 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.222.62" port: 36165 } has no permanent_uuid. Determining permanent_uuid...
Aug 09 12:13:18 dist-test-slave-2w3w krb5kdc[15382](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786277596, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.222.60@KRBTEST.COM
Aug 09 12:13:18 dist-test-slave-2w3w krb5kdc[15382](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786277597, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.222.60@KRBTEST.COM for kudu/127.0.222.62@KRBTEST.COM
I20260809 12:13:18.073210 15626 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 12:13:18.049830 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39924 (local address 127.0.222.60:45255)
0809 12:13:18.050659 (+ 829us) server_negotiation.cc:206] Beginning negotiation
0809 12:13:18.050670 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0809 12:13:18.050712 (+ 42us) server_negotiation.cc:407] Connection header received
0809 12:13:18.052454 (+ 1742us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 12:13:18.052475 (+ 21us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 12:13:18.052848 (+ 373us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 12:13:18.053248 (+ 400us) server_negotiation.cc:226] Negotiated authn=SASL
0809 12:13:18.055262 (+ 2014us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 12:13:18.056567 (+ 1305us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 12:13:18.058626 (+ 2059us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 12:13:18.059418 (+ 792us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 12:13:18.059546 (+ 128us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 12:13:18.065271 (+ 5725us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 12:13:18.065278 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 12:13:18.065288 (+ 10us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0809 12:13:18.065325 (+ 37us) server_negotiation.cc:922] calling sasl_server_start()
0809 12:13:18.068695 (+ 3370us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 12:13:18.069531 (+ 836us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 12:13:18.069536 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 12:13:18.069542 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0809 12:13:18.069752 (+ 210us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 12:13:18.070293 (+ 541us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 12:13:18.070301 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 12:13:18.070308 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0809 12:13:18.070987 (+ 679us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 12:13:18.071215 (+ 228us) server_negotiation.cc:1035] Waiting for connection context
0809 12:13:18.071716 (+ 501us) server_negotiation.cc:299] Negotiation successful
0809 12:13:18.072107 (+ 391us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":632,"thread_start_us":393,"threads_started":1}
I20260809 12:13:18.077697 15628 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 12:13:18.060621 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.222.62:36165 (local address 127.0.0.1:40192)
0809 12:13:18.061689 (+ 1068us) negotiation.cc:107] Waiting for socket to connect
0809 12:13:18.061738 (+ 49us) client_negotiation.cc:174] Beginning negotiation
0809 12:13:18.062023 (+ 285us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 12:13:18.062404 (+ 381us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 12:13:18.062414 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 12:13:18.063295 (+ 881us) client_negotiation.cc:189] Negotiated authn=SASL
0809 12:13:18.063696 (+ 401us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 12:13:18.063711 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 12:13:18.065006 (+ 1295us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 12:13:18.065011 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 12:13:18.066358 (+ 1347us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 12:13:18.066371 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 12:13:18.067497 (+ 1126us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 12:13:18.067502 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 12:13:18.067632 (+ 130us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 12:13:18.068250 (+ 618us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0809 12:13:18.068271 (+ 21us) client_negotiation.cc:656] calling sasl_client_start()
0809 12:13:18.070904 (+ 2633us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0809 12:13:18.074722 (+ 3818us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0809 12:13:18.074733 (+ 11us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0809 12:13:18.074750 (+ 17us) client_negotiation.cc:760] calling sasl_client_step()
0809 12:13:18.075285 (+ 535us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0809 12:13:18.075737 (+ 452us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0809 12:13:18.075744 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0809 12:13:18.075750 (+ 6us) client_negotiation.cc:760] calling sasl_client_step()
0809 12:13:18.075937 (+ 187us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0809 12:13:18.076574 (+ 637us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0809 12:13:18.076582 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0809 12:13:18.076732 (+ 150us) client_negotiation.cc:769] Sending connection context
0809 12:13:18.076949 (+ 217us) client_negotiation.cc:240] Negotiation successful
0809 12:13:18.077258 (+ 309us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":810,"thread_start_us":644,"threads_started":1}
I20260809 12:13:18.078531 15627 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 12:13:18.060659 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40192 (local address 127.0.222.62:36165)
0809 12:13:18.061310 (+ 651us) server_negotiation.cc:206] Beginning negotiation
0809 12:13:18.061316 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0809 12:13:18.061927 (+ 611us) server_negotiation.cc:407] Connection header received
0809 12:13:18.062147 (+ 220us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 12:13:18.062150 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 12:13:18.062213 (+ 63us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 12:13:18.062305 (+ 92us) server_negotiation.cc:226] Negotiated authn=SASL
0809 12:13:18.063911 (+ 1606us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 12:13:18.064819 (+ 908us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 12:13:18.066528 (+ 1709us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 12:13:18.067294 (+ 766us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 12:13:18.067435 (+ 141us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 12:13:18.071146 (+ 3711us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 12:13:18.071152 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 12:13:18.071156 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0809 12:13:18.071199 (+ 43us) server_negotiation.cc:922] calling sasl_server_start()
0809 12:13:18.074571 (+ 3372us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 12:13:18.075408 (+ 837us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 12:13:18.075413 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 12:13:18.075416 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0809 12:13:18.075608 (+ 192us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 12:13:18.076089 (+ 481us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 12:13:18.076094 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 12:13:18.076101 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0809 12:13:18.076417 (+ 316us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 12:13:18.076574 (+ 157us) server_negotiation.cc:1035] Waiting for connection context
0809 12:13:18.078059 (+ 1485us) server_negotiation.cc:299] Negotiation successful
0809 12:13:18.078241 (+ 182us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":481,"thread_start_us":312,"threads_started":1}
I20260809 12:13:18.079762 15623 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.222.61" port: 35925 } has no permanent_uuid. Determining permanent_uuid...
Aug 09 12:13:18 dist-test-slave-2w3w krb5kdc[15382](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786277596, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.222.62@KRBTEST.COM
Aug 09 12:13:18 dist-test-slave-2w3w krb5kdc[15382](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786277597, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.222.60@KRBTEST.COM for kudu/127.0.222.61@KRBTEST.COM
I20260809 12:13:18.092815 15629 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 12:13:18.076627 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40198 (local address 127.0.222.62:36165)
0809 12:13:18.077240 (+ 613us) server_negotiation.cc:206] Beginning negotiation
0809 12:13:18.077261 (+ 21us) server_negotiation.cc:399] Waiting for connection header
0809 12:13:18.077286 (+ 25us) server_negotiation.cc:407] Connection header received
0809 12:13:18.077346 (+ 60us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 12:13:18.077351 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 12:13:18.077420 (+ 69us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 12:13:18.077528 (+ 108us) server_negotiation.cc:226] Negotiated authn=SASL
0809 12:13:18.078949 (+ 1421us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 12:13:18.079962 (+ 1013us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 12:13:18.081654 (+ 1692us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 12:13:18.082099 (+ 445us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 12:13:18.082251 (+ 152us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 12:13:18.086392 (+ 4141us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 12:13:18.086398 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 12:13:18.086401 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0809 12:13:18.086431 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0809 12:13:18.089957 (+ 3526us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 12:13:18.090694 (+ 737us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 12:13:18.090701 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 12:13:18.090706 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0809 12:13:18.090851 (+ 145us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 12:13:18.091283 (+ 432us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 12:13:18.091289 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 12:13:18.091296 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0809 12:13:18.091662 (+ 366us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 12:13:18.091931 (+ 269us) server_negotiation.cc:1035] Waiting for connection context
0809 12:13:18.092258 (+ 327us) server_negotiation.cc:299] Negotiation successful
0809 12:13:18.092420 (+ 162us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":467,"thread_start_us":262,"threads_started":1}
I20260809 12:13:18.094590 15628 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 12:13:18.080580 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.222.61:35925 (local address 127.0.0.1:48284)
0809 12:13:18.080793 (+ 213us) negotiation.cc:107] Waiting for socket to connect
0809 12:13:18.080834 (+ 41us) client_negotiation.cc:174] Beginning negotiation
0809 12:13:18.080915 (+ 81us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 12:13:18.081229 (+ 314us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 12:13:18.081231 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 12:13:18.081548 (+ 317us) client_negotiation.cc:189] Negotiated authn=SASL
0809 12:13:18.081675 (+ 127us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 12:13:18.081683 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 12:13:18.082864 (+ 1181us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 12:13:18.082870 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 12:13:18.083910 (+ 1040us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 12:13:18.083923 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 12:13:18.084701 (+ 778us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 12:13:18.084707 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 12:13:18.084919 (+ 212us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 12:13:18.085542 (+ 623us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0809 12:13:18.085572 (+ 30us) client_negotiation.cc:656] calling sasl_client_start()
0809 12:13:18.087728 (+ 2156us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0809 12:13:18.091909 (+ 4181us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0809 12:13:18.091915 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0809 12:13:18.091919 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0809 12:13:18.092430 (+ 511us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0809 12:13:18.092953 (+ 523us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0809 12:13:18.092960 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0809 12:13:18.092965 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0809 12:13:18.093161 (+ 196us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0809 12:13:18.093732 (+ 571us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0809 12:13:18.093737 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0809 12:13:18.093896 (+ 159us) client_negotiation.cc:769] Sending connection context
0809 12:13:18.094013 (+ 117us) client_negotiation.cc:240] Negotiation successful
0809 12:13:18.094269 (+ 256us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":122}
I20260809 12:13:18.095168 15551 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 12:13:18.080617 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48284 (local address 127.0.222.61:35925)
0809 12:13:18.080814 (+ 197us) server_negotiation.cc:206] Beginning negotiation
0809 12:13:18.080820 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0809 12:13:18.080876 (+ 56us) server_negotiation.cc:407] Connection header received
0809 12:13:18.081028 (+ 152us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 12:13:18.081033 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 12:13:18.081103 (+ 70us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 12:13:18.081204 (+ 101us) server_negotiation.cc:226] Negotiated authn=SASL
0809 12:13:18.081837 (+ 633us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 12:13:18.082640 (+ 803us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 12:13:18.084079 (+ 1439us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 12:13:18.084532 (+ 453us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 12:13:18.084689 (+ 157us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 12:13:18.087962 (+ 3273us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 12:13:18.087969 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 12:13:18.087974 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0809 12:13:18.088023 (+ 49us) server_negotiation.cc:922] calling sasl_server_start()
0809 12:13:18.091781 (+ 3758us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 12:13:18.092577 (+ 796us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 12:13:18.092583 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 12:13:18.092587 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0809 12:13:18.092799 (+ 212us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 12:13:18.093310 (+ 511us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 12:13:18.093314 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 12:13:18.093316 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0809 12:13:18.093563 (+ 247us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 12:13:18.093743 (+ 180us) server_negotiation.cc:1035] Waiting for connection context
0809 12:13:18.094149 (+ 406us) server_negotiation.cc:299] Negotiation successful
0809 12:13:18.094382 (+ 233us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":65}
I20260809 12:13:18.096282 15623 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.222.60" port: 45255 } has no permanent_uuid. Determining permanent_uuid...
Aug 09 12:13:18 dist-test-slave-2w3w krb5kdc[15382](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786277597, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.222.60@KRBTEST.COM for kudu/127.0.222.60@KRBTEST.COM
Aug 09 12:13:18 dist-test-slave-2w3w krb5kdc[15382](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786277596, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.222.61@KRBTEST.COM
I20260809 12:13:18.110188 15628 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 12:13:18.097071 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.222.60:45255 (local address 127.0.0.1:39938)
0809 12:13:18.097242 (+ 171us) negotiation.cc:107] Waiting for socket to connect
0809 12:13:18.097260 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0809 12:13:18.097359 (+ 99us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 12:13:18.097847 (+ 488us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 12:13:18.097850 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 12:13:18.098158 (+ 308us) client_negotiation.cc:189] Negotiated authn=SASL
0809 12:13:18.098293 (+ 135us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 12:13:18.098300 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 12:13:18.099520 (+ 1220us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 12:13:18.099526 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 12:13:18.100658 (+ 1132us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 12:13:18.100670 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 12:13:18.101465 (+ 795us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 12:13:18.101470 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 12:13:18.101644 (+ 174us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 12:13:18.102144 (+ 500us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0809 12:13:18.102159 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0809 12:13:18.104222 (+ 2063us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0809 12:13:18.107465 (+ 3243us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0809 12:13:18.107470 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0809 12:13:18.107474 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0809 12:13:18.107994 (+ 520us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0809 12:13:18.108342 (+ 348us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0809 12:13:18.108349 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0809 12:13:18.108355 (+ 6us) client_negotiation.cc:760] calling sasl_client_step()
0809 12:13:18.108517 (+ 162us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0809 12:13:18.109029 (+ 512us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0809 12:13:18.109033 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0809 12:13:18.109184 (+ 151us) client_negotiation.cc:769] Sending connection context
0809 12:13:18.109332 (+ 148us) client_negotiation.cc:240] Negotiation successful
0809 12:13:18.109663 (+ 331us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":54}
I20260809 12:13:18.110184 15626 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 12:13:18.097334 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39938 (local address 127.0.222.60:45255)
0809 12:13:18.097520 (+ 186us) server_negotiation.cc:206] Beginning negotiation
0809 12:13:18.097529 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0809 12:13:18.097550 (+ 21us) server_negotiation.cc:407] Connection header received
0809 12:13:18.097602 (+ 52us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 12:13:18.097609 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 12:13:18.097689 (+ 80us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 12:13:18.097821 (+ 132us) server_negotiation.cc:226] Negotiated authn=SASL
0809 12:13:18.098435 (+ 614us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 12:13:18.099363 (+ 928us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 12:13:18.100805 (+ 1442us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 12:13:18.101299 (+ 494us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 12:13:18.101448 (+ 149us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 12:13:18.104421 (+ 2973us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 12:13:18.104426 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 12:13:18.104429 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0809 12:13:18.104457 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0809 12:13:18.107332 (+ 2875us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 12:13:18.108118 (+ 786us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 12:13:18.108122 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 12:13:18.108124 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0809 12:13:18.108231 (+ 107us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 12:13:18.108627 (+ 396us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 12:13:18.108633 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 12:13:18.108636 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0809 12:13:18.108914 (+ 278us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 12:13:18.109065 (+ 151us) server_negotiation.cc:1035] Waiting for connection context
0809 12:13:18.109414 (+ 349us) server_negotiation.cc:299] Negotiation successful
0809 12:13:18.109837 (+ 423us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":183,"server-negotiator.queue_time_us":74}
I20260809 12:13:18.111809 15630 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 12:13:18.094673 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48290 (local address 127.0.222.61:35925)
0809 12:13:18.095355 (+ 682us) server_negotiation.cc:206] Beginning negotiation
0809 12:13:18.095361 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0809 12:13:18.095384 (+ 23us) server_negotiation.cc:407] Connection header received
0809 12:13:18.095427 (+ 43us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 12:13:18.095432 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 12:13:18.095512 (+ 80us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 12:13:18.095632 (+ 120us) server_negotiation.cc:226] Negotiated authn=SASL
0809 12:13:18.097439 (+ 1807us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 12:13:18.098631 (+ 1192us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 12:13:18.100269 (+ 1638us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 12:13:18.100678 (+ 409us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 12:13:18.100836 (+ 158us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 12:13:18.105223 (+ 4387us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 12:13:18.105228 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 12:13:18.105231 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0809 12:13:18.105260 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0809 12:13:18.108295 (+ 3035us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 12:13:18.109122 (+ 827us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 12:13:18.109128 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 12:13:18.109132 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0809 12:13:18.109275 (+ 143us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 12:13:18.109675 (+ 400us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 12:13:18.109680 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 12:13:18.109684 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0809 12:13:18.109950 (+ 266us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 12:13:18.110229 (+ 279us) server_negotiation.cc:1035] Waiting for connection context
0809 12:13:18.110551 (+ 322us) server_negotiation.cc:299] Negotiation successful
0809 12:13:18.111460 (+ 909us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":520,"thread_start_us":346,"threads_started":1}
I20260809 12:13:18.114864 15468 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.222.60" port: 45255 } attempt: 4
I20260809 12:13:18.116010 15623 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P b3359c45352b4ec9a218caec9a9a949f: Bootstrap starting.
I20260809 12:13:18.116986 888 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260809 12:13:18.119935 15623 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P b3359c45352b4ec9a218caec9a9a949f: Neither blocks nor log segments found. Creating new log.
I20260809 12:13:18.121366 15623 log.cc:824] T 00000000000000000000000000000000 P b3359c45352b4ec9a218caec9a9a949f: Log is configured to *not* fsync() on all Append() calls
Aug 09 12:13:18 dist-test-slave-2w3w krb5kdc[15382](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786277596, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.222.62@KRBTEST.COM for kudu/127.0.222.60@KRBTEST.COM
I20260809 12:13:18.124715 15623 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P b3359c45352b4ec9a218caec9a9a949f: No bootstrap required, opened a new log
I20260809 12:13:18.132860 15470 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 12:13:18.115515 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.222.60:45255 (local address 127.0.0.1:39940)
0809 12:13:18.115708 (+ 193us) negotiation.cc:107] Waiting for socket to connect
0809 12:13:18.115729 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0809 12:13:18.115839 (+ 110us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 12:13:18.116288 (+ 449us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 12:13:18.116294 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 12:13:18.116785 (+ 491us) client_negotiation.cc:189] Negotiated authn=SASL
0809 12:13:18.117001 (+ 216us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 12:13:18.117014 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 12:13:18.118477 (+ 1463us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 12:13:18.118484 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 12:13:18.120089 (+ 1605us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 12:13:18.120108 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 12:13:18.121170 (+ 1062us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 12:13:18.121179 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 12:13:18.121443 (+ 264us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 12:13:18.122307 (+ 864us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0809 12:13:18.122325 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0809 12:13:18.125607 (+ 3282us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0809 12:13:18.129908 (+ 4301us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0809 12:13:18.129917 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0809 12:13:18.129922 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0809 12:13:18.130509 (+ 587us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0809 12:13:18.130938 (+ 429us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0809 12:13:18.130945 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0809 12:13:18.130950 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0809 12:13:18.131148 (+ 198us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0809 12:13:18.131851 (+ 703us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0809 12:13:18.131856 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0809 12:13:18.132032 (+ 176us) client_negotiation.cc:769] Sending connection context
0809 12:13:18.132164 (+ 132us) client_negotiation.cc:240] Negotiation successful
0809 12:13:18.132467 (+ 303us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":65}
I20260809 12:13:18.134131 15626 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 12:13:18.115823 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39940 (local address 127.0.222.60:45255)
0809 12:13:18.115990 (+ 167us) server_negotiation.cc:206] Beginning negotiation
0809 12:13:18.115997 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0809 12:13:18.116010 (+ 13us) server_negotiation.cc:407] Connection header received
0809 12:13:18.116046 (+ 36us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 12:13:18.116049 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 12:13:18.116133 (+ 84us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 12:13:18.116232 (+ 99us) server_negotiation.cc:226] Negotiated authn=SASL
0809 12:13:18.117168 (+ 936us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 12:13:18.118288 (+ 1120us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 12:13:18.120263 (+ 1975us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 12:13:18.120972 (+ 709us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 12:13:18.121138 (+ 166us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 12:13:18.125838 (+ 4700us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 12:13:18.125845 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 12:13:18.125849 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0809 12:13:18.125886 (+ 37us) server_negotiation.cc:922] calling sasl_server_start()
0809 12:13:18.129760 (+ 3874us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 12:13:18.130623 (+ 863us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 12:13:18.130629 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 12:13:18.130633 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0809 12:13:18.130789 (+ 156us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 12:13:18.131309 (+ 520us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 12:13:18.131313 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 12:13:18.131317 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0809 12:13:18.131686 (+ 369us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 12:13:18.133381 (+ 1695us) server_negotiation.cc:1035] Waiting for connection context
0809 12:13:18.133515 (+ 134us) server_negotiation.cc:299] Negotiation successful
0809 12:13:18.133741 (+ 226us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":50}
I20260809 12:13:18.136400 15623 raft_consensus.cc:348] T 00000000000000000000000000000000 P b3359c45352b4ec9a218caec9a9a949f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "962f780b4e6e42209bae1266526c1e08" member_type: VOTER last_known_addr { host: "127.0.222.62" port: 36165 } } peers { permanent_uuid: "2898ede5262c485c946a1aaa2d62a304" member_type: VOTER last_known_addr { host: "127.0.222.61" port: 35925 } } peers { permanent_uuid: "b3359c45352b4ec9a218caec9a9a949f" member_type: VOTER last_known_addr { host: "127.0.222.60" port: 45255 } }
I20260809 12:13:18.136936 15623 raft_consensus.cc:374] T 00000000000000000000000000000000 P b3359c45352b4ec9a218caec9a9a949f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260809 12:13:18.137071 15623 raft_consensus.cc:729] T 00000000000000000000000000000000 P b3359c45352b4ec9a218caec9a9a949f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b3359c45352b4ec9a218caec9a9a949f, State: Initialized, Role: FOLLOWER
I20260809 12:13:18.137880 15623 consensus_queue.cc:260] T 00000000000000000000000000000000 P b3359c45352b4ec9a218caec9a9a949f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "962f780b4e6e42209bae1266526c1e08" member_type: VOTER last_known_addr { host: "127.0.222.62" port: 36165 } } peers { permanent_uuid: "2898ede5262c485c946a1aaa2d62a304" member_type: VOTER last_known_addr { host: "127.0.222.61" port: 35925 } } peers { permanent_uuid: "b3359c45352b4ec9a218caec9a9a949f" member_type: VOTER last_known_addr { host: "127.0.222.60" port: 45255 } }
I20260809 12:13:18.141667 15468 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 962f780b4e6e42209bae1266526c1e08: Bootstrap starting.
I20260809 12:13:18.144591 15638 sys_catalog.cc:455] T 00000000000000000000000000000000 P b3359c45352b4ec9a218caec9a9a949f [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "962f780b4e6e42209bae1266526c1e08" member_type: VOTER last_known_addr { host: "127.0.222.62" port: 36165 } } peers { permanent_uuid: "2898ede5262c485c946a1aaa2d62a304" member_type: VOTER last_known_addr { host: "127.0.222.61" port: 35925 } } peers { permanent_uuid: "b3359c45352b4ec9a218caec9a9a949f" member_type: VOTER last_known_addr { host: "127.0.222.60" port: 45255 } } }
I20260809 12:13:18.145004 15638 sys_catalog.cc:458] T 00000000000000000000000000000000 P b3359c45352b4ec9a218caec9a9a949f [sys.catalog]: This master's current role is: FOLLOWER
I20260809 12:13:18.145870 15623 sys_catalog.cc:564] T 00000000000000000000000000000000 P b3359c45352b4ec9a218caec9a9a949f [sys.catalog]: configured and running, proceeding with master startup.
I20260809 12:13:18.146454 15468 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 962f780b4e6e42209bae1266526c1e08: Neither blocks nor log segments found. Creating new log.
I20260809 12:13:18.159056 15468 log.cc:824] T 00000000000000000000000000000000 P 962f780b4e6e42209bae1266526c1e08: Log is configured to *not* fsync() on all Append() calls
I20260809 12:13:18.161669 15629 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 12:13:18.130422 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40208 (local address 127.0.222.62:36165)
0809 12:13:18.130623 (+ 201us) server_negotiation.cc:206] Beginning negotiation
0809 12:13:18.130630 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0809 12:13:18.139128 (+ 8498us) server_negotiation.cc:407] Connection header received
0809 12:13:18.139385 (+ 257us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 12:13:18.139394 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 12:13:18.139474 (+ 80us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 12:13:18.139577 (+ 103us) server_negotiation.cc:226] Negotiated authn=SASL
0809 12:13:18.141900 (+ 2323us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 12:13:18.143385 (+ 1485us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 12:13:18.145739 (+ 2354us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 12:13:18.146431 (+ 692us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 12:13:18.146536 (+ 105us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 12:13:18.151420 (+ 4884us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 12:13:18.151429 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 12:13:18.151437 (+ 8us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0809 12:13:18.151486 (+ 49us) server_negotiation.cc:922] calling sasl_server_start()
0809 12:13:18.155992 (+ 4506us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 12:13:18.157323 (+ 1331us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 12:13:18.157332 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 12:13:18.157338 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0809 12:13:18.157515 (+ 177us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 12:13:18.158146 (+ 631us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 12:13:18.158154 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 12:13:18.158160 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0809 12:13:18.158621 (+ 461us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 12:13:18.158927 (+ 306us) server_negotiation.cc:1035] Waiting for connection context
0809 12:13:18.159776 (+ 849us) server_negotiation.cc:299] Negotiation successful
0809 12:13:18.160083 (+ 307us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":68}
I20260809 12:13:18.169440 15468 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 962f780b4e6e42209bae1266526c1e08: No bootstrap required, opened a new log
I20260809 12:13:18.170456 15626 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 12:13:18.139215 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39944 (local address 127.0.222.60:45255)
0809 12:13:18.139384 (+ 169us) server_negotiation.cc:206] Beginning negotiation
0809 12:13:18.139392 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0809 12:13:18.151579 (+ 12187us) server_negotiation.cc:407] Connection header received
0809 12:13:18.152022 (+ 443us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 12:13:18.152028 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 12:13:18.152102 (+ 74us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 12:13:18.152177 (+ 75us) server_negotiation.cc:226] Negotiated authn=SASL
0809 12:13:18.154228 (+ 2051us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 12:13:18.155341 (+ 1113us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 12:13:18.157742 (+ 2401us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 12:13:18.158409 (+ 667us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 12:13:18.158502 (+ 93us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 12:13:18.161978 (+ 3476us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 12:13:18.161987 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 12:13:18.161994 (+ 7us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0809 12:13:18.162035 (+ 41us) server_negotiation.cc:922] calling sasl_server_start()
0809 12:13:18.166483 (+ 4448us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 12:13:18.167379 (+ 896us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 12:13:18.167386 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 12:13:18.167391 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0809 12:13:18.167554 (+ 163us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 12:13:18.168036 (+ 482us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 12:13:18.168045 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 12:13:18.168051 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0809 12:13:18.168477 (+ 426us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 12:13:18.168736 (+ 259us) server_negotiation.cc:1035] Waiting for connection context
0809 12:13:18.169722 (+ 986us) server_negotiation.cc:299] Negotiation successful
0809 12:13:18.169952 (+ 230us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":44}
I20260809 12:13:18.170938 15630 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 12:13:18.130996 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48296 (local address 127.0.222.61:35925)
0809 12:13:18.131195 (+ 199us) server_negotiation.cc:206] Beginning negotiation
0809 12:13:18.131203 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0809 12:13:18.147032 (+ 15829us) server_negotiation.cc:407] Connection header received
0809 12:13:18.147159 (+ 127us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 12:13:18.147167 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 12:13:18.147272 (+ 105us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 12:13:18.147410 (+ 138us) server_negotiation.cc:226] Negotiated authn=SASL
0809 12:13:18.150264 (+ 2854us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 12:13:18.151553 (+ 1289us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 12:13:18.153812 (+ 2259us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 12:13:18.154476 (+ 664us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 12:13:18.154569 (+ 93us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 12:13:18.159752 (+ 5183us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 12:13:18.159760 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 12:13:18.159765 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0809 12:13:18.159809 (+ 44us) server_negotiation.cc:922] calling sasl_server_start()
0809 12:13:18.167292 (+ 7483us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 12:13:18.168145 (+ 853us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 12:13:18.168152 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 12:13:18.168154 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0809 12:13:18.168291 (+ 137us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 12:13:18.168736 (+ 445us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 12:13:18.168741 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 12:13:18.168744 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0809 12:13:18.169125 (+ 381us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 12:13:18.169337 (+ 212us) server_negotiation.cc:1035] Waiting for connection context
0809 12:13:18.170359 (+ 1022us) server_negotiation.cc:299] Negotiation successful
0809 12:13:18.170596 (+ 237us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":60}
I20260809 12:13:18.183957 15468 raft_consensus.cc:348] T 00000000000000000000000000000000 P 962f780b4e6e42209bae1266526c1e08 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "962f780b4e6e42209bae1266526c1e08" member_type: VOTER last_known_addr { host: "127.0.222.62" port: 36165 } } peers { permanent_uuid: "2898ede5262c485c946a1aaa2d62a304" member_type: VOTER last_known_addr { host: "127.0.222.61" port: 35925 } } peers { permanent_uuid: "b3359c45352b4ec9a218caec9a9a949f" member_type: VOTER last_known_addr { host: "127.0.222.60" port: 45255 } }
I20260809 12:13:18.184458 15468 raft_consensus.cc:374] T 00000000000000000000000000000000 P 962f780b4e6e42209bae1266526c1e08 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260809 12:13:18.184620 15468 raft_consensus.cc:729] T 00000000000000000000000000000000 P 962f780b4e6e42209bae1266526c1e08 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 962f780b4e6e42209bae1266526c1e08, State: Initialized, Role: FOLLOWER
I20260809 12:13:18.185402 15468 consensus_queue.cc:260] T 00000000000000000000000000000000 P 962f780b4e6e42209bae1266526c1e08 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "962f780b4e6e42209bae1266526c1e08" member_type: VOTER last_known_addr { host: "127.0.222.62" port: 36165 } } peers { permanent_uuid: "2898ede5262c485c946a1aaa2d62a304" member_type: VOTER last_known_addr { host: "127.0.222.61" port: 35925 } } peers { permanent_uuid: "b3359c45352b4ec9a218caec9a9a949f" member_type: VOTER last_known_addr { host: "127.0.222.60" port: 45255 } }
W20260809 12:13:18.185806 15651 catalog_manager.cc:1558] T 00000000000000000000000000000000 P b3359c45352b4ec9a218caec9a9a949f: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260809 12:13:18.185937 15651 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260809 12:13:18.195525 15654 sys_catalog.cc:455] T 00000000000000000000000000000000 P 962f780b4e6e42209bae1266526c1e08 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "962f780b4e6e42209bae1266526c1e08" member_type: VOTER last_known_addr { host: "127.0.222.62" port: 36165 } } peers { permanent_uuid: "2898ede5262c485c946a1aaa2d62a304" member_type: VOTER last_known_addr { host: "127.0.222.61" port: 35925 } } peers { permanent_uuid: "b3359c45352b4ec9a218caec9a9a949f" member_type: VOTER last_known_addr { host: "127.0.222.60" port: 45255 } } }
I20260809 12:13:18.195999 15654 sys_catalog.cc:458] T 00000000000000000000000000000000 P 962f780b4e6e42209bae1266526c1e08 [sys.catalog]: This master's current role is: FOLLOWER
I20260809 12:13:18.181170 15554 master_runner.cc:186] Error getting master registration for 127.0.222.62:36165: OK, instance_id { permanent_uuid: "962f780b4e6e42209bae1266526c1e08" instance_seqno: 1786277596972222 } error { code: CATALOG_MANAGER_NOT_INITIALIZED status { code: SERVICE_UNAVAILABLE message: "Catalog manager is not initialized. State: Starting" } }
I20260809 12:13:18.196745 15554 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20260809 12:13:18.197160 15468 sys_catalog.cc:564] T 00000000000000000000000000000000 P 962f780b4e6e42209bae1266526c1e08 [sys.catalog]: configured and running, proceeding with master startup.
I20260809 12:13:18.197844 15638 raft_consensus.cc:482] T 00000000000000000000000000000000 P b3359c45352b4ec9a218caec9a9a949f [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260809 12:13:18.198124 15638 raft_consensus.cc:504] T 00000000000000000000000000000000 P b3359c45352b4ec9a218caec9a9a949f [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "962f780b4e6e42209bae1266526c1e08" member_type: VOTER last_known_addr { host: "127.0.222.62" port: 36165 } } peers { permanent_uuid: "2898ede5262c485c946a1aaa2d62a304" member_type: VOTER last_known_addr { host: "127.0.222.61" port: 35925 } } peers { permanent_uuid: "b3359c45352b4ec9a218caec9a9a949f" member_type: VOTER last_known_addr { host: "127.0.222.60" port: 45255 } }
I20260809 12:13:18.199779 15638 leader_election.cc:288] T 00000000000000000000000000000000 P b3359c45352b4ec9a218caec9a9a949f [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 962f780b4e6e42209bae1266526c1e08 (127.0.222.62:36165), 2898ede5262c485c946a1aaa2d62a304 (127.0.222.61:35925)
I20260809 12:13:18.200896 15440 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "b3359c45352b4ec9a218caec9a9a949f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "962f780b4e6e42209bae1266526c1e08" is_pre_election: true
I20260809 12:13:18.201812 15440 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 962f780b4e6e42209bae1266526c1e08 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b3359c45352b4ec9a218caec9a9a949f in term 0.
I20260809 12:13:18.202600 15570 leader_election.cc:302] T 00000000000000000000000000000000 P b3359c45352b4ec9a218caec9a9a949f [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 962f780b4e6e42209bae1266526c1e08, b3359c45352b4ec9a218caec9a9a949f; no voters:
I20260809 12:13:18.203074 15638 raft_consensus.cc:2781] T 00000000000000000000000000000000 P b3359c45352b4ec9a218caec9a9a949f [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260809 12:13:18.203241 15638 raft_consensus.cc:482] T 00000000000000000000000000000000 P b3359c45352b4ec9a218caec9a9a949f [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260809 12:13:18.203351 15638 raft_consensus.cc:3037] T 00000000000000000000000000000000 P b3359c45352b4ec9a218caec9a9a949f [term 0 FOLLOWER]: Advancing to term 1
I20260809 12:13:18.207870 15638 raft_consensus.cc:504] T 00000000000000000000000000000000 P b3359c45352b4ec9a218caec9a9a949f [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "962f780b4e6e42209bae1266526c1e08" member_type: VOTER last_known_addr { host: "127.0.222.62" port: 36165 } } peers { permanent_uuid: "2898ede5262c485c946a1aaa2d62a304" member_type: VOTER last_known_addr { host: "127.0.222.61" port: 35925 } } peers { permanent_uuid: "b3359c45352b4ec9a218caec9a9a949f" member_type: VOTER last_known_addr { host: "127.0.222.60" port: 45255 } }
I20260809 12:13:18.208958 15440 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "b3359c45352b4ec9a218caec9a9a949f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "962f780b4e6e42209bae1266526c1e08"
I20260809 12:13:18.209178 15440 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 962f780b4e6e42209bae1266526c1e08 [term 0 FOLLOWER]: Advancing to term 1
W20260809 12:13:18.211452 15418 tablet.cc:2369] T 00000000000000000000000000000000 P 962f780b4e6e42209bae1266526c1e08: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260809 12:13:18.212617 15440 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 962f780b4e6e42209bae1266526c1e08 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b3359c45352b4ec9a218caec9a9a949f in term 1.
I20260809 12:13:18.213050 15570 leader_election.cc:302] T 00000000000000000000000000000000 P b3359c45352b4ec9a218caec9a9a949f [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 962f780b4e6e42209bae1266526c1e08, b3359c45352b4ec9a218caec9a9a949f; no voters:
I20260809 12:13:18.213939 15638 leader_election.cc:288] T 00000000000000000000000000000000 P b3359c45352b4ec9a218caec9a9a949f [CANDIDATE]: Term 1 election: Requested vote from peers 962f780b4e6e42209bae1266526c1e08 (127.0.222.62:36165), 2898ede5262c485c946a1aaa2d62a304 (127.0.222.61:35925)
I20260809 12:13:18.214823 15638 raft_consensus.cc:2781] T 00000000000000000000000000000000 P b3359c45352b4ec9a218caec9a9a949f [term 1 FOLLOWER]: Leader election won for term 1
I20260809 12:13:18.215006 15638 raft_consensus.cc:686] T 00000000000000000000000000000000 P b3359c45352b4ec9a218caec9a9a949f [term 1 LEADER]: Becoming Leader. State: Replica: b3359c45352b4ec9a218caec9a9a949f, State: Running, Role: LEADER
I20260809 12:13:18.215562 15638 consensus_queue.cc:237] T 00000000000000000000000000000000 P b3359c45352b4ec9a218caec9a9a949f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "962f780b4e6e42209bae1266526c1e08" member_type: VOTER last_known_addr { host: "127.0.222.62" port: 36165 } } peers { permanent_uuid: "2898ede5262c485c946a1aaa2d62a304" member_type: VOTER last_known_addr { host: "127.0.222.61" port: 35925 } } peers { permanent_uuid: "b3359c45352b4ec9a218caec9a9a949f" member_type: VOTER last_known_addr { host: "127.0.222.60" port: 45255 } }
I20260809 12:13:18.222491 15638 sys_catalog.cc:455] T 00000000000000000000000000000000 P b3359c45352b4ec9a218caec9a9a949f [sys.catalog]: SysCatalogTable state changed. Reason: New leader b3359c45352b4ec9a218caec9a9a949f. Latest consensus state: current_term: 1 leader_uuid: "b3359c45352b4ec9a218caec9a9a949f" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "962f780b4e6e42209bae1266526c1e08" member_type: VOTER last_known_addr { host: "127.0.222.62" port: 36165 } } peers { permanent_uuid: "2898ede5262c485c946a1aaa2d62a304" member_type: VOTER last_known_addr { host: "127.0.222.61" port: 35925 } } peers { permanent_uuid: "b3359c45352b4ec9a218caec9a9a949f" member_type: VOTER last_known_addr { host: "127.0.222.60" port: 45255 } } }
I20260809 12:13:18.222810 15638 sys_catalog.cc:458] T 00000000000000000000000000000000 P b3359c45352b4ec9a218caec9a9a949f [sys.catalog]: This master's current role is: LEADER
I20260809 12:13:18.222915 15547 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.222.60" port: 45255 } attempt: 5
I20260809 12:13:18.228796 15661 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
Aug 09 12:13:18 dist-test-slave-2w3w krb5kdc[15382](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786277597, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.222.61@KRBTEST.COM for kudu/127.0.222.60@KRBTEST.COM
I20260809 12:13:18.242632 15661 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260809 12:13:18.247735 15549 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 12:13:18.223542 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.222.60:45255 (local address 127.0.0.1:39954)
0809 12:13:18.223912 (+ 370us) negotiation.cc:107] Waiting for socket to connect
0809 12:13:18.223935 (+ 23us) client_negotiation.cc:174] Beginning negotiation
0809 12:13:18.224062 (+ 127us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 12:13:18.224927 (+ 865us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 12:13:18.224935 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 12:13:18.225510 (+ 575us) client_negotiation.cc:189] Negotiated authn=SASL
0809 12:13:18.225756 (+ 246us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 12:13:18.225772 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 12:13:18.227302 (+ 1530us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 12:13:18.227308 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 12:13:18.228758 (+ 1450us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 12:13:18.228782 (+ 24us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 12:13:18.230083 (+ 1301us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 12:13:18.230089 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 12:13:18.230305 (+ 216us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 12:13:18.231174 (+ 869us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0809 12:13:18.231209 (+ 35us) client_negotiation.cc:656] calling sasl_client_start()
0809 12:13:18.235136 (+ 3927us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0809 12:13:18.240988 (+ 5852us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0809 12:13:18.240996 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0809 12:13:18.241003 (+ 7us) client_negotiation.cc:760] calling sasl_client_step()
0809 12:13:18.241905 (+ 902us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0809 12:13:18.245051 (+ 3146us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0809 12:13:18.245060 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0809 12:13:18.245067 (+ 7us) client_negotiation.cc:760] calling sasl_client_step()
0809 12:13:18.245340 (+ 273us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0809 12:13:18.246301 (+ 961us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0809 12:13:18.246309 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0809 12:13:18.246541 (+ 232us) client_negotiation.cc:769] Sending connection context
0809 12:13:18.246735 (+ 194us) client_negotiation.cc:240] Negotiation successful
0809 12:13:18.247156 (+ 421us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":229}
I20260809 12:13:18.249256 15626 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 12:13:18.224271 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39954 (local address 127.0.222.60:45255)
0809 12:13:18.224439 (+ 168us) server_negotiation.cc:206] Beginning negotiation
0809 12:13:18.224447 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0809 12:13:18.224470 (+ 23us) server_negotiation.cc:407] Connection header received
0809 12:13:18.224526 (+ 56us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 12:13:18.224535 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 12:13:18.224621 (+ 86us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 12:13:18.224716 (+ 95us) server_negotiation.cc:226] Negotiated authn=SASL
0809 12:13:18.225951 (+ 1235us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 12:13:18.227108 (+ 1157us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 12:13:18.228947 (+ 1839us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 12:13:18.229675 (+ 728us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 12:13:18.229847 (+ 172us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 12:13:18.235421 (+ 5574us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 12:13:18.235432 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 12:13:18.235440 (+ 8us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0809 12:13:18.235492 (+ 52us) server_negotiation.cc:922] calling sasl_server_start()
0809 12:13:18.240796 (+ 5304us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 12:13:18.242110 (+ 1314us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 12:13:18.242118 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 12:13:18.242124 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0809 12:13:18.242372 (+ 248us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 12:13:18.245522 (+ 3150us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 12:13:18.245530 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 12:13:18.245536 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0809 12:13:18.245984 (+ 448us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 12:13:18.246157 (+ 173us) server_negotiation.cc:1035] Waiting for connection context
0809 12:13:18.248208 (+ 2051us) server_negotiation.cc:299] Negotiation successful
0809 12:13:18.248539 (+ 331us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":59}
I20260809 12:13:18.255709 15547 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 2898ede5262c485c946a1aaa2d62a304: Bootstrap starting.
I20260809 12:13:18.260097 15547 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 2898ede5262c485c946a1aaa2d62a304: Neither blocks nor log segments found. Creating new log.
I20260809 12:13:18.261452 15547 log.cc:824] T 00000000000000000000000000000000 P 2898ede5262c485c946a1aaa2d62a304: Log is configured to *not* fsync() on all Append() calls
I20260809 12:13:18.266134 15670 catalog_manager.cc:795] Waiting for catalog manager background task thread to start: Service unavailable: Catalog manager is not initialized. State: Starting
W20260809 12:13:18.275125 15575 tablet.cc:2369] T 00000000000000000000000000000000 P b3359c45352b4ec9a218caec9a9a949f: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260809 12:13:18.280113 15440 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 962f780b4e6e42209bae1266526c1e08 [term 1 FOLLOWER]: Refusing update from remote peer b3359c45352b4ec9a218caec9a9a949f: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260809 12:13:18.281121 15659 consensus_queue.cc:1035] T 00000000000000000000000000000000 P b3359c45352b4ec9a218caec9a9a949f [LEADER]: Connected to new peer: Peer: permanent_uuid: "962f780b4e6e42209bae1266526c1e08" member_type: VOTER last_known_addr { host: "127.0.222.62" port: 36165 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260809 12:13:18.271358 15396 master_runner.cc:186] Error getting master registration for 127.0.222.61:35925: OK, instance_id { permanent_uuid: "2898ede5262c485c946a1aaa2d62a304" instance_seqno: 1786277597494018 } error { code: CATALOG_MANAGER_NOT_INITIALIZED status { code: SERVICE_UNAVAILABLE message: "Catalog manager is not initialized. State: Starting" } }
I20260809 12:13:18.281955 15396 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20260809 12:13:18.285017 15547 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 2898ede5262c485c946a1aaa2d62a304: No bootstrap required, opened a new log
I20260809 12:13:18.296468 15547 raft_consensus.cc:348] T 00000000000000000000000000000000 P 2898ede5262c485c946a1aaa2d62a304 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "962f780b4e6e42209bae1266526c1e08" member_type: VOTER last_known_addr { host: "127.0.222.62" port: 36165 } } peers { permanent_uuid: "2898ede5262c485c946a1aaa2d62a304" member_type: VOTER last_known_addr { host: "127.0.222.61" port: 35925 } } peers { permanent_uuid: "b3359c45352b4ec9a218caec9a9a949f" member_type: VOTER last_known_addr { host: "127.0.222.60" port: 45255 } }
I20260809 12:13:18.297043 15547 raft_consensus.cc:374] T 00000000000000000000000000000000 P 2898ede5262c485c946a1aaa2d62a304 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260809 12:13:18.297219 15547 raft_consensus.cc:729] T 00000000000000000000000000000000 P 2898ede5262c485c946a1aaa2d62a304 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2898ede5262c485c946a1aaa2d62a304, State: Initialized, Role: FOLLOWER
I20260809 12:13:18.298135 15547 consensus_queue.cc:260] T 00000000000000000000000000000000 P 2898ede5262c485c946a1aaa2d62a304 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "962f780b4e6e42209bae1266526c1e08" member_type: VOTER last_known_addr { host: "127.0.222.62" port: 36165 } } peers { permanent_uuid: "2898ede5262c485c946a1aaa2d62a304" member_type: VOTER last_known_addr { host: "127.0.222.61" port: 35925 } } peers { permanent_uuid: "b3359c45352b4ec9a218caec9a9a949f" member_type: VOTER last_known_addr { host: "127.0.222.60" port: 45255 } }
I20260809 12:13:18.301378 15654 sys_catalog.cc:455] T 00000000000000000000000000000000 P 962f780b4e6e42209bae1266526c1e08 [sys.catalog]: SysCatalogTable state changed. Reason: New leader b3359c45352b4ec9a218caec9a9a949f. Latest consensus state: current_term: 1 leader_uuid: "b3359c45352b4ec9a218caec9a9a949f" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "962f780b4e6e42209bae1266526c1e08" member_type: VOTER last_known_addr { host: "127.0.222.62" port: 36165 } } peers { permanent_uuid: "2898ede5262c485c946a1aaa2d62a304" member_type: VOTER last_known_addr { host: "127.0.222.61" port: 35925 } } peers { permanent_uuid: "b3359c45352b4ec9a218caec9a9a949f" member_type: VOTER last_known_addr { host: "127.0.222.60" port: 45255 } } }
I20260809 12:13:18.301677 15654 sys_catalog.cc:458] T 00000000000000000000000000000000 P 962f780b4e6e42209bae1266526c1e08 [sys.catalog]: This master's current role is: FOLLOWER
I20260809 12:13:18.303557 15671 mvcc.cc:205] Tried to move back new op lower bound from 7316593042502074368 to 7316593042307440640. Current Snapshot: MvccSnapshot[applied={T|T < 7316593042502074368}]
I20260809 12:13:18.304322 15675 sys_catalog.cc:455] T 00000000000000000000000000000000 P 2898ede5262c485c946a1aaa2d62a304 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "962f780b4e6e42209bae1266526c1e08" member_type: VOTER last_known_addr { host: "127.0.222.62" port: 36165 } } peers { permanent_uuid: "2898ede5262c485c946a1aaa2d62a304" member_type: VOTER last_known_addr { host: "127.0.222.61" port: 35925 } } peers { permanent_uuid: "b3359c45352b4ec9a218caec9a9a949f" member_type: VOTER last_known_addr { host: "127.0.222.60" port: 45255 } } }
I20260809 12:13:18.306578 15675 sys_catalog.cc:458] T 00000000000000000000000000000000 P 2898ede5262c485c946a1aaa2d62a304 [sys.catalog]: This master's current role is: FOLLOWER
I20260809 12:13:18.306449 15547 sys_catalog.cc:564] T 00000000000000000000000000000000 P 2898ede5262c485c946a1aaa2d62a304 [sys.catalog]: configured and running, proceeding with master startup.
I20260809 12:13:18.306519 15517 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "b3359c45352b4ec9a218caec9a9a949f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2898ede5262c485c946a1aaa2d62a304"
I20260809 12:13:18.307367 15516 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 2898ede5262c485c946a1aaa2d62a304 [term 0 FOLLOWER]: Advancing to term 1
I20260809 12:13:18.309782 15659 sys_catalog.cc:455] T 00000000000000000000000000000000 P b3359c45352b4ec9a218caec9a9a949f [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "b3359c45352b4ec9a218caec9a9a949f" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "962f780b4e6e42209bae1266526c1e08" member_type: VOTER last_known_addr { host: "127.0.222.62" port: 36165 } } peers { permanent_uuid: "2898ede5262c485c946a1aaa2d62a304" member_type: VOTER last_known_addr { host: "127.0.222.61" port: 35925 } } peers { permanent_uuid: "b3359c45352b4ec9a218caec9a9a949f" member_type: VOTER last_known_addr { host: "127.0.222.60" port: 45255 } } }
I20260809 12:13:18.310151 15659 sys_catalog.cc:458] T 00000000000000000000000000000000 P b3359c45352b4ec9a218caec9a9a949f [sys.catalog]: This master's current role is: LEADER
I20260809 12:13:18.311406 15516 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 2898ede5262c485c946a1aaa2d62a304 [term 1 FOLLOWER]: Refusing update from remote peer b3359c45352b4ec9a218caec9a9a949f: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260809 12:13:18.311966 15661 catalog_manager.cc:1347] Generated new cluster ID: 28b8af55bf66471f894845a35cd17309
I20260809 12:13:18.312075 15661 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260809 12:13:18.311949 15518 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "b3359c45352b4ec9a218caec9a9a949f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2898ede5262c485c946a1aaa2d62a304" is_pre_election: true
I20260809 12:13:18.312252 15638 consensus_queue.cc:1035] T 00000000000000000000000000000000 P b3359c45352b4ec9a218caec9a9a949f [LEADER]: Connected to new peer: Peer: permanent_uuid: "2898ede5262c485c946a1aaa2d62a304" member_type: VOTER last_known_addr { host: "127.0.222.61" port: 35925 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260809 12:13:18.312371 15518 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 2898ede5262c485c946a1aaa2d62a304 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b3359c45352b4ec9a218caec9a9a949f in term 1.
I20260809 12:13:18.318733 15673 mvcc.cc:205] Tried to move back new op lower bound from 7316593042502074368 to 7316593042307440640. Current Snapshot: MvccSnapshot[applied={T|T < 7316593042502074368}]
I20260809 12:13:18.321491 15654 sys_catalog.cc:455] T 00000000000000000000000000000000 P 962f780b4e6e42209bae1266526c1e08 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "b3359c45352b4ec9a218caec9a9a949f" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "962f780b4e6e42209bae1266526c1e08" member_type: VOTER last_known_addr { host: "127.0.222.62" port: 36165 } } peers { permanent_uuid: "2898ede5262c485c946a1aaa2d62a304" member_type: VOTER last_known_addr { host: "127.0.222.61" port: 35925 } } peers { permanent_uuid: "b3359c45352b4ec9a218caec9a9a949f" member_type: VOTER last_known_addr { host: "127.0.222.60" port: 45255 } } }
I20260809 12:13:18.321822 15654 sys_catalog.cc:458] T 00000000000000000000000000000000 P 962f780b4e6e42209bae1266526c1e08 [sys.catalog]: This master's current role is: FOLLOWER
I20260809 12:13:18.342988 15675 sys_catalog.cc:455] T 00000000000000000000000000000000 P 2898ede5262c485c946a1aaa2d62a304 [sys.catalog]: SysCatalogTable state changed. Reason: New leader b3359c45352b4ec9a218caec9a9a949f. Latest consensus state: current_term: 1 leader_uuid: "b3359c45352b4ec9a218caec9a9a949f" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "962f780b4e6e42209bae1266526c1e08" member_type: VOTER last_known_addr { host: "127.0.222.62" port: 36165 } } peers { permanent_uuid: "2898ede5262c485c946a1aaa2d62a304" member_type: VOTER last_known_addr { host: "127.0.222.61" port: 35925 } } peers { permanent_uuid: "b3359c45352b4ec9a218caec9a9a949f" member_type: VOTER last_known_addr { host: "127.0.222.60" port: 45255 } } }
I20260809 12:13:18.343461 15675 sys_catalog.cc:458] T 00000000000000000000000000000000 P 2898ede5262c485c946a1aaa2d62a304 [sys.catalog]: This master's current role is: FOLLOWER
I20260809 12:13:18.344089 15685 sys_catalog.cc:455] T 00000000000000000000000000000000 P 2898ede5262c485c946a1aaa2d62a304 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "b3359c45352b4ec9a218caec9a9a949f" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "962f780b4e6e42209bae1266526c1e08" member_type: VOTER last_known_addr { host: "127.0.222.62" port: 36165 } } peers { permanent_uuid: "2898ede5262c485c946a1aaa2d62a304" member_type: VOTER last_known_addr { host: "127.0.222.61" port: 35925 } } peers { permanent_uuid: "b3359c45352b4ec9a218caec9a9a949f" member_type: VOTER last_known_addr { host: "127.0.222.60" port: 45255 } } }
I20260809 12:13:18.345466 15685 sys_catalog.cc:458] T 00000000000000000000000000000000 P 2898ede5262c485c946a1aaa2d62a304 [sys.catalog]: This master's current role is: FOLLOWER
I20260809 12:13:18.364809 15692 catalog_manager.cc:1259] Loaded cluster ID: 28b8af55bf66471f894845a35cd17309
I20260809 12:13:18.364959 15692 catalog_manager.cc:1552] T 00000000000000000000000000000000 P 2898ede5262c485c946a1aaa2d62a304: loading cluster ID for follower catalog manager: success
W20260809 12:13:18.366227 15692 catalog_manager.cc:1576] T 00000000000000000000000000000000 P 2898ede5262c485c946a1aaa2d62a304: acquiring CA information for follower catalog manager: Not found: root CA entry not found
W20260809 12:13:18.366410 15692 catalog_manager.cc:873] Not found: root CA entry not found: failed to prepare follower catalog manager, will retry
I20260809 12:13:18.369837 15638 sys_catalog.cc:455] T 00000000000000000000000000000000 P b3359c45352b4ec9a218caec9a9a949f [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "b3359c45352b4ec9a218caec9a9a949f" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "962f780b4e6e42209bae1266526c1e08" member_type: VOTER last_known_addr { host: "127.0.222.62" port: 36165 } } peers { permanent_uuid: "2898ede5262c485c946a1aaa2d62a304" member_type: VOTER last_known_addr { host: "127.0.222.61" port: 35925 } } peers { permanent_uuid: "b3359c45352b4ec9a218caec9a9a949f" member_type: VOTER last_known_addr { host: "127.0.222.60" port: 45255 } } }
I20260809 12:13:18.382908 15638 sys_catalog.cc:458] T 00000000000000000000000000000000 P b3359c45352b4ec9a218caec9a9a949f [sys.catalog]: This master's current role is: LEADER
I20260809 12:13:18.383242 15661 catalog_manager.cc:1370] Generated new certificate authority record
I20260809 12:13:18.384889 15661 catalog_manager.cc:1504] Loading token signing keys...
I20260809 12:13:18.401849 15661 catalog_manager.cc:5868] T 00000000000000000000000000000000 P b3359c45352b4ec9a218caec9a9a949f: Generated new TSK 0
I20260809 12:13:18.403083 15661 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260809 12:13:19.270653 15670 catalog_manager.cc:1259] Loaded cluster ID: 28b8af55bf66471f894845a35cd17309
I20260809 12:13:19.270805 15670 catalog_manager.cc:1552] T 00000000000000000000000000000000 P 962f780b4e6e42209bae1266526c1e08: loading cluster ID for follower catalog manager: success
I20260809 12:13:19.273494 15670 catalog_manager.cc:1574] T 00000000000000000000000000000000 P 962f780b4e6e42209bae1266526c1e08: acquiring CA information for follower catalog manager: success
I20260809 12:13:19.275549 15670 catalog_manager.cc:1602] T 00000000000000000000000000000000 P 962f780b4e6e42209bae1266526c1e08: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260809 12:13:19.369779 15692 catalog_manager.cc:1574] T 00000000000000000000000000000000 P 2898ede5262c485c946a1aaa2d62a304: acquiring CA information for follower catalog manager: success
I20260809 12:13:19.371747 15692 catalog_manager.cc:1602] T 00000000000000000000000000000000 P 2898ede5262c485c946a1aaa2d62a304: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260809 12:13:19.495410 888 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskgH5Pjm/build/asan/bin/kudu pid 15396
I20260809 12:13:19.496806 15463 generic_service.cc:196] Checking for leaks (request via RPC)
I20260809 12:13:19.612836 888 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskgH5Pjm/build/asan/bin/kudu with pid 15396
I20260809 12:13:19.637117 888 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskgH5Pjm/build/asan/bin/kudu pid 15475
I20260809 12:13:19.638175 15542 generic_service.cc:196] Checking for leaks (request via RPC)
I20260809 12:13:19.647140 15702 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 12:13:19.646205 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.222.62:36165 (local address 127.0.0.1:48866)
0809 12:13:19.646733 (+ 528us) negotiation.cc:107] Waiting for socket to connect
0809 12:13:19.646881 (+ 148us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.222.62:36165: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":367,"thread_start_us":262,"threads_started":1}
W20260809 12:13:19.647439 15570 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.222.62:36165: connect: Connection refused (error 111)
I20260809 12:13:19.648859 15702 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 12:13:19.648483 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.222.62:36165 (local address 127.0.0.1:48872)
0809 12:13:19.648653 (+ 170us) negotiation.cc:107] Waiting for socket to connect
0809 12:13:19.648740 (+ 87us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.222.62:36165: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":72}
W20260809 12:13:19.649041 15570 consensus_peers.cc:487] T 00000000000000000000000000000000 P b3359c45352b4ec9a218caec9a9a949f -> Peer 962f780b4e6e42209bae1266526c1e08 (127.0.222.62:36165): Couldn't send request to peer 962f780b4e6e42209bae1266526c1e08. Status: Network error: Client connection negotiation failed: client connection to 127.0.222.62:36165: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20260809 12:13:19.748533 888 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskgH5Pjm/build/asan/bin/kudu with pid 15475
I20260809 12:13:19.772635 888 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskgH5Pjm/build/asan/bin/kudu pid 15554
I20260809 12:13:19.773834 15618 generic_service.cc:196] Checking for leaks (request via RPC)
I20260809 12:13:19.884528 888 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskgH5Pjm/build/asan/bin/kudu with pid 15554
2026-08-09T12:13:19Z chronyd exiting
[ OK ] SecurityITest.IPKICACert (4842 ms)
[ RUN ] SecurityITest.IPKICACertWebServerEndPoints
Loading random data
Initializing database '/tmp/dist-test-taskgH5Pjm/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1786277382073938-888-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 09 12:13:19 dist-test-slave-2w3w krb5kdc[15707](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 09 12:13:19 dist-test-slave-2w3w krb5kdc[15707](info): set up 2 sockets
Aug 09 12:13:19 dist-test-slave-2w3w krb5kdc[15707](info): commencing operation
krb5kdc: starting...
W20260809 12:13:21.062876 888 mini_kdc.cc:121] Time spent starting KDC: real 1.133s user 0.000s sys 0.037s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskgH5Pjm/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1786277382073938-888-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskgH5Pjm/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1786277382073938-888-0/krb5kdc/test-user.keytab.
Aug 09 12:13:21 dist-test-slave-2w3w krb5kdc[15707](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786277601, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-09T12:13:21Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-09T12:13:21Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.222.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.222.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.222.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskgH5Pjm/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1786277382073938-888-0/krb5kdc/kudu_127.0.222.62.keytab.
Entry for principal kudu/127.0.222.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskgH5Pjm/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1786277382073938-888-0/krb5kdc/kudu_127.0.222.62.keytab.
I20260809 12:13:21.346493 888 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskgH5Pjm/build/asan/bin/kudu
/tmp/dist-test-taskgH5Pjm/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskgH5Pjm/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1786277382073938-888-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskgH5Pjm/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1786277382073938-888-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskgH5Pjm/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1786277382073938-888-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskgH5Pjm/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1786277382073938-888-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.222.62:43629
--webserver_interface=127.0.222.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskgH5Pjm/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1786277382073938-888-0/krb5kdc/kudu_127.0.222.62.keytab
--principal=kudu/127.0.222.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.222.20:35959
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.222.62:43629
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskgH5Pjm/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1786277382073938-888-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskgH5Pjm/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1786277382073938-888-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskgH5Pjm/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1786277382073938-888-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260809 12:13:21.690863 15721 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260809 12:13:21.691253 15721 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260809 12:13:21.691354 15721 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260809 12:13:21.698781 15721 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260809 12:13:21.698891 15721 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260809 12:13:21.698935 15721 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260809 12:13:21.699054 15721 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260809 12:13:21.699110 15721 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260809 12:13:21.709233 15721 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.222.20:35959
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskgH5Pjm/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1786277382073938-888-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskgH5Pjm/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1786277382073938-888-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.222.62:43629
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskgH5Pjm/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1786277382073938-888-0/krb5kdc/kudu_127.0.222.62.keytab
--principal=kudu/127.0.222.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.222.62:43629
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskgH5Pjm/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1786277382073938-888-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.222.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskgH5Pjm/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1786277382073938-888-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 09 Aug 2026 11:06:41 UTC on ccycloud.quasar-bwoogy.root.comops.site
build id 4575991
ASAN enabled
I20260809 12:13:21.711112 15721 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260809 12:13:21.712517 15721 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260809 12:13:21.724440 15726 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 12:13:21.724411 15729 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 12:13:21.724471 15727 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260809 12:13:21.726147 15721 server_base.cc:1034] running on GCE node
I20260809 12:13:21.726915 15721 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260809 12:13:21.728603 15721 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260809 12:13:21.729812 15721 hybrid_clock.cc:648] HybridClock initialized: now 1786277601729753 us; error 58 us; skew 500 ppm
I20260809 12:13:21.730131 15721 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 09 12:13:21 dist-test-slave-2w3w krb5kdc[15707](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786277601, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.222.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260809 12:13:21.738528 15721 init.cc:377] Logged in from keytab as kudu/127.0.222.62@KRBTEST.COM (short username kudu)
I20260809 12:13:21.740044 15721 webserver.cc:460] Webserver started at http://127.0.222.62:38527/ using document root <none> and password file <none>
I20260809 12:13:21.740753 15721 fs_manager.cc:359] Metadata directory not provided
I20260809 12:13:21.740855 15721 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260809 12:13:21.741114 15721 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260809 12:13:21.744351 15721 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskgH5Pjm/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1786277382073938-888-0/minicluster-data/master-0/data/instance:
uuid: "448d6c5f1d0f4b498c89ed2d0b29758d"
format_stamp: "Formatted at 2026-08-09 12:13:21 on dist-test-slave-2w3w"
server_key: "e66bbcd24f62761f4a832572579823d3"
server_key_iv: "4556e304042c9c3cf66df6c2a8326441"
server_key_version: "encryptionkey@0"
I20260809 12:13:21.745090 15721 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskgH5Pjm/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1786277382073938-888-0/minicluster-data/master-0/wal/instance:
uuid: "448d6c5f1d0f4b498c89ed2d0b29758d"
format_stamp: "Formatted at 2026-08-09 12:13:21 on dist-test-slave-2w3w"
server_key: "e66bbcd24f62761f4a832572579823d3"
server_key_iv: "4556e304042c9c3cf66df6c2a8326441"
server_key_version: "encryptionkey@0"
I20260809 12:13:21.750415 15721 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.000s
I20260809 12:13:21.753964 15736 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260809 12:13:21.755429 15721 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260809 12:13:21.755620 15721 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskgH5Pjm/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1786277382073938-888-0/minicluster-data/master-0/data,/tmp/dist-test-taskgH5Pjm/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1786277382073938-888-0/minicluster-data/master-0/wal
uuid: "448d6c5f1d0f4b498c89ed2d0b29758d"
format_stamp: "Formatted at 2026-08-09 12:13:21 on dist-test-slave-2w3w"
server_key: "e66bbcd24f62761f4a832572579823d3"
server_key_iv: "4556e304042c9c3cf66df6c2a8326441"
server_key_version: "encryptionkey@0"
I20260809 12:13:21.755829 15721 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskgH5Pjm/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1786277382073938-888-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskgH5Pjm/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1786277382073938-888-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskgH5Pjm/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1786277382073938-888-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260809 12:13:21.774890 15721 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260809 12:13:21.779018 15721 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260809 12:13:21.779320 15721 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260809 12:13:21.798306 15721 rpc_server.cc:307] RPC server started. Bound to: 127.0.222.62:43629
I20260809 12:13:21.798343 15791 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.222.62:43629 every 8 connection(s)
I20260809 12:13:21.799710 15721 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskgH5Pjm/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1786277382073938-888-0/minicluster-data/master-0/data/info.pb
I20260809 12:13:21.803463 15792 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260809 12:13:21.804090 888 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskgH5Pjm/build/asan/bin/kudu as pid 15721
I20260809 12:13:21.804308 888 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskgH5Pjm/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1786277382073938-888-0/minicluster-data/master-0/wal/instance
I20260809 12:13:21.805069 888 external_mini_cluster.cc:1428] Setting key cc4196f865485c3560a90f587db209f9
I20260809 12:13:21.817533 15792 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 448d6c5f1d0f4b498c89ed2d0b29758d: Bootstrap starting.
Aug 09 12:13:21 dist-test-slave-2w3w krb5kdc[15707](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786277601, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.222.62@KRBTEST.COM
I20260809 12:13:21.821410 15792 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 448d6c5f1d0f4b498c89ed2d0b29758d: Neither blocks nor log segments found. Creating new log.
I20260809 12:13:21.822643 15792 log.cc:824] T 00000000000000000000000000000000 P 448d6c5f1d0f4b498c89ed2d0b29758d: Log is configured to *not* fsync() on all Append() calls
I20260809 12:13:21.825644 15792 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 448d6c5f1d0f4b498c89ed2d0b29758d: No bootstrap required, opened a new log
I20260809 12:13:21.829506 15795 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 12:13:21.806893 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46060 (local address 127.0.222.62:43629)
0809 12:13:21.807795 (+ 902us) server_negotiation.cc:206] Beginning negotiation
0809 12:13:21.807809 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0809 12:13:21.807856 (+ 47us) server_negotiation.cc:407] Connection header received
0809 12:13:21.809633 (+ 1777us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 12:13:21.809658 (+ 25us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 12:13:21.810031 (+ 373us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 12:13:21.810359 (+ 328us) server_negotiation.cc:226] Negotiated authn=SASL
0809 12:13:21.812335 (+ 1976us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 12:13:21.813625 (+ 1290us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 12:13:21.815647 (+ 2022us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 12:13:21.816309 (+ 662us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 12:13:21.816497 (+ 188us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 12:13:21.821619 (+ 5122us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 12:13:21.821629 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 12:13:21.821642 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0809 12:13:21.821682 (+ 40us) server_negotiation.cc:922] calling sasl_server_start()
0809 12:13:21.825227 (+ 3545us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 12:13:21.826014 (+ 787us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 12:13:21.826021 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 12:13:21.826027 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0809 12:13:21.826207 (+ 180us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 12:13:21.826627 (+ 420us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 12:13:21.826631 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 12:13:21.826633 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0809 12:13:21.827243 (+ 610us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 12:13:21.827607 (+ 364us) server_negotiation.cc:1035] Waiting for connection context
0809 12:13:21.828124 (+ 517us) server_negotiation.cc:299] Negotiation successful
0809 12:13:21.828532 (+ 408us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":693,"thread_start_us":367,"threads_started":1}
I20260809 12:13:21.833449 888 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260809 12:13:21.834034 15792 raft_consensus.cc:348] T 00000000000000000000000000000000 P 448d6c5f1d0f4b498c89ed2d0b29758d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "448d6c5f1d0f4b498c89ed2d0b29758d" member_type: VOTER last_known_addr { host: "127.0.222.62" port: 43629 } }
I20260809 12:13:21.834467 15792 raft_consensus.cc:374] T 00000000000000000000000000000000 P 448d6c5f1d0f4b498c89ed2d0b29758d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260809 12:13:21.834568 15792 raft_consensus.cc:729] T 00000000000000000000000000000000 P 448d6c5f1d0f4b498c89ed2d0b29758d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 448d6c5f1d0f4b498c89ed2d0b29758d, State: Initialized, Role: FOLLOWER
I20260809 12:13:21.835237 15792 consensus_queue.cc:260] T 00000000000000000000000000000000 P 448d6c5f1d0f4b498c89ed2d0b29758d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "448d6c5f1d0f4b498c89ed2d0b29758d" member_type: VOTER last_known_addr { host: "127.0.222.62" port: 43629 } }
I20260809 12:13:21.835510 15792 raft_consensus.cc:388] T 00000000000000000000000000000000 P 448d6c5f1d0f4b498c89ed2d0b29758d [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260809 12:13:21.835642 15792 raft_consensus.cc:482] T 00000000000000000000000000000000 P 448d6c5f1d0f4b498c89ed2d0b29758d [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260809 12:13:21.835800 15792 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 448d6c5f1d0f4b498c89ed2d0b29758d [term 0 FOLLOWER]: Advancing to term 1
I20260809 12:13:21.838318 15792 raft_consensus.cc:504] T 00000000000000000000000000000000 P 448d6c5f1d0f4b498c89ed2d0b29758d [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "448d6c5f1d0f4b498c89ed2d0b29758d" member_type: VOTER last_known_addr { host: "127.0.222.62" port: 43629 } }
I20260809 12:13:21.838769 15792 leader_election.cc:302] T 00000000000000000000000000000000 P 448d6c5f1d0f4b498c89ed2d0b29758d [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 448d6c5f1d0f4b498c89ed2d0b29758d; no voters:
I20260809 12:13:21.839398 15792 leader_election.cc:288] T 00000000000000000000000000000000 P 448d6c5f1d0f4b498c89ed2d0b29758d [CANDIDATE]: Term 1 election: Requested vote from peers
I20260809 12:13:21.839550 15797 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 448d6c5f1d0f4b498c89ed2d0b29758d [term 1 FOLLOWER]: Leader election won for term 1
I20260809 12:13:21.840263 15797 raft_consensus.cc:686] T 00000000000000000000000000000000 P 448d6c5f1d0f4b498c89ed2d0b29758d [term 1 LEADER]: Becoming Leader. State: Replica: 448d6c5f1d0f4b498c89ed2d0b29758d, State: Running, Role: LEADER
I20260809 12:13:21.840797 15797 consensus_queue.cc:237] T 00000000000000000000000000000000 P 448d6c5f1d0f4b498c89ed2d0b29758d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "448d6c5f1d0f4b498c89ed2d0b29758d" member_type: VOTER last_known_addr { host: "127.0.222.62" port: 43629 } }
I20260809 12:13:21.841552 15792 sys_catalog.cc:564] T 00000000000000000000000000000000 P 448d6c5f1d0f4b498c89ed2d0b29758d [sys.catalog]: configured and running, proceeding with master startup.
I20260809 12:13:21.843830 15798 sys_catalog.cc:455] T 00000000000000000000000000000000 P 448d6c5f1d0f4b498c89ed2d0b29758d [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "448d6c5f1d0f4b498c89ed2d0b29758d" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "448d6c5f1d0f4b498c89ed2d0b29758d" member_type: VOTER last_known_addr { host: "127.0.222.62" port: 43629 } } }
I20260809 12:13:21.844339 15798 sys_catalog.cc:458] T 00000000000000000000000000000000 P 448d6c5f1d0f4b498c89ed2d0b29758d [sys.catalog]: This master's current role is: LEADER
I20260809 12:13:21.845749 15804 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260809 12:13:21.845769 15799 sys_catalog.cc:455] T 00000000000000000000000000000000 P 448d6c5f1d0f4b498c89ed2d0b29758d [sys.catalog]: SysCatalogTable state changed. Reason: New leader 448d6c5f1d0f4b498c89ed2d0b29758d. Latest consensus state: current_term: 1 leader_uuid: "448d6c5f1d0f4b498c89ed2d0b29758d" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "448d6c5f1d0f4b498c89ed2d0b29758d" member_type: VOTER last_known_addr { host: "127.0.222.62" port: 43629 } } }
I20260809 12:13:21.846238 15799 sys_catalog.cc:458] T 00000000000000000000000000000000 P 448d6c5f1d0f4b498c89ed2d0b29758d [sys.catalog]: This master's current role is: LEADER
I20260809 12:13:21.851315 15804 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260809 12:13:21.860540 15804 catalog_manager.cc:1347] Generated new cluster ID: 5e4874b895ab4e5d8a6bba902346b353
I20260809 12:13:21.860703 15804 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260809 12:13:21.869221 15804 catalog_manager.cc:1370] Generated new certificate authority record
I20260809 12:13:21.870594 15804 catalog_manager.cc:1504] Loading token signing keys...
I20260809 12:13:21.878611 15804 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 448d6c5f1d0f4b498c89ed2d0b29758d: Generated new TSK 0
I20260809 12:13:21.879523 15804 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260809 12:13:21.894949 888 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskgH5Pjm/build/asan/bin/kudu pid 15721
I20260809 12:13:21.896255 15786 generic_service.cc:196] Checking for leaks (request via RPC)
I20260809 12:13:21.989740 888 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskgH5Pjm/build/asan/bin/kudu with pid 15721
2026-08-09T12:13:22Z chronyd exiting
[ OK ] SecurityITest.IPKICACertWebServerEndPoints (2101 ms)
[ RUN ] SecurityITest.TestEnableAndDisableEncryptedIPKICert
Loading random data
Initializing database '/tmp/dist-test-taskgH5Pjm/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786277382073938-888-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 09 12:13:22 dist-test-slave-2w3w krb5kdc[15823](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 09 12:13:22 dist-test-slave-2w3w krb5kdc[15823](info): set up 2 sockets
Aug 09 12:13:22 dist-test-slave-2w3w krb5kdc[15823](info): commencing operation
krb5kdc: starting...
W20260809 12:13:23.149116 888 mini_kdc.cc:121] Time spent starting KDC: real 1.117s user 0.006s sys 0.030s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskgH5Pjm/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786277382073938-888-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskgH5Pjm/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786277382073938-888-0/krb5kdc/test-user.keytab.
Aug 09 12:13:23 dist-test-slave-2w3w krb5kdc[15823](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786277603, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-09T12:13:23Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-09T12:13:23Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.222.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.222.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.222.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskgH5Pjm/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786277382073938-888-0/krb5kdc/kudu_127.0.222.62.keytab.
Entry for principal kudu/127.0.222.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskgH5Pjm/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786277382073938-888-0/krb5kdc/kudu_127.0.222.62.keytab.
I20260809 12:13:23.442039 888 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskgH5Pjm/build/asan/bin/kudu
/tmp/dist-test-taskgH5Pjm/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskgH5Pjm/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786277382073938-888-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskgH5Pjm/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786277382073938-888-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskgH5Pjm/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786277382073938-888-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskgH5Pjm/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786277382073938-888-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.222.62:44147
--webserver_interface=127.0.222.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskgH5Pjm/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786277382073938-888-0/krb5kdc/kudu_127.0.222.62.keytab
--principal=kudu/127.0.222.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.222.20:41797
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.222.62:44147
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskgH5Pjm/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786277382073938-888-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskgH5Pjm/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786277382073938-888-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskgH5Pjm/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786277382073938-888-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260809 12:13:23.780926 15837 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260809 12:13:23.781230 15837 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260809 12:13:23.781424 15837 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260809 12:13:23.788808 15837 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260809 12:13:23.788918 15837 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260809 12:13:23.789024 15837 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260809 12:13:23.789140 15837 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260809 12:13:23.789223 15837 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260809 12:13:23.799226 15837 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.222.20:41797
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskgH5Pjm/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786277382073938-888-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskgH5Pjm/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786277382073938-888-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.222.62:44147
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskgH5Pjm/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786277382073938-888-0/krb5kdc/kudu_127.0.222.62.keytab
--principal=kudu/127.0.222.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.222.62:44147
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskgH5Pjm/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786277382073938-888-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.222.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskgH5Pjm/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786277382073938-888-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 09 Aug 2026 11:06:41 UTC on ccycloud.quasar-bwoogy.root.comops.site
build id 4575991
ASAN enabled
I20260809 12:13:23.800612 15837 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260809 12:13:23.802007 15837 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260809 12:13:23.814218 15842 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 12:13:23.814337 15845 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 12:13:23.814576 15843 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260809 12:13:23.815685 15837 server_base.cc:1034] running on GCE node
I20260809 12:13:23.816541 15837 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260809 12:13:23.818025 15837 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260809 12:13:23.819263 15837 hybrid_clock.cc:648] HybridClock initialized: now 1786277603819207 us; error 63 us; skew 500 ppm
I20260809 12:13:23.819624 15837 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 09 12:13:23 dist-test-slave-2w3w krb5kdc[15823](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786277603, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.222.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260809 12:13:23.822777 15837 init.cc:377] Logged in from keytab as kudu/127.0.222.62@KRBTEST.COM (short username kudu)
I20260809 12:13:23.824379 15837 webserver.cc:460] Webserver started at http://127.0.222.62:42133/ using document root <none> and password file <none>
I20260809 12:13:23.825110 15837 fs_manager.cc:359] Metadata directory not provided
I20260809 12:13:23.825265 15837 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260809 12:13:23.825589 15837 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260809 12:13:23.828711 15837 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskgH5Pjm/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786277382073938-888-0/minicluster-data/master-0/data/instance:
uuid: "6d7f8671e34d46459506bc2cba79c82c"
format_stamp: "Formatted at 2026-08-09 12:13:23 on dist-test-slave-2w3w"
server_key: "5cd736e973b6ffe5de54c3107192d76b"
server_key_iv: "493043018edc30ecf9a63340ec4f2759"
server_key_version: "encryptionkey@0"
I20260809 12:13:23.829502 15837 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskgH5Pjm/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786277382073938-888-0/minicluster-data/master-0/wal/instance:
uuid: "6d7f8671e34d46459506bc2cba79c82c"
format_stamp: "Formatted at 2026-08-09 12:13:23 on dist-test-slave-2w3w"
server_key: "5cd736e973b6ffe5de54c3107192d76b"
server_key_iv: "493043018edc30ecf9a63340ec4f2759"
server_key_version: "encryptionkey@0"
I20260809 12:13:23.834875 15837 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.001s
I20260809 12:13:23.838374 15852 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260809 12:13:23.839756 15837 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.000s sys 0.003s
I20260809 12:13:23.839960 15837 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskgH5Pjm/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786277382073938-888-0/minicluster-data/master-0/data,/tmp/dist-test-taskgH5Pjm/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786277382073938-888-0/minicluster-data/master-0/wal
uuid: "6d7f8671e34d46459506bc2cba79c82c"
format_stamp: "Formatted at 2026-08-09 12:13:23 on dist-test-slave-2w3w"
server_key: "5cd736e973b6ffe5de54c3107192d76b"
server_key_iv: "493043018edc30ecf9a63340ec4f2759"
server_key_version: "encryptionkey@0"
I20260809 12:13:23.840171 15837 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskgH5Pjm/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786277382073938-888-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskgH5Pjm/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786277382073938-888-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskgH5Pjm/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786277382073938-888-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260809 12:13:23.857324 15837 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260809 12:13:23.861490 15837 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260809 12:13:23.861793 15837 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260809 12:13:23.881078 15837 rpc_server.cc:307] RPC server started. Bound to: 127.0.222.62:44147
I20260809 12:13:23.881134 15905 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.222.62:44147 every 8 connection(s)
I20260809 12:13:23.882445 15837 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskgH5Pjm/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786277382073938-888-0/minicluster-data/master-0/data/info.pb
I20260809 12:13:23.886068 15906 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260809 12:13:23.890591 888 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskgH5Pjm/build/asan/bin/kudu as pid 15837
I20260809 12:13:23.890820 888 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskgH5Pjm/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786277382073938-888-0/minicluster-data/master-0/wal/instance
I20260809 12:13:23.891541 888 external_mini_cluster.cc:1428] Setting key 76fd1cc3599cd5cff47ee93a5bb8fd41
I20260809 12:13:23.897559 15906 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 6d7f8671e34d46459506bc2cba79c82c: Bootstrap starting.
I20260809 12:13:23.901530 15906 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 6d7f8671e34d46459506bc2cba79c82c: Neither blocks nor log segments found. Creating new log.
I20260809 12:13:23.902854 15906 log.cc:824] T 00000000000000000000000000000000 P 6d7f8671e34d46459506bc2cba79c82c: Log is configured to *not* fsync() on all Append() calls
I20260809 12:13:23.905961 15906 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 6d7f8671e34d46459506bc2cba79c82c: No bootstrap required, opened a new log
Aug 09 12:13:23 dist-test-slave-2w3w krb5kdc[15823](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786277603, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.222.62@KRBTEST.COM
I20260809 12:13:23.916137 15906 raft_consensus.cc:348] T 00000000000000000000000000000000 P 6d7f8671e34d46459506bc2cba79c82c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6d7f8671e34d46459506bc2cba79c82c" member_type: VOTER last_known_addr { host: "127.0.222.62" port: 44147 } }
I20260809 12:13:23.916438 15910 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 12:13:23.893754 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59012 (local address 127.0.222.62:44147)
0809 12:13:23.894679 (+ 925us) server_negotiation.cc:206] Beginning negotiation
0809 12:13:23.894692 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0809 12:13:23.894753 (+ 61us) server_negotiation.cc:407] Connection header received
0809 12:13:23.896649 (+ 1896us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 12:13:23.896674 (+ 25us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 12:13:23.897105 (+ 431us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 12:13:23.897447 (+ 342us) server_negotiation.cc:226] Negotiated authn=SASL
0809 12:13:23.899430 (+ 1983us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 12:13:23.900757 (+ 1327us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 12:13:23.902678 (+ 1921us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 12:13:23.903569 (+ 891us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 12:13:23.903779 (+ 210us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 12:13:23.908420 (+ 4641us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 12:13:23.908427 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 12:13:23.908442 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0809 12:13:23.908488 (+ 46us) server_negotiation.cc:922] calling sasl_server_start()
0809 12:13:23.911975 (+ 3487us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 12:13:23.912877 (+ 902us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 12:13:23.912882 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 12:13:23.912889 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0809 12:13:23.913071 (+ 182us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 12:13:23.913521 (+ 450us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 12:13:23.913524 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 12:13:23.913526 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0809 12:13:23.914098 (+ 572us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 12:13:23.914324 (+ 226us) server_negotiation.cc:1035] Waiting for connection context
0809 12:13:23.914882 (+ 558us) server_negotiation.cc:299] Negotiation successful
0809 12:13:23.915375 (+ 493us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":661,"thread_start_us":433,"threads_started":1}
I20260809 12:13:23.916543 15906 raft_consensus.cc:374] T 00000000000000000000000000000000 P 6d7f8671e34d46459506bc2cba79c82c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260809 12:13:23.916643 15906 raft_consensus.cc:729] T 00000000000000000000000000000000 P 6d7f8671e34d46459506bc2cba79c82c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6d7f8671e34d46459506bc2cba79c82c, State: Initialized, Role: FOLLOWER
I20260809 12:13:23.917253 15906 consensus_queue.cc:260] T 00000000000000000000000000000000 P 6d7f8671e34d46459506bc2cba79c82c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6d7f8671e34d46459506bc2cba79c82c" member_type: VOTER last_known_addr { host: "127.0.222.62" port: 44147 } }
I20260809 12:13:23.917519 15906 raft_consensus.cc:388] T 00000000000000000000000000000000 P 6d7f8671e34d46459506bc2cba79c82c [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260809 12:13:23.917687 15906 raft_consensus.cc:482] T 00000000000000000000000000000000 P 6d7f8671e34d46459506bc2cba79c82c [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260809 12:13:23.917879 15906 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 6d7f8671e34d46459506bc2cba79c82c [term 0 FOLLOWER]: Advancing to term 1
I20260809 12:13:23.920343 15906 raft_consensus.cc:504] T 00000000000000000000000000000000 P 6d7f8671e34d46459506bc2cba79c82c [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6d7f8671e34d46459506bc2cba79c82c" member_type: VOTER last_known_addr { host: "127.0.222.62" port: 44147 } }
I20260809 12:13:23.920842 15906 leader_election.cc:302] T 00000000000000000000000000000000 P 6d7f8671e34d46459506bc2cba79c82c [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 6d7f8671e34d46459506bc2cba79c82c; no voters:
I20260809 12:13:23.921662 15906 leader_election.cc:288] T 00000000000000000000000000000000 P 6d7f8671e34d46459506bc2cba79c82c [CANDIDATE]: Term 1 election: Requested vote from peers
I20260809 12:13:23.921802 15912 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 6d7f8671e34d46459506bc2cba79c82c [term 1 FOLLOWER]: Leader election won for term 1
I20260809 12:13:23.922613 15912 raft_consensus.cc:686] T 00000000000000000000000000000000 P 6d7f8671e34d46459506bc2cba79c82c [term 1 LEADER]: Becoming Leader. State: Replica: 6d7f8671e34d46459506bc2cba79c82c, State: Running, Role: LEADER
I20260809 12:13:23.923238 15912 consensus_queue.cc:237] T 00000000000000000000000000000000 P 6d7f8671e34d46459506bc2cba79c82c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6d7f8671e34d46459506bc2cba79c82c" member_type: VOTER last_known_addr { host: "127.0.222.62" port: 44147 } }
I20260809 12:13:23.924031 15906 sys_catalog.cc:564] T 00000000000000000000000000000000 P 6d7f8671e34d46459506bc2cba79c82c [sys.catalog]: configured and running, proceeding with master startup.
I20260809 12:13:23.926903 15914 sys_catalog.cc:455] T 00000000000000000000000000000000 P 6d7f8671e34d46459506bc2cba79c82c [sys.catalog]: SysCatalogTable state changed. Reason: New leader 6d7f8671e34d46459506bc2cba79c82c. Latest consensus state: current_term: 1 leader_uuid: "6d7f8671e34d46459506bc2cba79c82c" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6d7f8671e34d46459506bc2cba79c82c" member_type: VOTER last_known_addr { host: "127.0.222.62" port: 44147 } } }
I20260809 12:13:23.927403 15914 sys_catalog.cc:458] T 00000000000000000000000000000000 P 6d7f8671e34d46459506bc2cba79c82c [sys.catalog]: This master's current role is: LEADER
I20260809 12:13:23.927791 15913 sys_catalog.cc:455] T 00000000000000000000000000000000 P 6d7f8671e34d46459506bc2cba79c82c [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "6d7f8671e34d46459506bc2cba79c82c" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6d7f8671e34d46459506bc2cba79c82c" member_type: VOTER last_known_addr { host: "127.0.222.62" port: 44147 } } }
I20260809 12:13:23.929309 15913 sys_catalog.cc:458] T 00000000000000000000000000000000 P 6d7f8671e34d46459506bc2cba79c82c [sys.catalog]: This master's current role is: LEADER
I20260809 12:13:23.930938 15918 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260809 12:13:23.937615 15918 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260809 12:13:23.947199 15918 catalog_manager.cc:1347] Generated new cluster ID: 71a354fc623b4b8f864d7005c8295487
I20260809 12:13:23.947329 15918 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260809 12:13:23.965111 15918 catalog_manager.cc:1370] Generated new certificate authority record
I20260809 12:13:23.966490 15918 catalog_manager.cc:1504] Loading token signing keys...
I20260809 12:13:23.975750 15918 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 6d7f8671e34d46459506bc2cba79c82c: Generated new TSK 0
I20260809 12:13:23.976632 15918 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.0.222.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.222.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.222.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskgH5Pjm/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786277382073938-888-0/krb5kdc/kudu_127.0.222.1.keytab.
Entry for principal kudu/127.0.222.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskgH5Pjm/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786277382073938-888-0/krb5kdc/kudu_127.0.222.1.keytab.
I20260809 12:13:24.057133 888 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskgH5Pjm/build/asan/bin/kudu
/tmp/dist-test-taskgH5Pjm/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskgH5Pjm/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786277382073938-888-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskgH5Pjm/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786277382073938-888-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskgH5Pjm/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786277382073938-888-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskgH5Pjm/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786277382073938-888-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.222.1:0
--local_ip_for_outbound_sockets=127.0.222.1
--webserver_interface=127.0.222.1
--webserver_port=0
--tserver_master_addrs=127.0.222.62:44147
--keytab_file=/tmp/dist-test-taskgH5Pjm/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786277382073938-888-0/krb5kdc/kudu_127.0.222.1.keytab
--principal=kudu/127.0.222.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.222.20:41797
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskgH5Pjm/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786277382073938-888-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskgH5Pjm/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786277382073938-888-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskgH5Pjm/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786277382073938-888-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260809 12:13:24.401103 15934 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260809 12:13:24.401391 15934 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260809 12:13:24.401553 15934 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260809 12:13:24.408766 15934 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260809 12:13:24.408926 15934 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260809 12:13:24.409160 15934 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.222.1
I20260809 12:13:24.422210 15934 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.222.20:41797
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskgH5Pjm/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786277382073938-888-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskgH5Pjm/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786277382073938-888-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskgH5Pjm/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786277382073938-888-0/krb5kdc/kudu_127.0.222.1.keytab
--principal=kudu/127.0.222.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.222.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskgH5Pjm/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786277382073938-888-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.222.1
--webserver_port=0
--tserver_master_addrs=127.0.222.62:44147
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.222.1
--log_dir=/tmp/dist-test-taskgH5Pjm/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786277382073938-888-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 09 Aug 2026 11:06:41 UTC on ccycloud.quasar-bwoogy.root.comops.site
build id 4575991
ASAN enabled
I20260809 12:13:24.423676 15934 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260809 12:13:24.425203 15934 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260809 12:13:24.438876 15941 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 12:13:24.438822 15940 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 12:13:24.438930 15943 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260809 12:13:24.439814 15934 server_base.cc:1034] running on GCE node
I20260809 12:13:24.440505 15934 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260809 12:13:24.441466 15934 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260809 12:13:24.442701 15934 hybrid_clock.cc:648] HybridClock initialized: now 1786277604442656 us; error 61 us; skew 500 ppm
I20260809 12:13:24.443101 15934 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 09 12:13:24 dist-test-slave-2w3w krb5kdc[15823](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786277604, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.222.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260809 12:13:24.446626 15934 init.cc:377] Logged in from keytab as kudu/127.0.222.1@KRBTEST.COM (short username kudu)
I20260809 12:13:24.448207 15934 webserver.cc:460] Webserver started at http://127.0.222.1:44545/ using document root <none> and password file <none>
I20260809 12:13:24.448891 15934 fs_manager.cc:359] Metadata directory not provided
I20260809 12:13:24.449049 15934 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260809 12:13:24.449363 15934 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260809 12:13:24.452392 15934 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskgH5Pjm/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786277382073938-888-0/minicluster-data/ts-0/data/instance:
uuid: "2a033dd02e3d408eadf277165d4c5b95"
format_stamp: "Formatted at 2026-08-09 12:13:24 on dist-test-slave-2w3w"
server_key: "cbad234163032fa77b32d4e26b9f9057"
server_key_iv: "bbf5477b04df91cc882d7c584ab72de6"
server_key_version: "encryptionkey@0"
I20260809 12:13:24.453212 15934 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskgH5Pjm/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786277382073938-888-0/minicluster-data/ts-0/wal/instance:
uuid: "2a033dd02e3d408eadf277165d4c5b95"
format_stamp: "Formatted at 2026-08-09 12:13:24 on dist-test-slave-2w3w"
server_key: "cbad234163032fa77b32d4e26b9f9057"
server_key_iv: "bbf5477b04df91cc882d7c584ab72de6"
server_key_version: "encryptionkey@0"
I20260809 12:13:24.458402 15934 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.003s sys 0.003s
I20260809 12:13:24.461957 15951 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260809 12:13:24.463333 15934 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260809 12:13:24.463510 15934 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskgH5Pjm/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786277382073938-888-0/minicluster-data/ts-0/data,/tmp/dist-test-taskgH5Pjm/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786277382073938-888-0/minicluster-data/ts-0/wal
uuid: "2a033dd02e3d408eadf277165d4c5b95"
format_stamp: "Formatted at 2026-08-09 12:13:24 on dist-test-slave-2w3w"
server_key: "cbad234163032fa77b32d4e26b9f9057"
server_key_iv: "bbf5477b04df91cc882d7c584ab72de6"
server_key_version: "encryptionkey@0"
I20260809 12:13:24.463727 15934 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskgH5Pjm/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786277382073938-888-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskgH5Pjm/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786277382073938-888-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskgH5Pjm/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786277382073938-888-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260809 12:13:24.476159 15934 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260809 12:13:24.480786 15934 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260809 12:13:24.481123 15934 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260809 12:13:24.482160 15934 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260809 12:13:24.483615 15934 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260809 12:13:24.483723 15934 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260809 12:13:24.483889 15934 ts_tablet_manager.cc:610] Registered 0 tablets
I20260809 12:13:24.483963 15934 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260809 12:13:24.518636 15934 rpc_server.cc:307] RPC server started. Bound to: 127.0.222.1:40557
I20260809 12:13:24.518688 16068 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.222.1:40557 every 8 connection(s)
I20260809 12:13:24.520008 15934 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskgH5Pjm/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786277382073938-888-0/minicluster-data/ts-0/data/info.pb
I20260809 12:13:24.527747 888 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskgH5Pjm/build/asan/bin/kudu as pid 15934
I20260809 12:13:24.528036 888 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskgH5Pjm/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786277382073938-888-0/minicluster-data/ts-0/wal/instance
I20260809 12:13:24.528959 888 external_mini_cluster.cc:1428] Setting key e187096b4929058d5118fec841b5ba7d
Aug 09 12:13:24 dist-test-slave-2w3w krb5kdc[15823](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786277604, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.222.1@KRBTEST.COM for kudu/127.0.222.62@KRBTEST.COM
I20260809 12:13:24.544219 16071 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 12:13:24.522641 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.222.1:46671 (local address 127.0.222.62:44147)
0809 12:13:24.523197 (+ 556us) server_negotiation.cc:206] Beginning negotiation
0809 12:13:24.523204 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0809 12:13:24.524271 (+ 1067us) server_negotiation.cc:407] Connection header received
0809 12:13:24.526216 (+ 1945us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 12:13:24.526220 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 12:13:24.526297 (+ 77us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 12:13:24.526401 (+ 104us) server_negotiation.cc:226] Negotiated authn=SASL
0809 12:13:24.527939 (+ 1538us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 12:13:24.529129 (+ 1190us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 12:13:24.531341 (+ 2212us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 12:13:24.531976 (+ 635us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 12:13:24.532155 (+ 179us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 12:13:24.536652 (+ 4497us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 12:13:24.536661 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 12:13:24.536669 (+ 8us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0809 12:13:24.536712 (+ 43us) server_negotiation.cc:922] calling sasl_server_start()
0809 12:13:24.540152 (+ 3440us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 12:13:24.541108 (+ 956us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 12:13:24.541115 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 12:13:24.541121 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0809 12:13:24.541258 (+ 137us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 12:13:24.541751 (+ 493us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 12:13:24.541757 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 12:13:24.541762 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0809 12:13:24.542110 (+ 348us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 12:13:24.542240 (+ 130us) server_negotiation.cc:1035] Waiting for connection context
0809 12:13:24.543518 (+ 1278us) server_negotiation.cc:299] Negotiation successful
0809 12:13:24.543810 (+ 292us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":413,"thread_start_us":236,"threads_started":1}
I20260809 12:13:24.544953 16072 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 12:13:24.523289 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.222.62:44147 (local address 127.0.222.1:46671)
0809 12:13:24.524104 (+ 815us) negotiation.cc:107] Waiting for socket to connect
0809 12:13:24.524143 (+ 39us) client_negotiation.cc:174] Beginning negotiation
0809 12:13:24.525945 (+ 1802us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 12:13:24.526599 (+ 654us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 12:13:24.526607 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 12:13:24.527194 (+ 587us) client_negotiation.cc:189] Negotiated authn=SASL
0809 12:13:24.527746 (+ 552us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 12:13:24.527764 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 12:13:24.529285 (+ 1521us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 12:13:24.529289 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 12:13:24.531152 (+ 1863us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 12:13:24.531175 (+ 23us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 12:13:24.532232 (+ 1057us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 12:13:24.532237 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 12:13:24.532428 (+ 191us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 12:13:24.533220 (+ 792us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0809 12:13:24.533246 (+ 26us) client_negotiation.cc:656] calling sasl_client_start()
0809 12:13:24.536400 (+ 3154us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0809 12:13:24.540322 (+ 3922us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0809 12:13:24.540332 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0809 12:13:24.540351 (+ 19us) client_negotiation.cc:760] calling sasl_client_step()
0809 12:13:24.540984 (+ 633us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0809 12:13:24.541370 (+ 386us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0809 12:13:24.541377 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0809 12:13:24.541382 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0809 12:13:24.541633 (+ 251us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0809 12:13:24.542252 (+ 619us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0809 12:13:24.542262 (+ 10us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0809 12:13:24.542719 (+ 457us) client_negotiation.cc:769] Sending connection context
0809 12:13:24.543057 (+ 338us) client_negotiation.cc:240] Negotiation successful
0809 12:13:24.543627 (+ 570us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":550,"thread_start_us":273,"threads_started":1}
I20260809 12:13:24.546779 16069 heartbeater.cc:347] Connected to a master server at 127.0.222.62:44147
I20260809 12:13:24.547139 16069 heartbeater.cc:464] Registering TS with master...
I20260809 12:13:24.547868 16069 heartbeater.cc:511] Master 127.0.222.62:44147 requested a full tablet report, sending...
I20260809 12:13:24.550017 15870 ts_manager.cc:194] Registered new tserver with Master: 2a033dd02e3d408eadf277165d4c5b95 (127.0.222.1:40557)
I20260809 12:13:24.551522 15870 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.222.1@KRBTEST.COM'} at 127.0.222.1:46671
WARNING: no policy specified for kudu/127.0.222.2@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.222.2@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.222.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskgH5Pjm/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786277382073938-888-0/krb5kdc/kudu_127.0.222.2.keytab.
Entry for principal kudu/127.0.222.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskgH5Pjm/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786277382073938-888-0/krb5kdc/kudu_127.0.222.2.keytab.
I20260809 12:13:24.595592 888 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskgH5Pjm/build/asan/bin/kudu
/tmp/dist-test-taskgH5Pjm/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskgH5Pjm/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786277382073938-888-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskgH5Pjm/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786277382073938-888-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskgH5Pjm/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786277382073938-888-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskgH5Pjm/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786277382073938-888-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.222.2:0
--local_ip_for_outbound_sockets=127.0.222.2
--webserver_interface=127.0.222.2
--webserver_port=0
--tserver_master_addrs=127.0.222.62:44147
--keytab_file=/tmp/dist-test-taskgH5Pjm/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786277382073938-888-0/krb5kdc/kudu_127.0.222.2.keytab
--principal=kudu/127.0.222.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.222.20:41797
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskgH5Pjm/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786277382073938-888-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskgH5Pjm/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786277382073938-888-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskgH5Pjm/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786277382073938-888-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260809 12:13:24.941423 16075 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260809 12:13:24.941731 16075 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260809 12:13:24.941910 16075 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260809 12:13:24.949416 16075 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260809 12:13:24.949569 16075 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260809 12:13:24.949754 16075 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.222.2
I20260809 12:13:24.959435 16075 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.222.20:41797
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskgH5Pjm/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786277382073938-888-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskgH5Pjm/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786277382073938-888-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskgH5Pjm/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786277382073938-888-0/krb5kdc/kudu_127.0.222.2.keytab
--principal=kudu/127.0.222.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.222.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskgH5Pjm/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786277382073938-888-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.222.2
--webserver_port=0
--tserver_master_addrs=127.0.222.62:44147
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.222.2
--log_dir=/tmp/dist-test-taskgH5Pjm/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786277382073938-888-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 09 Aug 2026 11:06:41 UTC on ccycloud.quasar-bwoogy.root.comops.site
build id 4575991
ASAN enabled
I20260809 12:13:24.960729 16075 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260809 12:13:24.962129 16075 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260809 12:13:24.975879 16080 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 12:13:24.975844 16083 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 12:13:24.975798 16081 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260809 12:13:24.977010 16075 server_base.cc:1034] running on GCE node
I20260809 12:13:24.977711 16075 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260809 12:13:24.978713 16075 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260809 12:13:24.979954 16075 hybrid_clock.cc:648] HybridClock initialized: now 1786277604979892 us; error 73 us; skew 500 ppm
I20260809 12:13:24.980290 16075 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 09 12:13:24 dist-test-slave-2w3w krb5kdc[15823](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786277604, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.222.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260809 12:13:24.983601 16075 init.cc:377] Logged in from keytab as kudu/127.0.222.2@KRBTEST.COM (short username kudu)
I20260809 12:13:24.985234 16075 webserver.cc:460] Webserver started at http://127.0.222.2:46335/ using document root <none> and password file <none>
I20260809 12:13:24.985954 16075 fs_manager.cc:359] Metadata directory not provided
I20260809 12:13:24.986109 16075 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260809 12:13:24.986428 16075 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260809 12:13:24.989554 16075 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskgH5Pjm/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786277382073938-888-0/minicluster-data/ts-1/data/instance:
uuid: "5c1e1fc7cc4b4f59b9ca7298b6929b5c"
format_stamp: "Formatted at 2026-08-09 12:13:24 on dist-test-slave-2w3w"
server_key: "659ef2a30fb36e2536ff973fe9e8e5b4"
server_key_iv: "ca75656d9f5532d4adc03ec74619056d"
server_key_version: "encryptionkey@0"
I20260809 12:13:24.990414 16075 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskgH5Pjm/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786277382073938-888-0/minicluster-data/ts-1/wal/instance:
uuid: "5c1e1fc7cc4b4f59b9ca7298b6929b5c"
format_stamp: "Formatted at 2026-08-09 12:13:24 on dist-test-slave-2w3w"
server_key: "659ef2a30fb36e2536ff973fe9e8e5b4"
server_key_iv: "ca75656d9f5532d4adc03ec74619056d"
server_key_version: "encryptionkey@0"
I20260809 12:13:24.995781 16075 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.007s sys 0.001s
I20260809 12:13:24.999305 16090 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260809 12:13:25.000635 16075 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.001s
I20260809 12:13:25.000850 16075 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskgH5Pjm/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786277382073938-888-0/minicluster-data/ts-1/data,/tmp/dist-test-taskgH5Pjm/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786277382073938-888-0/minicluster-data/ts-1/wal
uuid: "5c1e1fc7cc4b4f59b9ca7298b6929b5c"
format_stamp: "Formatted at 2026-08-09 12:13:24 on dist-test-slave-2w3w"
server_key: "659ef2a30fb36e2536ff973fe9e8e5b4"
server_key_iv: "ca75656d9f5532d4adc03ec74619056d"
server_key_version: "encryptionkey@0"
I20260809 12:13:25.001077 16075 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskgH5Pjm/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786277382073938-888-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskgH5Pjm/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786277382073938-888-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskgH5Pjm/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786277382073938-888-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260809 12:13:25.033231 16075 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260809 12:13:25.037667 16075 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260809 12:13:25.037981 16075 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260809 12:13:25.039062 16075 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260809 12:13:25.040436 16075 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260809 12:13:25.040545 16075 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260809 12:13:25.040711 16075 ts_tablet_manager.cc:610] Registered 0 tablets
I20260809 12:13:25.040795 16075 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260809 12:13:25.085389 16075 rpc_server.cc:307] RPC server started. Bound to: 127.0.222.2:41313
I20260809 12:13:25.085450 16206 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.222.2:41313 every 8 connection(s)
I20260809 12:13:25.086745 16075 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskgH5Pjm/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786277382073938-888-0/minicluster-data/ts-1/data/info.pb
I20260809 12:13:25.095633 888 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskgH5Pjm/build/asan/bin/kudu as pid 16075
I20260809 12:13:25.095890 888 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskgH5Pjm/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786277382073938-888-0/minicluster-data/ts-1/wal/instance
I20260809 12:13:25.096489 888 external_mini_cluster.cc:1428] Setting key 4fb4d8892599440f1cd5bd15c3c2cf9e
Aug 09 12:13:25 dist-test-slave-2w3w krb5kdc[15823](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786277604, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.222.2@KRBTEST.COM for kudu/127.0.222.62@KRBTEST.COM
I20260809 12:13:25.109165 16209 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 12:13:25.089610 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.222.2:35747 (local address 127.0.222.62:44147)
0809 12:13:25.090153 (+ 543us) server_negotiation.cc:206] Beginning negotiation
0809 12:13:25.090159 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0809 12:13:25.091187 (+ 1028us) server_negotiation.cc:407] Connection header received
0809 12:13:25.093111 (+ 1924us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 12:13:25.093115 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 12:13:25.093184 (+ 69us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 12:13:25.093285 (+ 101us) server_negotiation.cc:226] Negotiated authn=SASL
0809 12:13:25.094816 (+ 1531us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 12:13:25.095631 (+ 815us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 12:13:25.097830 (+ 2199us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 12:13:25.098505 (+ 675us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 12:13:25.098642 (+ 137us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 12:13:25.102883 (+ 4241us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 12:13:25.102888 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 12:13:25.102892 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0809 12:13:25.102924 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0809 12:13:25.105946 (+ 3022us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 12:13:25.106748 (+ 802us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 12:13:25.106756 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 12:13:25.106761 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0809 12:13:25.106896 (+ 135us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 12:13:25.107379 (+ 483us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 12:13:25.107386 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 12:13:25.107389 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0809 12:13:25.107674 (+ 285us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 12:13:25.107836 (+ 162us) server_negotiation.cc:1035] Waiting for connection context
0809 12:13:25.108554 (+ 718us) server_negotiation.cc:299] Negotiation successful
0809 12:13:25.108775 (+ 221us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":408,"thread_start_us":268,"threads_started":1}
I20260809 12:13:25.110280 16210 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 12:13:25.090186 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.222.62:44147 (local address 127.0.222.2:35747)
0809 12:13:25.090948 (+ 762us) negotiation.cc:107] Waiting for socket to connect
0809 12:13:25.091053 (+ 105us) client_negotiation.cc:174] Beginning negotiation
0809 12:13:25.092836 (+ 1783us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 12:13:25.093543 (+ 707us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 12:13:25.093551 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 12:13:25.094064 (+ 513us) client_negotiation.cc:189] Negotiated authn=SASL
0809 12:13:25.094642 (+ 578us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 12:13:25.094659 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 12:13:25.095805 (+ 1146us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 12:13:25.095811 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 12:13:25.097691 (+ 1880us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 12:13:25.097703 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 12:13:25.098810 (+ 1107us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 12:13:25.098819 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 12:13:25.099118 (+ 299us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 12:13:25.099840 (+ 722us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0809 12:13:25.099880 (+ 40us) client_negotiation.cc:656] calling sasl_client_start()
0809 12:13:25.102647 (+ 2767us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0809 12:13:25.106090 (+ 3443us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0809 12:13:25.106099 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0809 12:13:25.106116 (+ 17us) client_negotiation.cc:760] calling sasl_client_step()
0809 12:13:25.106622 (+ 506us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0809 12:13:25.107053 (+ 431us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0809 12:13:25.107058 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0809 12:13:25.107061 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0809 12:13:25.107258 (+ 197us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0809 12:13:25.107827 (+ 569us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0809 12:13:25.107835 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0809 12:13:25.108189 (+ 354us) client_negotiation.cc:769] Sending connection context
0809 12:13:25.108460 (+ 271us) client_negotiation.cc:240] Negotiation successful
0809 12:13:25.109011 (+ 551us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":505,"thread_start_us":305,"threads_started":1}
I20260809 12:13:25.111876 16207 heartbeater.cc:347] Connected to a master server at 127.0.222.62:44147
I20260809 12:13:25.112186 16207 heartbeater.cc:464] Registering TS with master...
I20260809 12:13:25.112982 16207 heartbeater.cc:511] Master 127.0.222.62:44147 requested a full tablet report, sending...
I20260809 12:13:25.114516 15870 ts_manager.cc:194] Registered new tserver with Master: 5c1e1fc7cc4b4f59b9ca7298b6929b5c (127.0.222.2:41313)
I20260809 12:13:25.115481 15870 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.222.2@KRBTEST.COM'} at 127.0.222.2:35747
WARNING: no policy specified for kudu/127.0.222.3@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.222.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.222.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskgH5Pjm/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786277382073938-888-0/krb5kdc/kudu_127.0.222.3.keytab.
Entry for principal kudu/127.0.222.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskgH5Pjm/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786277382073938-888-0/krb5kdc/kudu_127.0.222.3.keytab.
I20260809 12:13:25.162227 888 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskgH5Pjm/build/asan/bin/kudu
/tmp/dist-test-taskgH5Pjm/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskgH5Pjm/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786277382073938-888-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskgH5Pjm/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786277382073938-888-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskgH5Pjm/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786277382073938-888-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskgH5Pjm/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786277382073938-888-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.222.3:0
--local_ip_for_outbound_sockets=127.0.222.3
--webserver_interface=127.0.222.3
--webserver_port=0
--tserver_master_addrs=127.0.222.62:44147
--keytab_file=/tmp/dist-test-taskgH5Pjm/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786277382073938-888-0/krb5kdc/kudu_127.0.222.3.keytab
--principal=kudu/127.0.222.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.222.20:41797
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskgH5Pjm/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786277382073938-888-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskgH5Pjm/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786277382073938-888-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskgH5Pjm/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786277382073938-888-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260809 12:13:25.507442 16213 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260809 12:13:25.507804 16213 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260809 12:13:25.507972 16213 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260809 12:13:25.515528 16213 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260809 12:13:25.515693 16213 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260809 12:13:25.515883 16213 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.222.3
I20260809 12:13:25.526183 16213 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.222.20:41797
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskgH5Pjm/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786277382073938-888-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskgH5Pjm/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786277382073938-888-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskgH5Pjm/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786277382073938-888-0/krb5kdc/kudu_127.0.222.3.keytab
--principal=kudu/127.0.222.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.222.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskgH5Pjm/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786277382073938-888-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.222.3
--webserver_port=0
--tserver_master_addrs=127.0.222.62:44147
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.222.3
--log_dir=/tmp/dist-test-taskgH5Pjm/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786277382073938-888-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 09 Aug 2026 11:06:41 UTC on ccycloud.quasar-bwoogy.root.comops.site
build id 4575991
ASAN enabled
I20260809 12:13:25.527496 16213 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260809 12:13:25.528854 16213 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260809 12:13:25.542758 16218 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 12:13:25.542649 16219 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 12:13:25.542730 16221 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260809 12:13:25.543560 16213 server_base.cc:1034] running on GCE node
I20260809 12:13:25.544305 16213 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260809 12:13:25.545442 16213 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260809 12:13:25.546694 16213 hybrid_clock.cc:648] HybridClock initialized: now 1786277605546631 us; error 53 us; skew 500 ppm
I20260809 12:13:25.547179 16213 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 09 12:13:25 dist-test-slave-2w3w krb5kdc[15823](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786277605, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.222.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260809 12:13:25.550477 16213 init.cc:377] Logged in from keytab as kudu/127.0.222.3@KRBTEST.COM (short username kudu)
I20260809 12:13:25.552037 16213 webserver.cc:460] Webserver started at http://127.0.222.3:36801/ using document root <none> and password file <none>
I20260809 12:13:25.552752 16213 fs_manager.cc:359] Metadata directory not provided
I20260809 12:13:25.552906 16213 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260809 12:13:25.553220 16213 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260809 12:13:25.554391 16069 heartbeater.cc:503] Master 127.0.222.62:44147 was elected leader, sending a full tablet report...
I20260809 12:13:25.556795 16213 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskgH5Pjm/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786277382073938-888-0/minicluster-data/ts-2/data/instance:
uuid: "1204c82bd36a468fbdf532c7aad13159"
format_stamp: "Formatted at 2026-08-09 12:13:25 on dist-test-slave-2w3w"
server_key: "69332dd618b10d8ba467b56a6ab50e51"
server_key_iv: "02d88d015dd0bb3411fe617ba0481989"
server_key_version: "encryptionkey@0"
I20260809 12:13:25.557608 16213 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskgH5Pjm/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786277382073938-888-0/minicluster-data/ts-2/wal/instance:
uuid: "1204c82bd36a468fbdf532c7aad13159"
format_stamp: "Formatted at 2026-08-09 12:13:25 on dist-test-slave-2w3w"
server_key: "69332dd618b10d8ba467b56a6ab50e51"
server_key_iv: "02d88d015dd0bb3411fe617ba0481989"
server_key_version: "encryptionkey@0"
I20260809 12:13:25.562865 16213 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.003s sys 0.004s
I20260809 12:13:25.566494 16228 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260809 12:13:25.568074 16213 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260809 12:13:25.568260 16213 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskgH5Pjm/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786277382073938-888-0/minicluster-data/ts-2/data,/tmp/dist-test-taskgH5Pjm/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786277382073938-888-0/minicluster-data/ts-2/wal
uuid: "1204c82bd36a468fbdf532c7aad13159"
format_stamp: "Formatted at 2026-08-09 12:13:25 on dist-test-slave-2w3w"
server_key: "69332dd618b10d8ba467b56a6ab50e51"
server_key_iv: "02d88d015dd0bb3411fe617ba0481989"
server_key_version: "encryptionkey@0"
I20260809 12:13:25.568472 16213 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskgH5Pjm/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786277382073938-888-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskgH5Pjm/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786277382073938-888-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskgH5Pjm/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786277382073938-888-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260809 12:13:25.581630 16213 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260809 12:13:25.586289 16213 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260809 12:13:25.586616 16213 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260809 12:13:25.587718 16213 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260809 12:13:25.589190 16213 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260809 12:13:25.589269 16213 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260809 12:13:25.589419 16213 ts_tablet_manager.cc:610] Registered 0 tablets
I20260809 12:13:25.589473 16213 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260809 12:13:25.625221 16213 rpc_server.cc:307] RPC server started. Bound to: 127.0.222.3:46283
I20260809 12:13:25.625260 16347 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.222.3:46283 every 8 connection(s)
I20260809 12:13:25.626588 16213 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskgH5Pjm/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786277382073938-888-0/minicluster-data/ts-2/data/info.pb
I20260809 12:13:25.632259 888 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskgH5Pjm/build/asan/bin/kudu as pid 16213
I20260809 12:13:25.632503 888 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskgH5Pjm/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786277382073938-888-0/minicluster-data/ts-2/wal/instance
I20260809 12:13:25.632992 888 external_mini_cluster.cc:1428] Setting key 431907fc329b27a18e4d9f40409f247b
Aug 09 12:13:25 dist-test-slave-2w3w krb5kdc[15823](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786277605, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.222.3@KRBTEST.COM for kudu/127.0.222.62@KRBTEST.COM
I20260809 12:13:25.649031 16351 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 12:13:25.629295 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.222.3:33845 (local address 127.0.222.62:44147)
0809 12:13:25.629763 (+ 468us) server_negotiation.cc:206] Beginning negotiation
0809 12:13:25.629768 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0809 12:13:25.630912 (+ 1144us) server_negotiation.cc:407] Connection header received
0809 12:13:25.633340 (+ 2428us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 12:13:25.633346 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 12:13:25.633448 (+ 102us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 12:13:25.633585 (+ 137us) server_negotiation.cc:226] Negotiated authn=SASL
0809 12:13:25.635611 (+ 2026us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 12:13:25.636564 (+ 953us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 12:13:25.638291 (+ 1727us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 12:13:25.638737 (+ 446us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 12:13:25.638888 (+ 151us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 12:13:25.642737 (+ 3849us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 12:13:25.642742 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 12:13:25.642746 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0809 12:13:25.642785 (+ 39us) server_negotiation.cc:922] calling sasl_server_start()
0809 12:13:25.645563 (+ 2778us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 12:13:25.646396 (+ 833us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 12:13:25.646400 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 12:13:25.646402 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0809 12:13:25.646537 (+ 135us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 12:13:25.647044 (+ 507us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 12:13:25.647052 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 12:13:25.647061 (+ 9us) server_negotiation.cc:967] calling sasl_server_step()
0809 12:13:25.647362 (+ 301us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 12:13:25.647547 (+ 185us) server_negotiation.cc:1035] Waiting for connection context
0809 12:13:25.648305 (+ 758us) server_negotiation.cc:299] Negotiation successful
0809 12:13:25.648549 (+ 244us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":350,"thread_start_us":225,"threads_started":1}
I20260809 12:13:25.650050 16352 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 12:13:25.629840 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.222.62:44147 (local address 127.0.222.3:33845)
0809 12:13:25.630699 (+ 859us) negotiation.cc:107] Waiting for socket to connect
0809 12:13:25.630740 (+ 41us) client_negotiation.cc:174] Beginning negotiation
0809 12:13:25.632983 (+ 2243us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 12:13:25.633864 (+ 881us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 12:13:25.633875 (+ 11us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 12:13:25.634572 (+ 697us) client_negotiation.cc:189] Negotiated authn=SASL
0809 12:13:25.635424 (+ 852us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 12:13:25.635447 (+ 23us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 12:13:25.636736 (+ 1289us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 12:13:25.636742 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 12:13:25.638124 (+ 1382us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 12:13:25.638137 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 12:13:25.639026 (+ 889us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 12:13:25.639038 (+ 12us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 12:13:25.639316 (+ 278us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 12:13:25.639981 (+ 665us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0809 12:13:25.640004 (+ 23us) client_negotiation.cc:656] calling sasl_client_start()
0809 12:13:25.642496 (+ 2492us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0809 12:13:25.645721 (+ 3225us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0809 12:13:25.645732 (+ 11us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0809 12:13:25.645750 (+ 18us) client_negotiation.cc:760] calling sasl_client_step()
0809 12:13:25.646231 (+ 481us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0809 12:13:25.646662 (+ 431us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0809 12:13:25.646668 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0809 12:13:25.646672 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0809 12:13:25.646891 (+ 219us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0809 12:13:25.647508 (+ 617us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0809 12:13:25.647516 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0809 12:13:25.647895 (+ 379us) client_negotiation.cc:769] Sending connection context
0809 12:13:25.648189 (+ 294us) client_negotiation.cc:240] Negotiation successful
0809 12:13:25.648793 (+ 604us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":596,"thread_start_us":324,"threads_started":1}
I20260809 12:13:25.651512 16348 heartbeater.cc:347] Connected to a master server at 127.0.222.62:44147
I20260809 12:13:25.651854 16348 heartbeater.cc:464] Registering TS with master...
I20260809 12:13:25.652599 16348 heartbeater.cc:511] Master 127.0.222.62:44147 requested a full tablet report, sending...
I20260809 12:13:25.654079 15870 ts_manager.cc:194] Registered new tserver with Master: 1204c82bd36a468fbdf532c7aad13159 (127.0.222.3:46283)
I20260809 12:13:25.654920 15870 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.222.3@KRBTEST.COM'} at 127.0.222.3:33845
I20260809 12:13:25.660912 888 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260809 12:13:25.678648 16351 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 12:13:25.665718 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59024 (local address 127.0.222.62:44147)
0809 12:13:25.665980 (+ 262us) server_negotiation.cc:206] Beginning negotiation
0809 12:13:25.665990 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0809 12:13:25.666328 (+ 338us) server_negotiation.cc:407] Connection header received
0809 12:13:25.666457 (+ 129us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 12:13:25.666461 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 12:13:25.666558 (+ 97us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 12:13:25.666644 (+ 86us) server_negotiation.cc:226] Negotiated authn=SASL
0809 12:13:25.668244 (+ 1600us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 12:13:25.669086 (+ 842us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 12:13:25.670414 (+ 1328us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 12:13:25.670807 (+ 393us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 12:13:25.670899 (+ 92us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 12:13:25.673565 (+ 2666us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 12:13:25.673569 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 12:13:25.673573 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0809 12:13:25.673605 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0809 12:13:25.676144 (+ 2539us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 12:13:25.676814 (+ 670us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 12:13:25.676818 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 12:13:25.676821 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0809 12:13:25.676949 (+ 128us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 12:13:25.677322 (+ 373us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 12:13:25.677326 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 12:13:25.677328 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0809 12:13:25.677664 (+ 336us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 12:13:25.677859 (+ 195us) server_negotiation.cc:1035] Waiting for connection context
0809 12:13:25.678118 (+ 259us) server_negotiation.cc:299] Negotiation successful
0809 12:13:25.678304 (+ 186us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":85}
I20260809 12:13:25.682940 15870 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:59024:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260809 12:13:25.690151 15870 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260809 12:13:25.712623 16363 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 12:13:25.702744 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.222.1:40557 (local address 127.0.0.1:58482)
0809 12:13:25.703665 (+ 921us) negotiation.cc:107] Waiting for socket to connect
0809 12:13:25.703714 (+ 49us) client_negotiation.cc:174] Beginning negotiation
0809 12:13:25.703897 (+ 183us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 12:13:25.705273 (+ 1376us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 12:13:25.705283 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 12:13:25.705351 (+ 68us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0809 12:13:25.705897 (+ 546us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 12:13:25.705916 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 12:13:25.708073 (+ 2157us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 12:13:25.708078 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 12:13:25.709774 (+ 1696us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 12:13:25.709788 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 12:13:25.711741 (+ 1953us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 12:13:25.711747 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 12:13:25.711960 (+ 213us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 12:13:25.712054 (+ 94us) client_negotiation.cc:769] Sending connection context
0809 12:13:25.712148 (+ 94us) client_negotiation.cc:240] Negotiation successful
0809 12:13:25.712295 (+ 147us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":681,"mutex_wait_us":1,"thread_start_us":2065,"threads_started":1}
I20260809 12:13:25.713966 16364 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 12:13:25.703122 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58482 (local address 127.0.222.1:40557)
0809 12:13:25.704595 (+ 1473us) server_negotiation.cc:206] Beginning negotiation
0809 12:13:25.704606 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0809 12:13:25.704638 (+ 32us) server_negotiation.cc:407] Connection header received
0809 12:13:25.704739 (+ 101us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 12:13:25.704746 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 12:13:25.705025 (+ 279us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 12:13:25.706196 (+ 1171us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0809 12:13:25.706452 (+ 256us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 12:13:25.707914 (+ 1462us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 12:13:25.709949 (+ 2035us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 12:13:25.711516 (+ 1567us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 12:13:25.713082 (+ 1566us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 12:13:25.713348 (+ 266us) server_negotiation.cc:1035] Waiting for connection context
0809 12:13:25.713419 (+ 71us) server_negotiation.cc:299] Negotiation successful
0809 12:13:25.713609 (+ 190us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1292,"thread_start_us":1119,"threads_started":1}
I20260809 12:13:25.714208 16362 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 12:13:25.702854 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.222.3:46283 (local address 127.0.0.1:53276)
0809 12:13:25.704890 (+ 2036us) negotiation.cc:107] Waiting for socket to connect
0809 12:13:25.704933 (+ 43us) client_negotiation.cc:174] Beginning negotiation
0809 12:13:25.705070 (+ 137us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 12:13:25.706162 (+ 1092us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 12:13:25.706167 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 12:13:25.706196 (+ 29us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0809 12:13:25.706543 (+ 347us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 12:13:25.706556 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 12:13:25.709405 (+ 2849us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 12:13:25.709411 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 12:13:25.711515 (+ 2104us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 12:13:25.711530 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 12:13:25.713261 (+ 1731us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 12:13:25.713266 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 12:13:25.713438 (+ 172us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 12:13:25.713467 (+ 29us) client_negotiation.cc:769] Sending connection context
0809 12:13:25.713559 (+ 92us) client_negotiation.cc:240] Negotiation successful
0809 12:13:25.713676 (+ 117us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1897,"mutex_wait_us":48,"thread_start_us":868,"threads_started":1}
I20260809 12:13:25.714247 16365 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 12:13:25.703464 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53276 (local address 127.0.222.3:46283)
0809 12:13:25.705457 (+ 1993us) server_negotiation.cc:206] Beginning negotiation
0809 12:13:25.705467 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0809 12:13:25.705519 (+ 52us) server_negotiation.cc:407] Connection header received
0809 12:13:25.705624 (+ 105us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 12:13:25.705633 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 12:13:25.705847 (+ 214us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 12:13:25.706729 (+ 882us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0809 12:13:25.707601 (+ 872us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 12:13:25.709260 (+ 1659us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 12:13:25.711712 (+ 2452us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 12:13:25.713057 (+ 1345us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 12:13:25.713192 (+ 135us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 12:13:25.713421 (+ 229us) server_negotiation.cc:1035] Waiting for connection context
0809 12:13:25.713629 (+ 208us) server_negotiation.cc:299] Negotiation successful
0809 12:13:25.713902 (+ 273us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1845,"thread_start_us":1758,"threads_started":1}
I20260809 12:13:25.714669 16366 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 12:13:25.704533 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.222.2:41313 (local address 127.0.0.1:52942)
0809 12:13:25.705873 (+ 1340us) negotiation.cc:107] Waiting for socket to connect
0809 12:13:25.705891 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0809 12:13:25.706024 (+ 133us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 12:13:25.707129 (+ 1105us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 12:13:25.707134 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 12:13:25.707159 (+ 25us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0809 12:13:25.707627 (+ 468us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 12:13:25.707638 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 12:13:25.709635 (+ 1997us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 12:13:25.709641 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 12:13:25.711908 (+ 2267us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 12:13:25.711923 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 12:13:25.713774 (+ 1851us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 12:13:25.713779 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 12:13:25.713950 (+ 171us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 12:13:25.713982 (+ 32us) client_negotiation.cc:769] Sending connection context
0809 12:13:25.714051 (+ 69us) client_negotiation.cc:240] Negotiation successful
0809 12:13:25.714178 (+ 127us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1208,"mutex_wait_us":2,"thread_start_us":1063,"threads_started":1}
I20260809 12:13:25.715359 16367 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 12:13:25.705091 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52942 (local address 127.0.222.2:41313)
0809 12:13:25.706542 (+ 1451us) server_negotiation.cc:206] Beginning negotiation
0809 12:13:25.706550 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0809 12:13:25.706578 (+ 28us) server_negotiation.cc:407] Connection header received
0809 12:13:25.706672 (+ 94us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 12:13:25.706679 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 12:13:25.706896 (+ 217us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 12:13:25.707135 (+ 239us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0809 12:13:25.707944 (+ 809us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 12:13:25.709472 (+ 1528us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 12:13:25.712078 (+ 2606us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 12:13:25.713515 (+ 1437us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 12:13:25.714312 (+ 797us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 12:13:25.714530 (+ 218us) server_negotiation.cc:1035] Waiting for connection context
0809 12:13:25.714603 (+ 73us) server_negotiation.cc:299] Negotiation successful
0809 12:13:25.715024 (+ 421us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1270,"thread_start_us":1155,"threads_started":1}
I20260809 12:13:25.718055 16281 tablet_service.cc:1467] Processing CreateTablet for tablet 5f0230cae1d54700be0f8714343a472b (DEFAULT_TABLE table=test-table [id=00d247d39e9345e5accd3e032377c913]), partition=RANGE (key) PARTITION UNBOUNDED
I20260809 12:13:25.718214 16002 tablet_service.cc:1467] Processing CreateTablet for tablet 5f0230cae1d54700be0f8714343a472b (DEFAULT_TABLE table=test-table [id=00d247d39e9345e5accd3e032377c913]), partition=RANGE (key) PARTITION UNBOUNDED
I20260809 12:13:25.719408 16281 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 5f0230cae1d54700be0f8714343a472b. 1 dirs total, 0 dirs full, 0 dirs failed
I20260809 12:13:25.718998 16140 tablet_service.cc:1467] Processing CreateTablet for tablet 5f0230cae1d54700be0f8714343a472b (DEFAULT_TABLE table=test-table [id=00d247d39e9345e5accd3e032377c913]), partition=RANGE (key) PARTITION UNBOUNDED
I20260809 12:13:25.719703 16002 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 5f0230cae1d54700be0f8714343a472b. 1 dirs total, 0 dirs full, 0 dirs failed
I20260809 12:13:25.720283 16140 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 5f0230cae1d54700be0f8714343a472b. 1 dirs total, 0 dirs full, 0 dirs failed
I20260809 12:13:25.731371 16368 tablet_bootstrap.cc:492] T 5f0230cae1d54700be0f8714343a472b P 1204c82bd36a468fbdf532c7aad13159: Bootstrap starting.
I20260809 12:13:25.735046 16368 tablet_bootstrap.cc:654] T 5f0230cae1d54700be0f8714343a472b P 1204c82bd36a468fbdf532c7aad13159: Neither blocks nor log segments found. Creating new log.
I20260809 12:13:25.736061 16369 tablet_bootstrap.cc:492] T 5f0230cae1d54700be0f8714343a472b P 2a033dd02e3d408eadf277165d4c5b95: Bootstrap starting.
I20260809 12:13:25.736730 16368 log.cc:824] T 5f0230cae1d54700be0f8714343a472b P 1204c82bd36a468fbdf532c7aad13159: Log is configured to *not* fsync() on all Append() calls
I20260809 12:13:25.737794 16370 tablet_bootstrap.cc:492] T 5f0230cae1d54700be0f8714343a472b P 5c1e1fc7cc4b4f59b9ca7298b6929b5c: Bootstrap starting.
I20260809 12:13:25.740085 16369 tablet_bootstrap.cc:654] T 5f0230cae1d54700be0f8714343a472b P 2a033dd02e3d408eadf277165d4c5b95: Neither blocks nor log segments found. Creating new log.
I20260809 12:13:25.741138 16370 tablet_bootstrap.cc:654] T 5f0230cae1d54700be0f8714343a472b P 5c1e1fc7cc4b4f59b9ca7298b6929b5c: Neither blocks nor log segments found. Creating new log.
I20260809 12:13:25.741338 16369 log.cc:824] T 5f0230cae1d54700be0f8714343a472b P 2a033dd02e3d408eadf277165d4c5b95: Log is configured to *not* fsync() on all Append() calls
I20260809 12:13:25.741904 16368 tablet_bootstrap.cc:492] T 5f0230cae1d54700be0f8714343a472b P 1204c82bd36a468fbdf532c7aad13159: No bootstrap required, opened a new log
I20260809 12:13:25.742251 16368 ts_tablet_manager.cc:1397] T 5f0230cae1d54700be0f8714343a472b P 1204c82bd36a468fbdf532c7aad13159: Time spent bootstrapping tablet: real 0.011s user 0.010s sys 0.000s
I20260809 12:13:25.742326 16370 log.cc:824] T 5f0230cae1d54700be0f8714343a472b P 5c1e1fc7cc4b4f59b9ca7298b6929b5c: Log is configured to *not* fsync() on all Append() calls
I20260809 12:13:25.745234 16369 tablet_bootstrap.cc:492] T 5f0230cae1d54700be0f8714343a472b P 2a033dd02e3d408eadf277165d4c5b95: No bootstrap required, opened a new log
I20260809 12:13:25.745445 16370 tablet_bootstrap.cc:492] T 5f0230cae1d54700be0f8714343a472b P 5c1e1fc7cc4b4f59b9ca7298b6929b5c: No bootstrap required, opened a new log
I20260809 12:13:25.745497 16369 ts_tablet_manager.cc:1397] T 5f0230cae1d54700be0f8714343a472b P 2a033dd02e3d408eadf277165d4c5b95: Time spent bootstrapping tablet: real 0.010s user 0.001s sys 0.005s
I20260809 12:13:25.745705 16370 ts_tablet_manager.cc:1397] T 5f0230cae1d54700be0f8714343a472b P 5c1e1fc7cc4b4f59b9ca7298b6929b5c: Time spent bootstrapping tablet: real 0.008s user 0.006s sys 0.000s
I20260809 12:13:25.752938 16370 raft_consensus.cc:348] T 5f0230cae1d54700be0f8714343a472b P 5c1e1fc7cc4b4f59b9ca7298b6929b5c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2a033dd02e3d408eadf277165d4c5b95" member_type: VOTER last_known_addr { host: "127.0.222.1" port: 40557 } } peers { permanent_uuid: "1204c82bd36a468fbdf532c7aad13159" member_type: VOTER last_known_addr { host: "127.0.222.3" port: 46283 } } peers { permanent_uuid: "5c1e1fc7cc4b4f59b9ca7298b6929b5c" member_type: VOTER last_known_addr { host: "127.0.222.2" port: 41313 } }
I20260809 12:13:25.753355 16370 raft_consensus.cc:374] T 5f0230cae1d54700be0f8714343a472b P 5c1e1fc7cc4b4f59b9ca7298b6929b5c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260809 12:13:25.753067 16368 raft_consensus.cc:348] T 5f0230cae1d54700be0f8714343a472b P 1204c82bd36a468fbdf532c7aad13159 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2a033dd02e3d408eadf277165d4c5b95" member_type: VOTER last_known_addr { host: "127.0.222.1" port: 40557 } } peers { permanent_uuid: "1204c82bd36a468fbdf532c7aad13159" member_type: VOTER last_known_addr { host: "127.0.222.3" port: 46283 } } peers { permanent_uuid: "5c1e1fc7cc4b4f59b9ca7298b6929b5c" member_type: VOTER last_known_addr { host: "127.0.222.2" port: 41313 } }
I20260809 12:13:25.753494 16368 raft_consensus.cc:374] T 5f0230cae1d54700be0f8714343a472b P 1204c82bd36a468fbdf532c7aad13159 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260809 12:13:25.753499 16370 raft_consensus.cc:729] T 5f0230cae1d54700be0f8714343a472b P 5c1e1fc7cc4b4f59b9ca7298b6929b5c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5c1e1fc7cc4b4f59b9ca7298b6929b5c, State: Initialized, Role: FOLLOWER
I20260809 12:13:25.753607 16368 raft_consensus.cc:729] T 5f0230cae1d54700be0f8714343a472b P 1204c82bd36a468fbdf532c7aad13159 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1204c82bd36a468fbdf532c7aad13159, State: Initialized, Role: FOLLOWER
I20260809 12:13:25.754434 16368 consensus_queue.cc:260] T 5f0230cae1d54700be0f8714343a472b P 1204c82bd36a468fbdf532c7aad13159 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2a033dd02e3d408eadf277165d4c5b95" member_type: VOTER last_known_addr { host: "127.0.222.1" port: 40557 } } peers { permanent_uuid: "1204c82bd36a468fbdf532c7aad13159" member_type: VOTER last_known_addr { host: "127.0.222.3" port: 46283 } } peers { permanent_uuid: "5c1e1fc7cc4b4f59b9ca7298b6929b5c" member_type: VOTER last_known_addr { host: "127.0.222.2" port: 41313 } }
I20260809 12:13:25.754350 16370 consensus_queue.cc:260] T 5f0230cae1d54700be0f8714343a472b P 5c1e1fc7cc4b4f59b9ca7298b6929b5c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2a033dd02e3d408eadf277165d4c5b95" member_type: VOTER last_known_addr { host: "127.0.222.1" port: 40557 } } peers { permanent_uuid: "1204c82bd36a468fbdf532c7aad13159" member_type: VOTER last_known_addr { host: "127.0.222.3" port: 46283 } } peers { permanent_uuid: "5c1e1fc7cc4b4f59b9ca7298b6929b5c" member_type: VOTER last_known_addr { host: "127.0.222.2" port: 41313 } }
I20260809 12:13:25.756167 16207 heartbeater.cc:503] Master 127.0.222.62:44147 was elected leader, sending a full tablet report...
I20260809 12:13:25.757022 16370 ts_tablet_manager.cc:1428] T 5f0230cae1d54700be0f8714343a472b P 5c1e1fc7cc4b4f59b9ca7298b6929b5c: Time spent starting tablet: real 0.011s user 0.011s sys 0.000s
I20260809 12:13:25.756690 16369 raft_consensus.cc:348] T 5f0230cae1d54700be0f8714343a472b P 2a033dd02e3d408eadf277165d4c5b95 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2a033dd02e3d408eadf277165d4c5b95" member_type: VOTER last_known_addr { host: "127.0.222.1" port: 40557 } } peers { permanent_uuid: "1204c82bd36a468fbdf532c7aad13159" member_type: VOTER last_known_addr { host: "127.0.222.3" port: 46283 } } peers { permanent_uuid: "5c1e1fc7cc4b4f59b9ca7298b6929b5c" member_type: VOTER last_known_addr { host: "127.0.222.2" port: 41313 } }
I20260809 12:13:25.757253 16369 raft_consensus.cc:374] T 5f0230cae1d54700be0f8714343a472b P 2a033dd02e3d408eadf277165d4c5b95 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260809 12:13:25.757357 16369 raft_consensus.cc:729] T 5f0230cae1d54700be0f8714343a472b P 2a033dd02e3d408eadf277165d4c5b95 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2a033dd02e3d408eadf277165d4c5b95, State: Initialized, Role: FOLLOWER
I20260809 12:13:25.757978 16369 consensus_queue.cc:260] T 5f0230cae1d54700be0f8714343a472b P 2a033dd02e3d408eadf277165d4c5b95 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2a033dd02e3d408eadf277165d4c5b95" member_type: VOTER last_known_addr { host: "127.0.222.1" port: 40557 } } peers { permanent_uuid: "1204c82bd36a468fbdf532c7aad13159" member_type: VOTER last_known_addr { host: "127.0.222.3" port: 46283 } } peers { permanent_uuid: "5c1e1fc7cc4b4f59b9ca7298b6929b5c" member_type: VOTER last_known_addr { host: "127.0.222.2" port: 41313 } }
I20260809 12:13:25.758582 16348 heartbeater.cc:503] Master 127.0.222.62:44147 was elected leader, sending a full tablet report...
I20260809 12:13:25.758612 16368 ts_tablet_manager.cc:1428] T 5f0230cae1d54700be0f8714343a472b P 1204c82bd36a468fbdf532c7aad13159: Time spent starting tablet: real 0.016s user 0.012s sys 0.004s
I20260809 12:13:25.760455 16369 ts_tablet_manager.cc:1428] T 5f0230cae1d54700be0f8714343a472b P 2a033dd02e3d408eadf277165d4c5b95: Time spent starting tablet: real 0.015s user 0.014s sys 0.002s
W20260809 12:13:25.772389 16070 tablet.cc:2369] T 5f0230cae1d54700be0f8714343a472b P 2a033dd02e3d408eadf277165d4c5b95: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260809 12:13:25.824978 16374 raft_consensus.cc:482] T 5f0230cae1d54700be0f8714343a472b P 1204c82bd36a468fbdf532c7aad13159 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260809 12:13:25.825366 16374 raft_consensus.cc:504] T 5f0230cae1d54700be0f8714343a472b P 1204c82bd36a468fbdf532c7aad13159 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2a033dd02e3d408eadf277165d4c5b95" member_type: VOTER last_known_addr { host: "127.0.222.1" port: 40557 } } peers { permanent_uuid: "1204c82bd36a468fbdf532c7aad13159" member_type: VOTER last_known_addr { host: "127.0.222.3" port: 46283 } } peers { permanent_uuid: "5c1e1fc7cc4b4f59b9ca7298b6929b5c" member_type: VOTER last_known_addr { host: "127.0.222.2" port: 41313 } }
I20260809 12:13:25.827319 16374 leader_election.cc:288] T 5f0230cae1d54700be0f8714343a472b P 1204c82bd36a468fbdf532c7aad13159 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 2a033dd02e3d408eadf277165d4c5b95 (127.0.222.1:40557), 5c1e1fc7cc4b4f59b9ca7298b6929b5c (127.0.222.2:41313)
I20260809 12:13:25.827968 16375 raft_consensus.cc:482] T 5f0230cae1d54700be0f8714343a472b P 5c1e1fc7cc4b4f59b9ca7298b6929b5c [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260809 12:13:25.828621 16375 raft_consensus.cc:504] T 5f0230cae1d54700be0f8714343a472b P 5c1e1fc7cc4b4f59b9ca7298b6929b5c [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2a033dd02e3d408eadf277165d4c5b95" member_type: VOTER last_known_addr { host: "127.0.222.1" port: 40557 } } peers { permanent_uuid: "1204c82bd36a468fbdf532c7aad13159" member_type: VOTER last_known_addr { host: "127.0.222.3" port: 46283 } } peers { permanent_uuid: "5c1e1fc7cc4b4f59b9ca7298b6929b5c" member_type: VOTER last_known_addr { host: "127.0.222.2" port: 41313 } }
I20260809 12:13:25.833595 16375 leader_election.cc:288] T 5f0230cae1d54700be0f8714343a472b P 5c1e1fc7cc4b4f59b9ca7298b6929b5c [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 2a033dd02e3d408eadf277165d4c5b95 (127.0.222.1:40557), 1204c82bd36a468fbdf532c7aad13159 (127.0.222.3:46283)
I20260809 12:13:25.836081 16352 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 12:13:25.827515 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.222.1:40557 (local address 127.0.222.3:45845)
0809 12:13:25.827825 (+ 310us) negotiation.cc:107] Waiting for socket to connect
0809 12:13:25.827847 (+ 22us) client_negotiation.cc:174] Beginning negotiation
0809 12:13:25.827982 (+ 135us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 12:13:25.828913 (+ 931us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 12:13:25.828920 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 12:13:25.828948 (+ 28us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0809 12:13:25.829145 (+ 197us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 12:13:25.829158 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 12:13:25.830624 (+ 1466us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 12:13:25.830630 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 12:13:25.833225 (+ 2595us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 12:13:25.833249 (+ 24us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 12:13:25.835329 (+ 2080us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 12:13:25.835336 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 12:13:25.835527 (+ 191us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 12:13:25.835567 (+ 40us) client_negotiation.cc:769] Sending connection context
0809 12:13:25.835641 (+ 74us) client_negotiation.cc:240] Negotiation successful
0809 12:13:25.835785 (+ 144us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":155,"mutex_wait_us":1}
I20260809 12:13:25.837229 16364 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 12:13:25.827936 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.222.3:45845 (local address 127.0.222.1:40557)
0809 12:13:25.828237 (+ 301us) server_negotiation.cc:206] Beginning negotiation
0809 12:13:25.828243 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0809 12:13:25.828273 (+ 30us) server_negotiation.cc:407] Connection header received
0809 12:13:25.828368 (+ 95us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 12:13:25.828373 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 12:13:25.828445 (+ 72us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 12:13:25.828526 (+ 81us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0809 12:13:25.829364 (+ 838us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 12:13:25.830414 (+ 1050us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 12:13:25.833403 (+ 2989us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 12:13:25.835122 (+ 1719us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 12:13:25.836627 (+ 1505us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 12:13:25.836752 (+ 125us) server_negotiation.cc:1035] Waiting for connection context
0809 12:13:25.836803 (+ 51us) server_negotiation.cc:299] Negotiation successful
0809 12:13:25.836938 (+ 135us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":156}
I20260809 12:13:25.838272 16367 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 12:13:25.828133 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.222.3:46715 (local address 127.0.222.2:41313)
0809 12:13:25.828830 (+ 697us) server_negotiation.cc:206] Beginning negotiation
0809 12:13:25.828836 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0809 12:13:25.829781 (+ 945us) server_negotiation.cc:407] Connection header received
0809 12:13:25.829971 (+ 190us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 12:13:25.829976 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 12:13:25.830041 (+ 65us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 12:13:25.830140 (+ 99us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0809 12:13:25.830770 (+ 630us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 12:13:25.832075 (+ 1305us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 12:13:25.835213 (+ 3138us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 12:13:25.836627 (+ 1414us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 12:13:25.836774 (+ 147us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 12:13:25.836882 (+ 108us) server_negotiation.cc:1035] Waiting for connection context
0809 12:13:25.837850 (+ 968us) server_negotiation.cc:299] Negotiation successful
0809 12:13:25.838014 (+ 164us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":530}
I20260809 12:13:25.838416 16023 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5f0230cae1d54700be0f8714343a472b" candidate_uuid: "1204c82bd36a468fbdf532c7aad13159" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2a033dd02e3d408eadf277165d4c5b95" is_pre_election: true
I20260809 12:13:25.839025 16023 raft_consensus.cc:2454] T 5f0230cae1d54700be0f8714343a472b P 2a033dd02e3d408eadf277165d4c5b95 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1204c82bd36a468fbdf532c7aad13159 in term 0.
W20260809 12:13:25.839088 16208 tablet.cc:2369] T 5f0230cae1d54700be0f8714343a472b P 5c1e1fc7cc4b4f59b9ca7298b6929b5c: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260809 12:13:25.840054 16231 leader_election.cc:302] T 5f0230cae1d54700be0f8714343a472b P 1204c82bd36a468fbdf532c7aad13159 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1204c82bd36a468fbdf532c7aad13159, 2a033dd02e3d408eadf277165d4c5b95; no voters:
I20260809 12:13:25.840550 16374 raft_consensus.cc:2781] T 5f0230cae1d54700be0f8714343a472b P 1204c82bd36a468fbdf532c7aad13159 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260809 12:13:25.840734 16374 raft_consensus.cc:482] T 5f0230cae1d54700be0f8714343a472b P 1204c82bd36a468fbdf532c7aad13159 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260809 12:13:25.840551 16377 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 12:13:25.827682 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.222.2:41313 (local address 127.0.222.3:46715)
0809 12:13:25.829649 (+ 1967us) negotiation.cc:107] Waiting for socket to connect
0809 12:13:25.829668 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0809 12:13:25.829814 (+ 146us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 12:13:25.830174 (+ 360us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 12:13:25.830178 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 12:13:25.830200 (+ 22us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0809 12:13:25.830568 (+ 368us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 12:13:25.830581 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 12:13:25.832347 (+ 1766us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 12:13:25.832354 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 12:13:25.835038 (+ 2684us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 12:13:25.835053 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 12:13:25.837417 (+ 2364us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 12:13:25.837423 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 12:13:25.837625 (+ 202us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 12:13:25.837657 (+ 32us) client_negotiation.cc:769] Sending connection context
0809 12:13:25.838514 (+ 857us) client_negotiation.cc:240] Negotiation successful
0809 12:13:25.838670 (+ 156us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1838,"mutex_wait_us":282,"thread_start_us":1624,"threads_started":1}
I20260809 12:13:25.840868 16374 raft_consensus.cc:3037] T 5f0230cae1d54700be0f8714343a472b P 1204c82bd36a468fbdf532c7aad13159 [term 0 FOLLOWER]: Advancing to term 1
I20260809 12:13:25.841931 16160 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5f0230cae1d54700be0f8714343a472b" candidate_uuid: "1204c82bd36a468fbdf532c7aad13159" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5c1e1fc7cc4b4f59b9ca7298b6929b5c" is_pre_election: true
I20260809 12:13:25.842425 16160 raft_consensus.cc:2454] T 5f0230cae1d54700be0f8714343a472b P 5c1e1fc7cc4b4f59b9ca7298b6929b5c [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1204c82bd36a468fbdf532c7aad13159 in term 0.
I20260809 12:13:25.843302 16379 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 12:13:25.830904 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.222.1:40557 (local address 127.0.222.2:39287)
0809 12:13:25.833662 (+ 2758us) negotiation.cc:107] Waiting for socket to connect
0809 12:13:25.833681 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0809 12:13:25.833844 (+ 163us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 12:13:25.835025 (+ 1181us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 12:13:25.835030 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 12:13:25.835058 (+ 28us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0809 12:13:25.835699 (+ 641us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 12:13:25.835712 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 12:13:25.837384 (+ 1672us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 12:13:25.837389 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 12:13:25.839686 (+ 2297us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 12:13:25.839701 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 12:13:25.842357 (+ 2656us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 12:13:25.842368 (+ 11us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 12:13:25.842563 (+ 195us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 12:13:25.842622 (+ 59us) client_negotiation.cc:769] Sending connection context
0809 12:13:25.842707 (+ 85us) client_negotiation.cc:240] Negotiation successful
0809 12:13:25.842874 (+ 167us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":2605,"mutex_wait_us":5,"thread_start_us":9101,"threads_started":1}
I20260809 12:13:25.843349 16378 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 12:13:25.831191 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.222.2:39287 (local address 127.0.222.1:40557)
0809 12:13:25.834432 (+ 3241us) server_negotiation.cc:206] Beginning negotiation
0809 12:13:25.834438 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0809 12:13:25.834461 (+ 23us) server_negotiation.cc:407] Connection header received
0809 12:13:25.834523 (+ 62us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 12:13:25.834530 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 12:13:25.834600 (+ 70us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 12:13:25.834699 (+ 99us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0809 12:13:25.835933 (+ 1234us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 12:13:25.837228 (+ 1295us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 12:13:25.839873 (+ 2645us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 12:13:25.841173 (+ 1300us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 12:13:25.841288 (+ 115us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 12:13:25.841398 (+ 110us) server_negotiation.cc:1035] Waiting for connection context
0809 12:13:25.842844 (+ 1446us) server_negotiation.cc:299] Negotiation successful
0809 12:13:25.843029 (+ 185us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":3093,"thread_start_us":2823,"threads_started":1}
I20260809 12:13:25.844313 16380 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 12:13:25.831184 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.222.3:46283 (local address 127.0.222.2:48793)
0809 12:13:25.835772 (+ 4588us) negotiation.cc:107] Waiting for socket to connect
0809 12:13:25.835796 (+ 24us) client_negotiation.cc:174] Beginning negotiation
0809 12:13:25.835965 (+ 169us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 12:13:25.836494 (+ 529us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 12:13:25.836499 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 12:13:25.836531 (+ 32us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0809 12:13:25.836952 (+ 421us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 12:13:25.836963 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 12:13:25.838445 (+ 1482us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 12:13:25.838451 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 12:13:25.840543 (+ 2092us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 12:13:25.840560 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 12:13:25.842578 (+ 2018us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 12:13:25.842586 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 12:13:25.843613 (+ 1027us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 12:13:25.843647 (+ 34us) client_negotiation.cc:769] Sending connection context
0809 12:13:25.843797 (+ 150us) client_negotiation.cc:240] Negotiation successful
0809 12:13:25.843926 (+ 129us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":4439,"mutex_wait_us":30,"thread_start_us":2083,"threads_started":1}
I20260809 12:13:25.844278 16023 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5f0230cae1d54700be0f8714343a472b" candidate_uuid: "5c1e1fc7cc4b4f59b9ca7298b6929b5c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2a033dd02e3d408eadf277165d4c5b95" is_pre_election: true
I20260809 12:13:25.844419 16365 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 12:13:25.831629 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.222.2:48793 (local address 127.0.222.3:46283)
0809 12:13:25.831809 (+ 180us) server_negotiation.cc:206] Beginning negotiation
0809 12:13:25.831814 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0809 12:13:25.835867 (+ 4053us) server_negotiation.cc:407] Connection header received
0809 12:13:25.836141 (+ 274us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 12:13:25.836146 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 12:13:25.836224 (+ 78us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 12:13:25.836288 (+ 64us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0809 12:13:25.837131 (+ 843us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 12:13:25.838246 (+ 1115us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 12:13:25.840740 (+ 2494us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 12:13:25.842117 (+ 1377us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 12:13:25.842232 (+ 115us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 12:13:25.842341 (+ 109us) server_negotiation.cc:1035] Waiting for connection context
0809 12:13:25.843887 (+ 1546us) server_negotiation.cc:299] Negotiation successful
0809 12:13:25.844069 (+ 182us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":55}
I20260809 12:13:25.844250 16374 raft_consensus.cc:504] T 5f0230cae1d54700be0f8714343a472b P 1204c82bd36a468fbdf532c7aad13159 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2a033dd02e3d408eadf277165d4c5b95" member_type: VOTER last_known_addr { host: "127.0.222.1" port: 40557 } } peers { permanent_uuid: "1204c82bd36a468fbdf532c7aad13159" member_type: VOTER last_known_addr { host: "127.0.222.3" port: 46283 } } peers { permanent_uuid: "5c1e1fc7cc4b4f59b9ca7298b6929b5c" member_type: VOTER last_known_addr { host: "127.0.222.2" port: 41313 } }
I20260809 12:13:25.844521 16023 raft_consensus.cc:2454] T 5f0230cae1d54700be0f8714343a472b P 2a033dd02e3d408eadf277165d4c5b95 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 5c1e1fc7cc4b4f59b9ca7298b6929b5c in term 0.
I20260809 12:13:25.845034 16093 leader_election.cc:302] T 5f0230cae1d54700be0f8714343a472b P 5c1e1fc7cc4b4f59b9ca7298b6929b5c [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2a033dd02e3d408eadf277165d4c5b95, 5c1e1fc7cc4b4f59b9ca7298b6929b5c; no voters:
I20260809 12:13:25.845487 16375 raft_consensus.cc:2781] T 5f0230cae1d54700be0f8714343a472b P 5c1e1fc7cc4b4f59b9ca7298b6929b5c [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260809 12:13:25.845671 16375 raft_consensus.cc:482] T 5f0230cae1d54700be0f8714343a472b P 5c1e1fc7cc4b4f59b9ca7298b6929b5c [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260809 12:13:25.845458 16301 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5f0230cae1d54700be0f8714343a472b" candidate_uuid: "5c1e1fc7cc4b4f59b9ca7298b6929b5c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1204c82bd36a468fbdf532c7aad13159" is_pre_election: true
I20260809 12:13:25.845700 16160 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5f0230cae1d54700be0f8714343a472b" candidate_uuid: "1204c82bd36a468fbdf532c7aad13159" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5c1e1fc7cc4b4f59b9ca7298b6929b5c"
I20260809 12:13:25.845793 16375 raft_consensus.cc:3037] T 5f0230cae1d54700be0f8714343a472b P 5c1e1fc7cc4b4f59b9ca7298b6929b5c [term 0 FOLLOWER]: Advancing to term 1
I20260809 12:13:25.845839 16023 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5f0230cae1d54700be0f8714343a472b" candidate_uuid: "1204c82bd36a468fbdf532c7aad13159" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2a033dd02e3d408eadf277165d4c5b95"
I20260809 12:13:25.845988 16301 raft_consensus.cc:2379] T 5f0230cae1d54700be0f8714343a472b P 1204c82bd36a468fbdf532c7aad13159 [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 5c1e1fc7cc4b4f59b9ca7298b6929b5c in current term 1: Already voted for candidate 1204c82bd36a468fbdf532c7aad13159 in this term.
I20260809 12:13:25.846047 16023 raft_consensus.cc:3037] T 5f0230cae1d54700be0f8714343a472b P 2a033dd02e3d408eadf277165d4c5b95 [term 0 FOLLOWER]: Advancing to term 1
I20260809 12:13:25.846143 16374 leader_election.cc:288] T 5f0230cae1d54700be0f8714343a472b P 1204c82bd36a468fbdf532c7aad13159 [CANDIDATE]: Term 1 election: Requested vote from peers 2a033dd02e3d408eadf277165d4c5b95 (127.0.222.1:40557), 5c1e1fc7cc4b4f59b9ca7298b6929b5c (127.0.222.2:41313)
I20260809 12:13:25.849223 16023 raft_consensus.cc:2454] T 5f0230cae1d54700be0f8714343a472b P 2a033dd02e3d408eadf277165d4c5b95 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1204c82bd36a468fbdf532c7aad13159 in term 1.
I20260809 12:13:25.849133 16375 raft_consensus.cc:504] T 5f0230cae1d54700be0f8714343a472b P 5c1e1fc7cc4b4f59b9ca7298b6929b5c [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2a033dd02e3d408eadf277165d4c5b95" member_type: VOTER last_known_addr { host: "127.0.222.1" port: 40557 } } peers { permanent_uuid: "1204c82bd36a468fbdf532c7aad13159" member_type: VOTER last_known_addr { host: "127.0.222.3" port: 46283 } } peers { permanent_uuid: "5c1e1fc7cc4b4f59b9ca7298b6929b5c" member_type: VOTER last_known_addr { host: "127.0.222.2" port: 41313 } }
I20260809 12:13:25.849610 16160 raft_consensus.cc:2379] T 5f0230cae1d54700be0f8714343a472b P 5c1e1fc7cc4b4f59b9ca7298b6929b5c [term 1 FOLLOWER]: Leader election vote request: Denying vote to candidate 1204c82bd36a468fbdf532c7aad13159 in current term 1: Already voted for candidate 5c1e1fc7cc4b4f59b9ca7298b6929b5c in this term.
I20260809 12:13:25.849633 16231 leader_election.cc:302] T 5f0230cae1d54700be0f8714343a472b P 1204c82bd36a468fbdf532c7aad13159 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1204c82bd36a468fbdf532c7aad13159, 2a033dd02e3d408eadf277165d4c5b95; no voters:
I20260809 12:13:25.849936 16374 raft_consensus.cc:2781] T 5f0230cae1d54700be0f8714343a472b P 1204c82bd36a468fbdf532c7aad13159 [term 1 FOLLOWER]: Leader election won for term 1
I20260809 12:13:25.850051 16375 leader_election.cc:288] T 5f0230cae1d54700be0f8714343a472b P 5c1e1fc7cc4b4f59b9ca7298b6929b5c [CANDIDATE]: Term 1 election: Requested vote from peers 2a033dd02e3d408eadf277165d4c5b95 (127.0.222.1:40557), 1204c82bd36a468fbdf532c7aad13159 (127.0.222.3:46283)
I20260809 12:13:25.850262 16023 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5f0230cae1d54700be0f8714343a472b" candidate_uuid: "5c1e1fc7cc4b4f59b9ca7298b6929b5c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2a033dd02e3d408eadf277165d4c5b95"
I20260809 12:13:25.850436 16301 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5f0230cae1d54700be0f8714343a472b" candidate_uuid: "5c1e1fc7cc4b4f59b9ca7298b6929b5c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1204c82bd36a468fbdf532c7aad13159"
I20260809 12:13:25.850598 16023 raft_consensus.cc:2379] T 5f0230cae1d54700be0f8714343a472b P 2a033dd02e3d408eadf277165d4c5b95 [term 1 FOLLOWER]: Leader election vote request: Denying vote to candidate 5c1e1fc7cc4b4f59b9ca7298b6929b5c in current term 1: Already voted for candidate 1204c82bd36a468fbdf532c7aad13159 in this term.
I20260809 12:13:25.851337 16374 raft_consensus.cc:686] T 5f0230cae1d54700be0f8714343a472b P 1204c82bd36a468fbdf532c7aad13159 [term 1 LEADER]: Becoming Leader. State: Replica: 1204c82bd36a468fbdf532c7aad13159, State: Running, Role: LEADER
I20260809 12:13:25.851876 16374 consensus_queue.cc:237] T 5f0230cae1d54700be0f8714343a472b P 1204c82bd36a468fbdf532c7aad13159 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2a033dd02e3d408eadf277165d4c5b95" member_type: VOTER last_known_addr { host: "127.0.222.1" port: 40557 } } peers { permanent_uuid: "1204c82bd36a468fbdf532c7aad13159" member_type: VOTER last_known_addr { host: "127.0.222.3" port: 46283 } } peers { permanent_uuid: "5c1e1fc7cc4b4f59b9ca7298b6929b5c" member_type: VOTER last_known_addr { host: "127.0.222.2" port: 41313 } }
I20260809 12:13:25.855546 16091 leader_election.cc:302] T 5f0230cae1d54700be0f8714343a472b P 5c1e1fc7cc4b4f59b9ca7298b6929b5c [CANDIDATE]: Term 1 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 5c1e1fc7cc4b4f59b9ca7298b6929b5c; no voters: 1204c82bd36a468fbdf532c7aad13159, 2a033dd02e3d408eadf277165d4c5b95
I20260809 12:13:25.855901 16375 raft_consensus.cc:2726] T 5f0230cae1d54700be0f8714343a472b P 5c1e1fc7cc4b4f59b9ca7298b6929b5c [term 1 FOLLOWER]: Leader election lost for term 1. Reason: could not achieve majority
I20260809 12:13:25.856675 15868 catalog_manager.cc:5495] T 5f0230cae1d54700be0f8714343a472b P 1204c82bd36a468fbdf532c7aad13159 reported cstate change: term changed from 0 to 1, leader changed from <none> to 1204c82bd36a468fbdf532c7aad13159 (127.0.222.3). New cstate: current_term: 1 leader_uuid: "1204c82bd36a468fbdf532c7aad13159" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2a033dd02e3d408eadf277165d4c5b95" member_type: VOTER last_known_addr { host: "127.0.222.1" port: 40557 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "1204c82bd36a468fbdf532c7aad13159" member_type: VOTER last_known_addr { host: "127.0.222.3" port: 46283 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "5c1e1fc7cc4b4f59b9ca7298b6929b5c" member_type: VOTER last_known_addr { host: "127.0.222.2" port: 41313 } health_report { overall_health: UNKNOWN } } }
W20260809 12:13:25.878378 16350 tablet.cc:2369] T 5f0230cae1d54700be0f8714343a472b P 1204c82bd36a468fbdf532c7aad13159: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260809 12:13:25.908694 16365 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 12:13:25.903313 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53278 (local address 127.0.222.3:46283)
0809 12:13:25.903480 (+ 167us) server_negotiation.cc:206] Beginning negotiation
0809 12:13:25.903485 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0809 12:13:25.903520 (+ 35us) server_negotiation.cc:407] Connection header received
0809 12:13:25.903736 (+ 216us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 12:13:25.903740 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 12:13:25.903830 (+ 90us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 12:13:25.903958 (+ 128us) server_negotiation.cc:226] Negotiated authn=TOKEN
0809 12:13:25.904360 (+ 402us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 12:13:25.905154 (+ 794us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 12:13:25.906661 (+ 1507us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 12:13:25.907082 (+ 421us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 12:13:25.907178 (+ 96us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 12:13:25.907691 (+ 513us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0809 12:13:25.908207 (+ 516us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0809 12:13:25.908340 (+ 133us) server_negotiation.cc:1035] Waiting for connection context
0809 12:13:25.908401 (+ 61us) server_negotiation.cc:299] Negotiation successful
0809 12:13:25.908465 (+ 64us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":57}
I20260809 12:13:25.917124 16160 raft_consensus.cc:1261] T 5f0230cae1d54700be0f8714343a472b P 5c1e1fc7cc4b4f59b9ca7298b6929b5c [term 1 FOLLOWER]: Refusing update from remote peer 1204c82bd36a468fbdf532c7aad13159: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260809 12:13:25.917107 16023 raft_consensus.cc:1261] T 5f0230cae1d54700be0f8714343a472b P 2a033dd02e3d408eadf277165d4c5b95 [term 1 FOLLOWER]: Refusing update from remote peer 1204c82bd36a468fbdf532c7aad13159: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260809 12:13:25.918098 16382 consensus_queue.cc:1035] T 5f0230cae1d54700be0f8714343a472b P 1204c82bd36a468fbdf532c7aad13159 [LEADER]: Connected to new peer: Peer: permanent_uuid: "2a033dd02e3d408eadf277165d4c5b95" member_type: VOTER last_known_addr { host: "127.0.222.1" port: 40557 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260809 12:13:25.918490 16374 consensus_queue.cc:1035] T 5f0230cae1d54700be0f8714343a472b P 1204c82bd36a468fbdf532c7aad13159 [LEADER]: Connected to new peer: Peer: permanent_uuid: "5c1e1fc7cc4b4f59b9ca7298b6929b5c" member_type: VOTER last_known_addr { host: "127.0.222.2" port: 41313 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260809 12:13:25.932554 16385 mvcc.cc:205] Tried to move back new op lower bound from 7316593073825918976 to 7316593073572933632. Current Snapshot: MvccSnapshot[applied={T|T < 7316593073825918976}]
I20260809 12:13:25.957458 15868 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:59024:
table { table_name: "test-table" } modify_external_catalogs: true
I20260809 12:13:25.957759 15868 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:59024:
table { table_name: "test-table" } modify_external_catalogs: true
I20260809 12:13:25.961571 15868 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 6d7f8671e34d46459506bc2cba79c82c: Sending DeleteTablet for 3 replicas of tablet 5f0230cae1d54700be0f8714343a472b
I20260809 12:13:25.962987 16002 tablet_service.cc:1514] Processing DeleteTablet for tablet 5f0230cae1d54700be0f8714343a472b with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-09 12:13:25 UTC) from {username='kudu', principal='kudu/127.0.222.62@KRBTEST.COM'} at 127.0.0.1:58482
I20260809 12:13:25.963169 16140 tablet_service.cc:1514] Processing DeleteTablet for tablet 5f0230cae1d54700be0f8714343a472b with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-09 12:13:25 UTC) from {username='kudu', principal='kudu/127.0.222.62@KRBTEST.COM'} at 127.0.0.1:52942
I20260809 12:13:25.963233 16281 tablet_service.cc:1514] Processing DeleteTablet for tablet 5f0230cae1d54700be0f8714343a472b with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-09 12:13:25 UTC) from {username='kudu', principal='kudu/127.0.222.62@KRBTEST.COM'} at 127.0.0.1:53276
I20260809 12:13:25.964310 888 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskgH5Pjm/build/asan/bin/kudu pid 15934
I20260809 12:13:25.972470 16396 tablet_replica.cc:330] T 5f0230cae1d54700be0f8714343a472b P 5c1e1fc7cc4b4f59b9ca7298b6929b5c: stopping tablet replica
I20260809 12:13:25.973141 16396 raft_consensus.cc:2229] T 5f0230cae1d54700be0f8714343a472b P 5c1e1fc7cc4b4f59b9ca7298b6929b5c [term 1 FOLLOWER]: Raft consensus shutting down.
I20260809 12:13:25.973824 16396 raft_consensus.cc:2258] T 5f0230cae1d54700be0f8714343a472b P 5c1e1fc7cc4b4f59b9ca7298b6929b5c [term 1 FOLLOWER]: Raft consensus is shut down!
I20260809 12:13:25.974797 16395 tablet_replica.cc:330] T 5f0230cae1d54700be0f8714343a472b P 1204c82bd36a468fbdf532c7aad13159: stopping tablet replica
I20260809 12:13:25.975216 16395 raft_consensus.cc:2229] T 5f0230cae1d54700be0f8714343a472b P 1204c82bd36a468fbdf532c7aad13159 [term 1 LEADER]: Raft consensus shutting down.
I20260809 12:13:25.976168 16394 tablet_replica.cc:330] T 5f0230cae1d54700be0f8714343a472b P 2a033dd02e3d408eadf277165d4c5b95: stopping tablet replica
I20260809 12:13:25.976616 16394 raft_consensus.cc:2229] T 5f0230cae1d54700be0f8714343a472b P 2a033dd02e3d408eadf277165d4c5b95 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260809 12:13:25.977144 16394 raft_consensus.cc:2258] T 5f0230cae1d54700be0f8714343a472b P 2a033dd02e3d408eadf277165d4c5b95 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260809 12:13:25.975921 16395 raft_consensus.cc:2258] T 5f0230cae1d54700be0f8714343a472b P 1204c82bd36a468fbdf532c7aad13159 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260809 12:13:25.979506 16394 ts_tablet_manager.cc:1905] T 5f0230cae1d54700be0f8714343a472b P 2a033dd02e3d408eadf277165d4c5b95: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260809 12:13:25.979770 16395 ts_tablet_manager.cc:1905] T 5f0230cae1d54700be0f8714343a472b P 1204c82bd36a468fbdf532c7aad13159: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260809 12:13:25.980319 16396 ts_tablet_manager.cc:1905] T 5f0230cae1d54700be0f8714343a472b P 5c1e1fc7cc4b4f59b9ca7298b6929b5c: Deleting tablet data with delete state TABLET_DATA_DELETED
Aug 09 12:13:25 dist-test-slave-2w3w krb5kdc[15823](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786277603, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.222.1@KRBTEST.COM
I20260809 12:13:25.987690 16395 ts_tablet_manager.cc:1918] T 5f0230cae1d54700be0f8714343a472b P 1204c82bd36a468fbdf532c7aad13159: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260809 12:13:25.987984 16395 log.cc:1196] T 5f0230cae1d54700be0f8714343a472b P 1204c82bd36a468fbdf532c7aad13159: Deleting WAL directory at /tmp/dist-test-taskgH5Pjm/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786277382073938-888-0/minicluster-data/ts-2/wal/wals/5f0230cae1d54700be0f8714343a472b
I20260809 12:13:25.988718 16395 ts_tablet_manager.cc:1939] T 5f0230cae1d54700be0f8714343a472b P 1204c82bd36a468fbdf532c7aad13159: Deleting consensus metadata
I20260809 12:13:25.990825 15854 catalog_manager.cc:4841] TS 1204c82bd36a468fbdf532c7aad13159 (127.0.222.3:46283): tablet 5f0230cae1d54700be0f8714343a472b (table test-table [id=00d247d39e9345e5accd3e032377c913]) successfully deleted
I20260809 12:13:25.993222 16394 ts_tablet_manager.cc:1918] T 5f0230cae1d54700be0f8714343a472b P 2a033dd02e3d408eadf277165d4c5b95: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260809 12:13:25.993505 16394 log.cc:1196] T 5f0230cae1d54700be0f8714343a472b P 2a033dd02e3d408eadf277165d4c5b95: Deleting WAL directory at /tmp/dist-test-taskgH5Pjm/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786277382073938-888-0/minicluster-data/ts-0/wal/wals/5f0230cae1d54700be0f8714343a472b
I20260809 12:13:25.994200 16394 ts_tablet_manager.cc:1939] T 5f0230cae1d54700be0f8714343a472b P 2a033dd02e3d408eadf277165d4c5b95: Deleting consensus metadata
I20260809 12:13:25.994704 16396 ts_tablet_manager.cc:1918] T 5f0230cae1d54700be0f8714343a472b P 5c1e1fc7cc4b4f59b9ca7298b6929b5c: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260809 12:13:25.994998 16396 log.cc:1196] T 5f0230cae1d54700be0f8714343a472b P 5c1e1fc7cc4b4f59b9ca7298b6929b5c: Deleting WAL directory at /tmp/dist-test-taskgH5Pjm/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786277382073938-888-0/minicluster-data/ts-1/wal/wals/5f0230cae1d54700be0f8714343a472b
I20260809 12:13:25.995626 16396 ts_tablet_manager.cc:1939] T 5f0230cae1d54700be0f8714343a472b P 5c1e1fc7cc4b4f59b9ca7298b6929b5c: Deleting consensus metadata
I20260809 12:13:25.996336 15856 catalog_manager.cc:4841] TS 2a033dd02e3d408eadf277165d4c5b95 (127.0.222.1:40557): tablet 5f0230cae1d54700be0f8714343a472b (table test-table [id=00d247d39e9345e5accd3e032377c913]) successfully deleted
I20260809 12:13:25.996663 16378 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 12:13:25.965746 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58498 (local address 127.0.222.1:40557)
0809 12:13:25.965978 (+ 232us) server_negotiation.cc:206] Beginning negotiation
0809 12:13:25.965987 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0809 12:13:25.975371 (+ 9384us) server_negotiation.cc:407] Connection header received
0809 12:13:25.975624 (+ 253us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 12:13:25.975634 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 12:13:25.975766 (+ 132us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 12:13:25.975912 (+ 146us) server_negotiation.cc:226] Negotiated authn=SASL
0809 12:13:25.978406 (+ 2494us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 12:13:25.979519 (+ 1113us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 12:13:25.981695 (+ 2176us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 12:13:25.982322 (+ 627us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 12:13:25.982421 (+ 99us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 12:13:25.988556 (+ 6135us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 12:13:25.988571 (+ 15us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 12:13:25.988592 (+ 21us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0809 12:13:25.988639 (+ 47us) server_negotiation.cc:922] calling sasl_server_start()
0809 12:13:25.992742 (+ 4103us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 12:13:25.993771 (+ 1029us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 12:13:25.993784 (+ 13us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 12:13:25.993788 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0809 12:13:25.993960 (+ 172us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 12:13:25.994485 (+ 525us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 12:13:25.994490 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 12:13:25.994493 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0809 12:13:25.995016 (+ 523us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 12:13:25.995354 (+ 338us) server_negotiation.cc:1035] Waiting for connection context
0809 12:13:25.996059 (+ 705us) server_negotiation.cc:299] Negotiation successful
0809 12:13:25.996330 (+ 271us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":45}
I20260809 12:13:25.997506 15854 catalog_manager.cc:4841] TS 5c1e1fc7cc4b4f59b9ca7298b6929b5c (127.0.222.2:41313): tablet 5f0230cae1d54700be0f8714343a472b (table test-table [id=00d247d39e9345e5accd3e032377c913]) successfully deleted
I20260809 12:13:25.998271 16064 generic_service.cc:196] Checking for leaks (request via RPC)
I20260809 12:13:26.091256 888 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskgH5Pjm/build/asan/bin/kudu with pid 15934
I20260809 12:13:26.115219 888 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskgH5Pjm/build/asan/bin/kudu pid 16075
Aug 09 12:13:26 dist-test-slave-2w3w krb5kdc[15823](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786277603, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.222.2@KRBTEST.COM
I20260809 12:13:26.136526 16367 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 12:13:26.116133 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52944 (local address 127.0.222.2:41313)
0809 12:13:26.116347 (+ 214us) server_negotiation.cc:206] Beginning negotiation
0809 12:13:26.116353 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0809 12:13:26.116373 (+ 20us) server_negotiation.cc:407] Connection header received
0809 12:13:26.116527 (+ 154us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 12:13:26.116532 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 12:13:26.116624 (+ 92us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 12:13:26.116731 (+ 107us) server_negotiation.cc:226] Negotiated authn=SASL
0809 12:13:26.117994 (+ 1263us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 12:13:26.118921 (+ 927us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 12:13:26.120318 (+ 1397us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 12:13:26.120744 (+ 426us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 12:13:26.120888 (+ 144us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 12:13:26.130211 (+ 9323us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 12:13:26.130227 (+ 16us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 12:13:26.130265 (+ 38us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0809 12:13:26.130329 (+ 64us) server_negotiation.cc:922] calling sasl_server_start()
0809 12:13:26.133491 (+ 3162us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 12:13:26.134233 (+ 742us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 12:13:26.134250 (+ 17us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 12:13:26.134252 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0809 12:13:26.134390 (+ 138us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 12:13:26.134808 (+ 418us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 12:13:26.134813 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 12:13:26.134817 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0809 12:13:26.135306 (+ 489us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 12:13:26.135615 (+ 309us) server_negotiation.cc:1035] Waiting for connection context
0809 12:13:26.135949 (+ 334us) server_negotiation.cc:299] Negotiation successful
0809 12:13:26.136175 (+ 226us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":77}
I20260809 12:13:26.137311 16202 generic_service.cc:196] Checking for leaks (request via RPC)
I20260809 12:13:26.230613 888 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskgH5Pjm/build/asan/bin/kudu with pid 16075
I20260809 12:13:26.251596 888 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskgH5Pjm/build/asan/bin/kudu pid 16213
Aug 09 12:13:26 dist-test-slave-2w3w krb5kdc[15823](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786277603, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.222.3@KRBTEST.COM
I20260809 12:13:26.270327 16365 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 12:13:26.252565 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53288 (local address 127.0.222.3:46283)
0809 12:13:26.252745 (+ 180us) server_negotiation.cc:206] Beginning negotiation
0809 12:13:26.252749 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0809 12:13:26.252767 (+ 18us) server_negotiation.cc:407] Connection header received
0809 12:13:26.252950 (+ 183us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 12:13:26.252954 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 12:13:26.253045 (+ 91us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 12:13:26.253139 (+ 94us) server_negotiation.cc:226] Negotiated authn=SASL
0809 12:13:26.254947 (+ 1808us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 12:13:26.255747 (+ 800us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 12:13:26.257517 (+ 1770us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 12:13:26.257918 (+ 401us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 12:13:26.258015 (+ 97us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 12:13:26.264041 (+ 6026us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 12:13:26.264056 (+ 15us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 12:13:26.264071 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0809 12:13:26.264122 (+ 51us) server_negotiation.cc:922] calling sasl_server_start()
0809 12:13:26.267368 (+ 3246us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 12:13:26.268109 (+ 741us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 12:13:26.268117 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 12:13:26.268119 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0809 12:13:26.268245 (+ 126us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 12:13:26.268641 (+ 396us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 12:13:26.268648 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 12:13:26.268656 (+ 8us) server_negotiation.cc:967] calling sasl_server_step()
0809 12:13:26.269048 (+ 392us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 12:13:26.269373 (+ 325us) server_negotiation.cc:1035] Waiting for connection context
0809 12:13:26.269733 (+ 360us) server_negotiation.cc:299] Negotiation successful
0809 12:13:26.269995 (+ 262us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":58}
I20260809 12:13:26.271132 16343 generic_service.cc:196] Checking for leaks (request via RPC)
I20260809 12:13:26.397039 888 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskgH5Pjm/build/asan/bin/kudu with pid 16213
I20260809 12:13:26.421209 888 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskgH5Pjm/build/asan/bin/kudu pid 15837
I20260809 12:13:26.422508 15901 generic_service.cc:196] Checking for leaks (request via RPC)
I20260809 12:13:26.541213 888 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskgH5Pjm/build/asan/bin/kudu with pid 15837
I20260809 12:13:26.577936 888 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskgH5Pjm/build/asan/bin/kudu
/tmp/dist-test-taskgH5Pjm/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskgH5Pjm/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786277382073938-888-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskgH5Pjm/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786277382073938-888-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskgH5Pjm/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786277382073938-888-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskgH5Pjm/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786277382073938-888-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.222.62:44147
--webserver_interface=127.0.222.62
--webserver_port=42133
--keytab_file=/tmp/dist-test-taskgH5Pjm/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786277382073938-888-0/krb5kdc/kudu_127.0.222.62.keytab
--principal=kudu/127.0.222.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.222.20:41797
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.222.62:44147
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--ipki_private_key_password_cmd=echo foo with env {KRB5CCNAME=/tmp/dist-test-taskgH5Pjm/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786277382073938-888-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskgH5Pjm/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786277382073938-888-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskgH5Pjm/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786277382073938-888-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260809 12:13:26.931146 16402 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260809 12:13:26.931542 16402 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260809 12:13:26.931725 16402 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260809 12:13:26.940919 16402 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260809 12:13:26.941084 16402 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260809 12:13:26.941174 16402 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260809 12:13:26.941259 16402 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260809 12:13:26.941341 16402 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260809 12:13:26.952172 16402 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.222.20:41797
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskgH5Pjm/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786277382073938-888-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskgH5Pjm/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786277382073938-888-0/minicluster-data/master-0/wal
--ipki_private_key_password_cmd=echo foo
--ipki_ca_key_size=768
--master_addresses=127.0.222.62:44147
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskgH5Pjm/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786277382073938-888-0/krb5kdc/kudu_127.0.222.62.keytab
--principal=kudu/127.0.222.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.222.62:44147
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskgH5Pjm/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786277382073938-888-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.222.62
--webserver_port=42133
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskgH5Pjm/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786277382073938-888-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 09 Aug 2026 11:06:41 UTC on ccycloud.quasar-bwoogy.root.comops.site
build id 4575991
ASAN enabled
I20260809 12:13:26.953585 16402 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260809 12:13:26.955034 16402 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260809 12:13:26.983793 16409 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 12:13:26.983798 16411 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260809 12:13:26.984743 16402 server_base.cc:1034] running on GCE node
W20260809 12:13:26.985203 16408 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260809 12:13:26.986349 16402 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260809 12:13:26.988176 16402 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260809 12:13:26.989404 16402 hybrid_clock.cc:648] HybridClock initialized: now 1786277606989355 us; error 38 us; skew 500 ppm
I20260809 12:13:26.989873 16402 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 09 12:13:26 dist-test-slave-2w3w krb5kdc[15823](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786277606, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.222.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260809 12:13:26.998728 16402 init.cc:377] Logged in from keytab as kudu/127.0.222.62@KRBTEST.COM (short username kudu)
I20260809 12:13:27.000797 16402 webserver.cc:460] Webserver started at http://127.0.222.62:42133/ using document root <none> and password file <none>
I20260809 12:13:27.001669 16402 fs_manager.cc:359] Metadata directory not provided
I20260809 12:13:27.001869 16402 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260809 12:13:27.008594 16402 fs_manager.cc:709] Time spent opening directory manager: real 0.004s user 0.000s sys 0.004s
I20260809 12:13:27.012111 16419 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260809 12:13:27.013638 16402 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.003s
I20260809 12:13:27.013947 16402 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskgH5Pjm/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786277382073938-888-0/minicluster-data/master-0/data,/tmp/dist-test-taskgH5Pjm/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786277382073938-888-0/minicluster-data/master-0/wal
uuid: "6d7f8671e34d46459506bc2cba79c82c"
format_stamp: "Formatted at 2026-08-09 12:13:23 on dist-test-slave-2w3w"
server_key: "5cd736e973b6ffe5de54c3107192d76b"
server_key_iv: "493043018edc30ecf9a63340ec4f2759"
server_key_version: "encryptionkey@0"
I20260809 12:13:27.015225 16402 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskgH5Pjm/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786277382073938-888-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskgH5Pjm/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786277382073938-888-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskgH5Pjm/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786277382073938-888-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260809 12:13:27.033769 16402 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260809 12:13:27.038949 16402 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260809 12:13:27.039345 16402 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260809 12:13:27.061339 16402 rpc_server.cc:307] RPC server started. Bound to: 127.0.222.62:44147
I20260809 12:13:27.061362 16475 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.222.62:44147 every 8 connection(s)
I20260809 12:13:27.063067 16402 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskgH5Pjm/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786277382073938-888-0/minicluster-data/master-0/data/info.pb
I20260809 12:13:27.068516 888 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskgH5Pjm/build/asan/bin/kudu as pid 16402
I20260809 12:13:27.069561 888 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskgH5Pjm/build/asan/bin/kudu
/tmp/dist-test-taskgH5Pjm/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskgH5Pjm/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786277382073938-888-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskgH5Pjm/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786277382073938-888-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskgH5Pjm/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786277382073938-888-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskgH5Pjm/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786277382073938-888-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.222.1:40557
--local_ip_for_outbound_sockets=127.0.222.1
--tserver_master_addrs=127.0.222.62:44147
--webserver_port=44545
--webserver_interface=127.0.222.1
--keytab_file=/tmp/dist-test-taskgH5Pjm/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786277382073938-888-0/krb5kdc/kudu_127.0.222.1.keytab
--principal=kudu/127.0.222.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.222.20:41797
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskgH5Pjm/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786277382073938-888-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskgH5Pjm/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786277382073938-888-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskgH5Pjm/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786277382073938-888-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260809 12:13:27.071394 16476 sys_catalog.cc:263] Verifying existing consensus state
I20260809 12:13:27.076879 16476 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 6d7f8671e34d46459506bc2cba79c82c: Bootstrap starting.
I20260809 12:13:27.098922 16476 log.cc:824] T 00000000000000000000000000000000 P 6d7f8671e34d46459506bc2cba79c82c: Log is configured to *not* fsync() on all Append() calls
I20260809 12:13:27.111882 16476 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 6d7f8671e34d46459506bc2cba79c82c: Bootstrap replayed 1/1 log segments. Stats: ops{read=8 overwritten=0 applied=8 ignored=0} inserts{seen=5 ignored=0} mutations{seen=4 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260809 12:13:27.112581 16476 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 6d7f8671e34d46459506bc2cba79c82c: Bootstrap complete.
I20260809 12:13:27.122643 16476 raft_consensus.cc:348] T 00000000000000000000000000000000 P 6d7f8671e34d46459506bc2cba79c82c [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6d7f8671e34d46459506bc2cba79c82c" member_type: VOTER last_known_addr { host: "127.0.222.62" port: 44147 } }
I20260809 12:13:27.123613 16476 raft_consensus.cc:729] T 00000000000000000000000000000000 P 6d7f8671e34d46459506bc2cba79c82c [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6d7f8671e34d46459506bc2cba79c82c, State: Initialized, Role: FOLLOWER
I20260809 12:13:27.124246 16476 consensus_queue.cc:260] T 00000000000000000000000000000000 P 6d7f8671e34d46459506bc2cba79c82c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 8, Last appended: 1.8, Last appended by leader: 8, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6d7f8671e34d46459506bc2cba79c82c" member_type: VOTER last_known_addr { host: "127.0.222.62" port: 44147 } }
I20260809 12:13:27.124504 16476 raft_consensus.cc:388] T 00000000000000000000000000000000 P 6d7f8671e34d46459506bc2cba79c82c [term 1 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260809 12:13:27.124655 16476 raft_consensus.cc:482] T 00000000000000000000000000000000 P 6d7f8671e34d46459506bc2cba79c82c [term 1 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260809 12:13:27.124845 16476 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 6d7f8671e34d46459506bc2cba79c82c [term 1 FOLLOWER]: Advancing to term 2
I20260809 12:13:27.129093 16476 raft_consensus.cc:504] T 00000000000000000000000000000000 P 6d7f8671e34d46459506bc2cba79c82c [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6d7f8671e34d46459506bc2cba79c82c" member_type: VOTER last_known_addr { host: "127.0.222.62" port: 44147 } }
I20260809 12:13:27.129678 16476 leader_election.cc:302] T 00000000000000000000000000000000 P 6d7f8671e34d46459506bc2cba79c82c [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 6d7f8671e34d46459506bc2cba79c82c; no voters:
I20260809 12:13:27.130352 16476 leader_election.cc:288] T 00000000000000000000000000000000 P 6d7f8671e34d46459506bc2cba79c82c [CANDIDATE]: Term 2 election: Requested vote from peers
I20260809 12:13:27.130507 16480 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 6d7f8671e34d46459506bc2cba79c82c [term 2 FOLLOWER]: Leader election won for term 2
I20260809 12:13:27.131841 16480 raft_consensus.cc:686] T 00000000000000000000000000000000 P 6d7f8671e34d46459506bc2cba79c82c [term 2 LEADER]: Becoming Leader. State: Replica: 6d7f8671e34d46459506bc2cba79c82c, State: Running, Role: LEADER
I20260809 12:13:27.132522 16480 consensus_queue.cc:237] T 00000000000000000000000000000000 P 6d7f8671e34d46459506bc2cba79c82c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 1.8, Last appended by leader: 8, Current term: 2, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6d7f8671e34d46459506bc2cba79c82c" member_type: VOTER last_known_addr { host: "127.0.222.62" port: 44147 } }
I20260809 12:13:27.132890 16476 sys_catalog.cc:564] T 00000000000000000000000000000000 P 6d7f8671e34d46459506bc2cba79c82c [sys.catalog]: configured and running, proceeding with master startup.
I20260809 12:13:27.135449 16481 sys_catalog.cc:455] T 00000000000000000000000000000000 P 6d7f8671e34d46459506bc2cba79c82c [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 2 leader_uuid: "6d7f8671e34d46459506bc2cba79c82c" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6d7f8671e34d46459506bc2cba79c82c" member_type: VOTER last_known_addr { host: "127.0.222.62" port: 44147 } } }
I20260809 12:13:27.135919 16481 sys_catalog.cc:458] T 00000000000000000000000000000000 P 6d7f8671e34d46459506bc2cba79c82c [sys.catalog]: This master's current role is: LEADER
I20260809 12:13:27.138324 16482 sys_catalog.cc:455] T 00000000000000000000000000000000 P 6d7f8671e34d46459506bc2cba79c82c [sys.catalog]: SysCatalogTable state changed. Reason: New leader 6d7f8671e34d46459506bc2cba79c82c. Latest consensus state: current_term: 2 leader_uuid: "6d7f8671e34d46459506bc2cba79c82c" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6d7f8671e34d46459506bc2cba79c82c" member_type: VOTER last_known_addr { host: "127.0.222.62" port: 44147 } } }
I20260809 12:13:27.138753 16482 sys_catalog.cc:458] T 00000000000000000000000000000000 P 6d7f8671e34d46459506bc2cba79c82c [sys.catalog]: This master's current role is: LEADER
I20260809 12:13:27.140503 16486 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260809 12:13:27.149364 16486 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260809 12:13:27.151533 16486 catalog_manager.cc:1259] Loaded cluster ID: 71a354fc623b4b8f864d7005c8295487
I20260809 12:13:27.151691 16486 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260809 12:13:27.155205 16486 catalog_manager.cc:1314] IPKI private key was stored in cleartext, attempting to encrypt it.
I20260809 12:13:27.173111 16486 catalog_manager.cc:1324] IPKI private key successfully encrypted.
I20260809 12:13:27.182893 16486 catalog_manager.cc:1504] Loading token signing keys...
I20260809 12:13:27.184819 16486 catalog_manager.cc:5879] T 00000000000000000000000000000000 P 6d7f8671e34d46459506bc2cba79c82c: Loaded TSK: 0
I20260809 12:13:27.186385 16486 catalog_manager.cc:1514] Initializing in-progress tserver states...
W20260809 12:13:27.436748 16478 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260809 12:13:27.437101 16478 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260809 12:13:27.437273 16478 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260809 12:13:27.445335 16478 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260809 12:13:27.445490 16478 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260809 12:13:27.445683 16478 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.222.1
I20260809 12:13:27.455911 16478 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.222.20:41797
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskgH5Pjm/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786277382073938-888-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskgH5Pjm/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786277382073938-888-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskgH5Pjm/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786277382073938-888-0/krb5kdc/kudu_127.0.222.1.keytab
--principal=kudu/127.0.222.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.222.1:40557
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskgH5Pjm/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786277382073938-888-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.222.1
--webserver_port=44545
--tserver_master_addrs=127.0.222.62:44147
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.222.1
--log_dir=/tmp/dist-test-taskgH5Pjm/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786277382073938-888-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 09 Aug 2026 11:06:41 UTC on ccycloud.quasar-bwoogy.root.comops.site
build id 4575991
ASAN enabled
I20260809 12:13:27.457296 16478 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260809 12:13:27.458662 16478 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260809 12:13:27.474088 16503 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 12:13:27.474038 16504 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 12:13:27.474905 16506 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260809 12:13:27.475514 16478 server_base.cc:1034] running on GCE node
I20260809 12:13:27.476154 16478 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260809 12:13:27.477239 16478 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260809 12:13:27.478469 16478 hybrid_clock.cc:648] HybridClock initialized: now 1786277607478408 us; error 61 us; skew 500 ppm
I20260809 12:13:27.478801 16478 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 09 12:13:27 dist-test-slave-2w3w krb5kdc[15823](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786277607, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.222.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260809 12:13:27.482404 16478 init.cc:377] Logged in from keytab as kudu/127.0.222.1@KRBTEST.COM (short username kudu)
I20260809 12:13:27.484071 16478 webserver.cc:460] Webserver started at http://127.0.222.1:44545/ using document root <none> and password file <none>
I20260809 12:13:27.484779 16478 fs_manager.cc:359] Metadata directory not provided
I20260809 12:13:27.484948 16478 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260809 12:13:27.490736 16478 fs_manager.cc:709] Time spent opening directory manager: real 0.004s user 0.003s sys 0.002s
I20260809 12:13:27.493793 16514 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260809 12:13:27.495538 16478 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260809 12:13:27.495723 16478 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskgH5Pjm/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786277382073938-888-0/minicluster-data/ts-0/data,/tmp/dist-test-taskgH5Pjm/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786277382073938-888-0/minicluster-data/ts-0/wal
uuid: "2a033dd02e3d408eadf277165d4c5b95"
format_stamp: "Formatted at 2026-08-09 12:13:24 on dist-test-slave-2w3w"
server_key: "cbad234163032fa77b32d4e26b9f9057"
server_key_iv: "bbf5477b04df91cc882d7c584ab72de6"
server_key_version: "encryptionkey@0"
I20260809 12:13:27.496773 16478 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskgH5Pjm/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786277382073938-888-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskgH5Pjm/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786277382073938-888-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskgH5Pjm/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786277382073938-888-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260809 12:13:27.526783 16478 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260809 12:13:27.531663 16478 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260809 12:13:27.531985 16478 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260809 12:13:27.533151 16478 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260809 12:13:27.534644 16478 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260809 12:13:27.534794 16478 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260809 12:13:27.534925 16478 ts_tablet_manager.cc:610] Registered 0 tablets
I20260809 12:13:27.535036 16478 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.001s
I20260809 12:13:27.569886 16478 rpc_server.cc:307] RPC server started. Bound to: 127.0.222.1:40557
I20260809 12:13:27.569976 16644 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.222.1:40557 every 8 connection(s)
I20260809 12:13:27.571449 16478 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskgH5Pjm/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786277382073938-888-0/minicluster-data/ts-0/data/info.pb
I20260809 12:13:27.582058 888 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskgH5Pjm/build/asan/bin/kudu as pid 16478
I20260809 12:13:27.582823 888 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskgH5Pjm/build/asan/bin/kudu
/tmp/dist-test-taskgH5Pjm/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskgH5Pjm/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786277382073938-888-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskgH5Pjm/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786277382073938-888-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskgH5Pjm/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786277382073938-888-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskgH5Pjm/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786277382073938-888-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.222.2:41313
--local_ip_for_outbound_sockets=127.0.222.2
--tserver_master_addrs=127.0.222.62:44147
--webserver_port=46335
--webserver_interface=127.0.222.2
--keytab_file=/tmp/dist-test-taskgH5Pjm/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786277382073938-888-0/krb5kdc/kudu_127.0.222.2.keytab
--principal=kudu/127.0.222.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.222.20:41797
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskgH5Pjm/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786277382073938-888-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskgH5Pjm/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786277382073938-888-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskgH5Pjm/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786277382073938-888-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Aug 09 12:13:27 dist-test-slave-2w3w krb5kdc[15823](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786277607, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.222.1@KRBTEST.COM for kudu/127.0.222.62@KRBTEST.COM
I20260809 12:13:27.597455 16648 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 12:13:27.574555 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.222.62:44147 (local address 127.0.222.1:55623)
0809 12:13:27.575422 (+ 867us) negotiation.cc:107] Waiting for socket to connect
0809 12:13:27.575467 (+ 45us) client_negotiation.cc:174] Beginning negotiation
0809 12:13:27.577181 (+ 1714us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 12:13:27.578433 (+ 1252us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 12:13:27.578440 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 12:13:27.578940 (+ 500us) client_negotiation.cc:189] Negotiated authn=SASL
0809 12:13:27.579535 (+ 595us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 12:13:27.579553 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 12:13:27.581177 (+ 1624us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 12:13:27.581181 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 12:13:27.583335 (+ 2154us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 12:13:27.583355 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 12:13:27.584436 (+ 1081us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 12:13:27.584446 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 12:13:27.584785 (+ 339us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 12:13:27.585588 (+ 803us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0809 12:13:27.585623 (+ 35us) client_negotiation.cc:656] calling sasl_client_start()
0809 12:13:27.588825 (+ 3202us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0809 12:13:27.592550 (+ 3725us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0809 12:13:27.592563 (+ 13us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0809 12:13:27.592582 (+ 19us) client_negotiation.cc:760] calling sasl_client_step()
0809 12:13:27.593302 (+ 720us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0809 12:13:27.593858 (+ 556us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0809 12:13:27.593867 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0809 12:13:27.593873 (+ 6us) client_negotiation.cc:760] calling sasl_client_step()
0809 12:13:27.594157 (+ 284us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0809 12:13:27.595406 (+ 1249us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0809 12:13:27.595413 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0809 12:13:27.595737 (+ 324us) client_negotiation.cc:769] Sending connection context
0809 12:13:27.595973 (+ 236us) client_negotiation.cc:240] Negotiation successful
0809 12:13:27.596398 (+ 425us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":596,"thread_start_us":401,"threads_started":1}
I20260809 12:13:27.598008 16647 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 12:13:27.574362 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.222.1:55623 (local address 127.0.222.62:44147)
0809 12:13:27.575068 (+ 706us) server_negotiation.cc:206] Beginning negotiation
0809 12:13:27.575079 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0809 12:13:27.575590 (+ 511us) server_negotiation.cc:407] Connection header received
0809 12:13:27.577626 (+ 2036us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 12:13:27.577638 (+ 12us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 12:13:27.577940 (+ 302us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 12:13:27.578240 (+ 300us) server_negotiation.cc:226] Negotiated authn=SASL
0809 12:13:27.579763 (+ 1523us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 12:13:27.581024 (+ 1261us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 12:13:27.583495 (+ 2471us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 12:13:27.584174 (+ 679us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 12:13:27.584326 (+ 152us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 12:13:27.589177 (+ 4851us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 12:13:27.589185 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 12:13:27.589201 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0809 12:13:27.589250 (+ 49us) server_negotiation.cc:922] calling sasl_server_start()
0809 12:13:27.592361 (+ 3111us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 12:13:27.593505 (+ 1144us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 12:13:27.593513 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 12:13:27.593519 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0809 12:13:27.593705 (+ 186us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 12:13:27.594321 (+ 616us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 12:13:27.594326 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 12:13:27.594328 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0809 12:13:27.595251 (+ 923us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 12:13:27.595488 (+ 237us) server_negotiation.cc:1035] Waiting for connection context
0809 12:13:27.596264 (+ 776us) server_negotiation.cc:299] Negotiation successful
0809 12:13:27.596760 (+ 496us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":446,"thread_start_us":294,"threads_started":1}
I20260809 12:13:27.601189 16645 heartbeater.cc:347] Connected to a master server at 127.0.222.62:44147
I20260809 12:13:27.601562 16645 heartbeater.cc:464] Registering TS with master...
I20260809 12:13:27.602387 16645 heartbeater.cc:511] Master 127.0.222.62:44147 requested a full tablet report, sending...
I20260809 12:13:27.604763 16437 ts_manager.cc:194] Registered new tserver with Master: 2a033dd02e3d408eadf277165d4c5b95 (127.0.222.1:40557)
I20260809 12:13:27.606258 16437 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.222.1@KRBTEST.COM'} at 127.0.222.1:55623
W20260809 12:13:27.937373 16649 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260809 12:13:27.937671 16649 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260809 12:13:27.937858 16649 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260809 12:13:27.946784 16649 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260809 12:13:27.946900 16649 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260809 12:13:27.947131 16649 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.222.2
I20260809 12:13:27.957370 16649 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.222.20:41797
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskgH5Pjm/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786277382073938-888-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskgH5Pjm/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786277382073938-888-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskgH5Pjm/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786277382073938-888-0/krb5kdc/kudu_127.0.222.2.keytab
--principal=kudu/127.0.222.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.222.2:41313
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskgH5Pjm/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786277382073938-888-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.222.2
--webserver_port=46335
--tserver_master_addrs=127.0.222.62:44147
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.222.2
--log_dir=/tmp/dist-test-taskgH5Pjm/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786277382073938-888-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 09 Aug 2026 11:06:41 UTC on ccycloud.quasar-bwoogy.root.comops.site
build id 4575991
ASAN enabled
I20260809 12:13:27.958730 16649 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260809 12:13:27.960122 16649 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260809 12:13:27.974078 16655 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 12:13:27.974043 16657 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 12:13:27.974123 16654 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260809 12:13:27.976514 16649 server_base.cc:1034] running on GCE node
I20260809 12:13:27.977219 16649 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260809 12:13:27.978233 16649 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260809 12:13:27.979440 16649 hybrid_clock.cc:648] HybridClock initialized: now 1786277607979382 us; error 46 us; skew 500 ppm
I20260809 12:13:27.979738 16649 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 09 12:13:27 dist-test-slave-2w3w krb5kdc[15823](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786277607, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.222.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260809 12:13:27.983107 16649 init.cc:377] Logged in from keytab as kudu/127.0.222.2@KRBTEST.COM (short username kudu)
I20260809 12:13:27.984635 16649 webserver.cc:460] Webserver started at http://127.0.222.2:46335/ using document root <none> and password file <none>
I20260809 12:13:27.985421 16649 fs_manager.cc:359] Metadata directory not provided
I20260809 12:13:27.985572 16649 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260809 12:13:27.991266 16649 fs_manager.cc:709] Time spent opening directory manager: real 0.004s user 0.003s sys 0.000s
I20260809 12:13:27.994119 16665 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260809 12:13:27.995724 16649 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260809 12:13:27.995916 16649 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskgH5Pjm/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786277382073938-888-0/minicluster-data/ts-1/data,/tmp/dist-test-taskgH5Pjm/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786277382073938-888-0/minicluster-data/ts-1/wal
uuid: "5c1e1fc7cc4b4f59b9ca7298b6929b5c"
format_stamp: "Formatted at 2026-08-09 12:13:24 on dist-test-slave-2w3w"
server_key: "659ef2a30fb36e2536ff973fe9e8e5b4"
server_key_iv: "ca75656d9f5532d4adc03ec74619056d"
server_key_version: "encryptionkey@0"
I20260809 12:13:27.996932 16649 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskgH5Pjm/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786277382073938-888-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskgH5Pjm/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786277382073938-888-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskgH5Pjm/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786277382073938-888-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260809 12:13:28.010757 16649 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260809 12:13:28.014870 16649 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260809 12:13:28.015201 16649 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260809 12:13:28.016275 16649 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260809 12:13:28.017620 16649 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260809 12:13:28.017740 16649 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260809 12:13:28.017885 16649 ts_tablet_manager.cc:610] Registered 0 tablets
I20260809 12:13:28.017966 16649 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260809 12:13:28.052525 16649 rpc_server.cc:307] RPC server started. Bound to: 127.0.222.2:41313
I20260809 12:13:28.052582 16786 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.222.2:41313 every 8 connection(s)
I20260809 12:13:28.054019 16649 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskgH5Pjm/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786277382073938-888-0/minicluster-data/ts-1/data/info.pb
I20260809 12:13:28.057132 888 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskgH5Pjm/build/asan/bin/kudu as pid 16649
I20260809 12:13:28.057780 888 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskgH5Pjm/build/asan/bin/kudu
/tmp/dist-test-taskgH5Pjm/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskgH5Pjm/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786277382073938-888-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskgH5Pjm/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786277382073938-888-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskgH5Pjm/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786277382073938-888-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskgH5Pjm/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786277382073938-888-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.222.3:46283
--local_ip_for_outbound_sockets=127.0.222.3
--tserver_master_addrs=127.0.222.62:44147
--webserver_port=36801
--webserver_interface=127.0.222.3
--keytab_file=/tmp/dist-test-taskgH5Pjm/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786277382073938-888-0/krb5kdc/kudu_127.0.222.3.keytab
--principal=kudu/127.0.222.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.222.20:41797
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskgH5Pjm/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786277382073938-888-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskgH5Pjm/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786277382073938-888-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskgH5Pjm/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786277382073938-888-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Aug 09 12:13:28 dist-test-slave-2w3w krb5kdc[15823](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786277607, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.222.2@KRBTEST.COM for kudu/127.0.222.62@KRBTEST.COM
I20260809 12:13:28.086442 16647 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 12:13:28.056704 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.222.2:53749 (local address 127.0.222.62:44147)
0809 12:13:28.056941 (+ 237us) server_negotiation.cc:206] Beginning negotiation
0809 12:13:28.056947 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0809 12:13:28.058287 (+ 1340us) server_negotiation.cc:407] Connection header received
0809 12:13:28.060253 (+ 1966us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 12:13:28.060263 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 12:13:28.060364 (+ 101us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 12:13:28.060492 (+ 128us) server_negotiation.cc:226] Negotiated authn=SASL
0809 12:13:28.069448 (+ 8956us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 12:13:28.070503 (+ 1055us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 12:13:28.072947 (+ 2444us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 12:13:28.073830 (+ 883us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 12:13:28.073981 (+ 151us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 12:13:28.078782 (+ 4801us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 12:13:28.078788 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 12:13:28.078793 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0809 12:13:28.078834 (+ 41us) server_negotiation.cc:922] calling sasl_server_start()
0809 12:13:28.082525 (+ 3691us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 12:13:28.083681 (+ 1156us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 12:13:28.083687 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 12:13:28.083690 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0809 12:13:28.083859 (+ 169us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 12:13:28.084453 (+ 594us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 12:13:28.084457 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 12:13:28.084459 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0809 12:13:28.084823 (+ 364us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 12:13:28.084940 (+ 117us) server_negotiation.cc:1035] Waiting for connection context
0809 12:13:28.086010 (+ 1070us) server_negotiation.cc:299] Negotiation successful
0809 12:13:28.086196 (+ 186us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":86}
I20260809 12:13:28.088212 16789 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 12:13:28.057194 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.222.62:44147 (local address 127.0.222.2:53749)
0809 12:13:28.058074 (+ 880us) negotiation.cc:107] Waiting for socket to connect
0809 12:13:28.058130 (+ 56us) client_negotiation.cc:174] Beginning negotiation
0809 12:13:28.059956 (+ 1826us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 12:13:28.060780 (+ 824us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 12:13:28.060792 (+ 12us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 12:13:28.061584 (+ 792us) client_negotiation.cc:189] Negotiated authn=SASL
0809 12:13:28.069192 (+ 7608us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 12:13:28.069220 (+ 28us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 12:13:28.070696 (+ 1476us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 12:13:28.070702 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 12:13:28.072765 (+ 2063us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 12:13:28.072782 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 12:13:28.074083 (+ 1301us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 12:13:28.074090 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 12:13:28.074383 (+ 293us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 12:13:28.075264 (+ 881us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0809 12:13:28.075294 (+ 30us) client_negotiation.cc:656] calling sasl_client_start()
0809 12:13:28.078502 (+ 3208us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0809 12:13:28.082697 (+ 4195us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0809 12:13:28.082708 (+ 11us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0809 12:13:28.082730 (+ 22us) client_negotiation.cc:760] calling sasl_client_step()
0809 12:13:28.083511 (+ 781us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0809 12:13:28.083994 (+ 483us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0809 12:13:28.084002 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0809 12:13:28.084009 (+ 7us) client_negotiation.cc:760] calling sasl_client_step()
0809 12:13:28.084299 (+ 290us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0809 12:13:28.084968 (+ 669us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0809 12:13:28.084979 (+ 11us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0809 12:13:28.085507 (+ 528us) client_negotiation.cc:769] Sending connection context
0809 12:13:28.085873 (+ 366us) client_negotiation.cc:240] Negotiation successful
0809 12:13:28.086539 (+ 666us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":569,"thread_start_us":350,"threads_started":1}
I20260809 12:13:28.089900 16787 heartbeater.cc:347] Connected to a master server at 127.0.222.62:44147
I20260809 12:13:28.090237 16787 heartbeater.cc:464] Registering TS with master...
I20260809 12:13:28.091068 16787 heartbeater.cc:511] Master 127.0.222.62:44147 requested a full tablet report, sending...
I20260809 12:13:28.092585 16437 ts_manager.cc:194] Registered new tserver with Master: 5c1e1fc7cc4b4f59b9ca7298b6929b5c (127.0.222.2:41313)
I20260809 12:13:28.093422 16437 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.222.2@KRBTEST.COM'} at 127.0.222.2:53749
W20260809 12:13:28.410672 16790 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260809 12:13:28.411091 16790 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260809 12:13:28.411284 16790 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260809 12:13:28.418916 16790 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260809 12:13:28.419114 16790 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260809 12:13:28.419296 16790 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.222.3
I20260809 12:13:28.429405 16790 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.222.20:41797
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskgH5Pjm/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786277382073938-888-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskgH5Pjm/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786277382073938-888-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskgH5Pjm/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786277382073938-888-0/krb5kdc/kudu_127.0.222.3.keytab
--principal=kudu/127.0.222.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.222.3:46283
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskgH5Pjm/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786277382073938-888-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.222.3
--webserver_port=36801
--tserver_master_addrs=127.0.222.62:44147
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.222.3
--log_dir=/tmp/dist-test-taskgH5Pjm/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786277382073938-888-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 09 Aug 2026 11:06:41 UTC on ccycloud.quasar-bwoogy.root.comops.site
build id 4575991
ASAN enabled
I20260809 12:13:28.430868 16790 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260809 12:13:28.432396 16790 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260809 12:13:28.446177 16795 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 12:13:28.446120 16796 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 12:13:28.446206 16798 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260809 12:13:28.447103 16790 server_base.cc:1034] running on GCE node
I20260809 12:13:28.447667 16790 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260809 12:13:28.448602 16790 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260809 12:13:28.449844 16790 hybrid_clock.cc:648] HybridClock initialized: now 1786277608449776 us; error 54 us; skew 500 ppm
I20260809 12:13:28.450183 16790 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 09 12:13:28 dist-test-slave-2w3w krb5kdc[15823](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786277608, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.222.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260809 12:13:28.453434 16790 init.cc:377] Logged in from keytab as kudu/127.0.222.3@KRBTEST.COM (short username kudu)
I20260809 12:13:28.455045 16790 webserver.cc:460] Webserver started at http://127.0.222.3:36801/ using document root <none> and password file <none>
I20260809 12:13:28.455713 16790 fs_manager.cc:359] Metadata directory not provided
I20260809 12:13:28.455838 16790 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260809 12:13:28.461333 16790 fs_manager.cc:709] Time spent opening directory manager: real 0.004s user 0.000s sys 0.005s
I20260809 12:13:28.464277 16805 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260809 12:13:28.465651 16790 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.001s
I20260809 12:13:28.465842 16790 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskgH5Pjm/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786277382073938-888-0/minicluster-data/ts-2/data,/tmp/dist-test-taskgH5Pjm/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786277382073938-888-0/minicluster-data/ts-2/wal
uuid: "1204c82bd36a468fbdf532c7aad13159"
format_stamp: "Formatted at 2026-08-09 12:13:25 on dist-test-slave-2w3w"
server_key: "69332dd618b10d8ba467b56a6ab50e51"
server_key_iv: "02d88d015dd0bb3411fe617ba0481989"
server_key_version: "encryptionkey@0"
I20260809 12:13:28.466830 16790 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskgH5Pjm/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786277382073938-888-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskgH5Pjm/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786277382073938-888-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskgH5Pjm/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786277382073938-888-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260809 12:13:28.481670 16790 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260809 12:13:28.486274 16790 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260809 12:13:28.486512 16790 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260809 12:13:28.487564 16790 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260809 12:13:28.488925 16790 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260809 12:13:28.489014 16790 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260809 12:13:28.489097 16790 ts_tablet_manager.cc:610] Registered 0 tablets
I20260809 12:13:28.489153 16790 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260809 12:13:28.524168 16790 rpc_server.cc:307] RPC server started. Bound to: 127.0.222.3:46283
I20260809 12:13:28.524261 16923 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.222.3:46283 every 8 connection(s)
I20260809 12:13:28.525672 16790 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskgH5Pjm/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786277382073938-888-0/minicluster-data/ts-2/data/info.pb
I20260809 12:13:28.531081 888 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskgH5Pjm/build/asan/bin/kudu as pid 16790
Aug 09 12:13:28 dist-test-slave-2w3w krb5kdc[15823](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786277608, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.222.3@KRBTEST.COM for kudu/127.0.222.62@KRBTEST.COM
I20260809 12:13:28.549214 16647 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 12:13:28.528315 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.222.3:54137 (local address 127.0.222.62:44147)
0809 12:13:28.528525 (+ 210us) server_negotiation.cc:206] Beginning negotiation
0809 12:13:28.528530 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0809 12:13:28.529680 (+ 1150us) server_negotiation.cc:407] Connection header received
0809 12:13:28.532176 (+ 2496us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 12:13:28.532183 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 12:13:28.532281 (+ 98us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 12:13:28.532424 (+ 143us) server_negotiation.cc:226] Negotiated authn=SASL
0809 12:13:28.534442 (+ 2018us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 12:13:28.535689 (+ 1247us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 12:13:28.537539 (+ 1850us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 12:13:28.537971 (+ 432us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 12:13:28.538130 (+ 159us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 12:13:28.542433 (+ 4303us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 12:13:28.542438 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 12:13:28.542441 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0809 12:13:28.542473 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0809 12:13:28.545765 (+ 3292us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 12:13:28.546560 (+ 795us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 12:13:28.546573 (+ 13us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 12:13:28.546576 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0809 12:13:28.546710 (+ 134us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 12:13:28.547262 (+ 552us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 12:13:28.547267 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 12:13:28.547272 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0809 12:13:28.547547 (+ 275us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 12:13:28.547721 (+ 174us) server_negotiation.cc:1035] Waiting for connection context
0809 12:13:28.548470 (+ 749us) server_negotiation.cc:299] Negotiation successful
0809 12:13:28.548871 (+ 401us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":162,"server-negotiator.queue_time_us":67,"spinlock_wait_cycles":896}
I20260809 12:13:28.549228 16927 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 12:13:28.532134 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45492 (local address 127.0.222.62:44147)
0809 12:13:28.532667 (+ 533us) server_negotiation.cc:206] Beginning negotiation
0809 12:13:28.532672 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0809 12:13:28.535010 (+ 2338us) server_negotiation.cc:407] Connection header received
0809 12:13:28.535172 (+ 162us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 12:13:28.535176 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 12:13:28.535229 (+ 53us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 12:13:28.535310 (+ 81us) server_negotiation.cc:226] Negotiated authn=SASL
0809 12:13:28.536966 (+ 1656us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 12:13:28.537936 (+ 970us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 12:13:28.539467 (+ 1531us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 12:13:28.540131 (+ 664us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 12:13:28.540263 (+ 132us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 12:13:28.543387 (+ 3124us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 12:13:28.543393 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 12:13:28.543396 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0809 12:13:28.543436 (+ 40us) server_negotiation.cc:922] calling sasl_server_start()
0809 12:13:28.546224 (+ 2788us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 12:13:28.546924 (+ 700us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 12:13:28.546930 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 12:13:28.546934 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0809 12:13:28.547130 (+ 196us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 12:13:28.547532 (+ 402us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 12:13:28.547537 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 12:13:28.547540 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0809 12:13:28.547852 (+ 312us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 12:13:28.548105 (+ 253us) server_negotiation.cc:1035] Waiting for connection context
0809 12:13:28.548452 (+ 347us) server_negotiation.cc:299] Negotiation successful
0809 12:13:28.548700 (+ 248us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":410,"thread_start_us":292,"threads_started":1}
I20260809 12:13:28.550344 16926 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 12:13:28.528766 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.222.62:44147 (local address 127.0.222.3:54137)
0809 12:13:28.529509 (+ 743us) negotiation.cc:107] Waiting for socket to connect
0809 12:13:28.529554 (+ 45us) client_negotiation.cc:174] Beginning negotiation
0809 12:13:28.531848 (+ 2294us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 12:13:28.532835 (+ 987us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 12:13:28.532844 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 12:13:28.533616 (+ 772us) client_negotiation.cc:189] Negotiated authn=SASL
0809 12:13:28.534212 (+ 596us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 12:13:28.534230 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 12:13:28.535854 (+ 1624us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 12:13:28.535858 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 12:13:28.537381 (+ 1523us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 12:13:28.537396 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 12:13:28.538217 (+ 821us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 12:13:28.538222 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 12:13:28.538430 (+ 208us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 12:13:28.539181 (+ 751us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0809 12:13:28.539204 (+ 23us) client_negotiation.cc:656] calling sasl_client_start()
0809 12:13:28.542204 (+ 3000us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0809 12:13:28.545924 (+ 3720us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0809 12:13:28.545934 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0809 12:13:28.545953 (+ 19us) client_negotiation.cc:760] calling sasl_client_step()
0809 12:13:28.546420 (+ 467us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0809 12:13:28.546831 (+ 411us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0809 12:13:28.546839 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0809 12:13:28.546844 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0809 12:13:28.547126 (+ 282us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0809 12:13:28.547690 (+ 564us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0809 12:13:28.547698 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0809 12:13:28.548073 (+ 375us) client_negotiation.cc:769] Sending connection context
0809 12:13:28.548346 (+ 273us) client_negotiation.cc:240] Negotiation successful
0809 12:13:28.548913 (+ 567us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":500,"thread_start_us":300,"threads_started":1}
I20260809 12:13:28.551910 16924 heartbeater.cc:347] Connected to a master server at 127.0.222.62:44147
I20260809 12:13:28.552249 16924 heartbeater.cc:464] Registering TS with master...
I20260809 12:13:28.552999 16924 heartbeater.cc:511] Master 127.0.222.62:44147 requested a full tablet report, sending...
I20260809 12:13:28.554469 16437 ts_manager.cc:194] Registered new tserver with Master: 1204c82bd36a468fbdf532c7aad13159 (127.0.222.3:46283)
I20260809 12:13:28.555441 16437 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.222.3@KRBTEST.COM'} at 127.0.222.3:54137
I20260809 12:13:28.564742 888 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260809 12:13:28.573066 16927 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 12:13:28.565857 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45496 (local address 127.0.222.62:44147)
0809 12:13:28.566078 (+ 221us) server_negotiation.cc:206] Beginning negotiation
0809 12:13:28.566084 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0809 12:13:28.567214 (+ 1130us) server_negotiation.cc:407] Connection header received
0809 12:13:28.567415 (+ 201us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 12:13:28.567419 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 12:13:28.567567 (+ 148us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 12:13:28.567697 (+ 130us) server_negotiation.cc:226] Negotiated authn=TOKEN
0809 12:13:28.568110 (+ 413us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 12:13:28.568801 (+ 691us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 12:13:28.570775 (+ 1974us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 12:13:28.571273 (+ 498us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 12:13:28.571436 (+ 163us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 12:13:28.571977 (+ 541us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0809 12:13:28.572465 (+ 488us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0809 12:13:28.572568 (+ 103us) server_negotiation.cc:1035] Waiting for connection context
0809 12:13:28.572735 (+ 167us) server_negotiation.cc:299] Negotiation successful
0809 12:13:28.572805 (+ 70us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":70}
I20260809 12:13:28.574035 16437 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin'} at 127.0.0.1:45496:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260809 12:13:28.581389 16437 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260809 12:13:28.609472 16645 heartbeater.cc:503] Master 127.0.222.62:44147 was elected leader, sending a full tablet report...
I20260809 12:13:28.615809 16936 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 12:13:28.593552 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56626 (local address 127.0.222.3:46283)
0809 12:13:28.606227 (+ 12675us) server_negotiation.cc:206] Beginning negotiation
0809 12:13:28.606236 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0809 12:13:28.606276 (+ 40us) server_negotiation.cc:407] Connection header received
0809 12:13:28.606394 (+ 118us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 12:13:28.606402 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 12:13:28.606701 (+ 299us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 12:13:28.607996 (+ 1295us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0809 12:13:28.608308 (+ 312us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 12:13:28.609772 (+ 1464us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 12:13:28.612954 (+ 3182us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 12:13:28.614504 (+ 1550us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 12:13:28.614655 (+ 151us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 12:13:28.614913 (+ 258us) server_negotiation.cc:1035] Waiting for connection context
0809 12:13:28.615282 (+ 369us) server_negotiation.cc:299] Negotiation successful
0809 12:13:28.615458 (+ 176us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":12477,"thread_start_us":13341,"threads_started":1}
I20260809 12:13:28.615903 16935 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 12:13:28.593303 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.222.3:46283 (local address 127.0.0.1:56626)
0809 12:13:28.594120 (+ 817us) negotiation.cc:107] Waiting for socket to connect
0809 12:13:28.594172 (+ 52us) client_negotiation.cc:174] Beginning negotiation
0809 12:13:28.594336 (+ 164us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 12:13:28.607240 (+ 12904us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 12:13:28.607249 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 12:13:28.607281 (+ 32us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0809 12:13:28.607805 (+ 524us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 12:13:28.607816 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 12:13:28.609982 (+ 2166us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 12:13:28.609987 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 12:13:28.612761 (+ 2774us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 12:13:28.612777 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 12:13:28.614727 (+ 1950us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 12:13:28.614733 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 12:13:28.614940 (+ 207us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 12:13:28.615043 (+ 103us) client_negotiation.cc:769] Sending connection context
0809 12:13:28.615237 (+ 194us) client_negotiation.cc:240] Negotiation successful
0809 12:13:28.615446 (+ 209us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":547,"mutex_wait_us":105,"thread_start_us":361,"threads_started":1}
I20260809 12:13:28.616359 16940 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 12:13:28.594942 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53672 (local address 127.0.222.2:41313)
0809 12:13:28.605521 (+ 10579us) server_negotiation.cc:206] Beginning negotiation
0809 12:13:28.605536 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0809 12:13:28.606538 (+ 1002us) server_negotiation.cc:407] Connection header received
0809 12:13:28.606651 (+ 113us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 12:13:28.606657 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 12:13:28.606899 (+ 242us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 12:13:28.607151 (+ 252us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0809 12:13:28.608166 (+ 1015us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 12:13:28.611094 (+ 2928us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 12:13:28.613141 (+ 2047us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 12:13:28.614504 (+ 1363us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 12:13:28.614647 (+ 143us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 12:13:28.614881 (+ 234us) server_negotiation.cc:1035] Waiting for connection context
0809 12:13:28.615829 (+ 948us) server_negotiation.cc:299] Negotiation successful
0809 12:13:28.616019 (+ 190us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":10374,"thread_start_us":10198,"threads_started":1}
I20260809 12:13:28.616528 16937 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 12:13:28.594241 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51386 (local address 127.0.222.1:40557)
0809 12:13:28.606807 (+ 12566us) server_negotiation.cc:206] Beginning negotiation
0809 12:13:28.606815 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0809 12:13:28.606841 (+ 26us) server_negotiation.cc:407] Connection header received
0809 12:13:28.606931 (+ 90us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 12:13:28.606940 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 12:13:28.607196 (+ 256us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 12:13:28.607393 (+ 197us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0809 12:13:28.608373 (+ 980us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 12:13:28.609637 (+ 1264us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 12:13:28.613307 (+ 3670us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 12:13:28.614620 (+ 1313us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 12:13:28.614803 (+ 183us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 12:13:28.615059 (+ 256us) server_negotiation.cc:1035] Waiting for connection context
0809 12:13:28.616026 (+ 967us) server_negotiation.cc:299] Negotiation successful
0809 12:13:28.616198 (+ 172us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":12396,"thread_start_us":13083,"threads_started":1}
I20260809 12:13:28.617034 16938 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 12:13:28.593830 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.222.1:40557 (local address 127.0.0.1:51386)
0809 12:13:28.605877 (+ 12047us) negotiation.cc:107] Waiting for socket to connect
0809 12:13:28.605904 (+ 27us) client_negotiation.cc:174] Beginning negotiation
0809 12:13:28.606102 (+ 198us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 12:13:28.607676 (+ 1574us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 12:13:28.607680 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 12:13:28.607708 (+ 28us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0809 12:13:28.608200 (+ 492us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 12:13:28.608209 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 12:13:28.609806 (+ 1597us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 12:13:28.609811 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 12:13:28.613118 (+ 3307us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 12:13:28.613132 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 12:13:28.615154 (+ 2022us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 12:13:28.615160 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 12:13:28.615458 (+ 298us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 12:13:28.615628 (+ 170us) client_negotiation.cc:769] Sending connection context
0809 12:13:28.615719 (+ 91us) client_negotiation.cc:240] Negotiation successful
0809 12:13:28.615874 (+ 155us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":11899,"mutex_wait_us":866,"thread_start_us":11514,"threads_started":1}
I20260809 12:13:28.619922 16857 tablet_service.cc:1467] Processing CreateTablet for tablet 72a95f55b65240db982c2c443ebb0862 (DEFAULT_TABLE table=test-table [id=fe0aaad699bf4737978fae44f8b05436]), partition=RANGE (key) PARTITION UNBOUNDED
I20260809 12:13:28.620256 16568 tablet_service.cc:1467] Processing CreateTablet for tablet 72a95f55b65240db982c2c443ebb0862 (DEFAULT_TABLE table=test-table [id=fe0aaad699bf4737978fae44f8b05436]), partition=RANGE (key) PARTITION UNBOUNDED
I20260809 12:13:28.621272 16857 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 72a95f55b65240db982c2c443ebb0862. 1 dirs total, 0 dirs full, 0 dirs failed
I20260809 12:13:28.621513 16568 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 72a95f55b65240db982c2c443ebb0862. 1 dirs total, 0 dirs full, 0 dirs failed
I20260809 12:13:28.627272 16939 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 12:13:28.594500 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.222.2:41313 (local address 127.0.0.1:53672)
0809 12:13:28.606272 (+ 11772us) negotiation.cc:107] Waiting for socket to connect
0809 12:13:28.606294 (+ 22us) client_negotiation.cc:174] Beginning negotiation
0809 12:13:28.606420 (+ 126us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 12:13:28.607432 (+ 1012us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 12:13:28.607442 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 12:13:28.607474 (+ 32us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0809 12:13:28.607912 (+ 438us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 12:13:28.607930 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 12:13:28.611279 (+ 3349us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 12:13:28.611287 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 12:13:28.612918 (+ 1631us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 12:13:28.612935 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 12:13:28.615117 (+ 2182us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 12:13:28.615126 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 12:13:28.615459 (+ 333us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 12:13:28.615627 (+ 168us) client_negotiation.cc:769] Sending connection context
0809 12:13:28.615719 (+ 92us) client_negotiation.cc:240] Negotiation successful
0809 12:13:28.616860 (+ 1141us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":11632,"mutex_wait_us":1104,"thread_start_us":11074,"threads_started":1}
I20260809 12:13:28.631803 16716 tablet_service.cc:1467] Processing CreateTablet for tablet 72a95f55b65240db982c2c443ebb0862 (DEFAULT_TABLE table=test-table [id=fe0aaad699bf4737978fae44f8b05436]), partition=RANGE (key) PARTITION UNBOUNDED
I20260809 12:13:28.634006 16716 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 72a95f55b65240db982c2c443ebb0862. 1 dirs total, 0 dirs full, 0 dirs failed
I20260809 12:13:28.634583 16941 tablet_bootstrap.cc:492] T 72a95f55b65240db982c2c443ebb0862 P 1204c82bd36a468fbdf532c7aad13159: Bootstrap starting.
I20260809 12:13:28.637739 16941 tablet_bootstrap.cc:654] T 72a95f55b65240db982c2c443ebb0862 P 1204c82bd36a468fbdf532c7aad13159: Neither blocks nor log segments found. Creating new log.
I20260809 12:13:28.638682 16942 tablet_bootstrap.cc:492] T 72a95f55b65240db982c2c443ebb0862 P 2a033dd02e3d408eadf277165d4c5b95: Bootstrap starting.
I20260809 12:13:28.640159 16941 log.cc:824] T 72a95f55b65240db982c2c443ebb0862 P 1204c82bd36a468fbdf532c7aad13159: Log is configured to *not* fsync() on all Append() calls
I20260809 12:13:28.643144 16942 tablet_bootstrap.cc:654] T 72a95f55b65240db982c2c443ebb0862 P 2a033dd02e3d408eadf277165d4c5b95: Neither blocks nor log segments found. Creating new log.
I20260809 12:13:28.644734 16942 log.cc:824] T 72a95f55b65240db982c2c443ebb0862 P 2a033dd02e3d408eadf277165d4c5b95: Log is configured to *not* fsync() on all Append() calls
I20260809 12:13:28.647751 16941 tablet_bootstrap.cc:492] T 72a95f55b65240db982c2c443ebb0862 P 1204c82bd36a468fbdf532c7aad13159: No bootstrap required, opened a new log
I20260809 12:13:28.648015 16942 tablet_bootstrap.cc:492] T 72a95f55b65240db982c2c443ebb0862 P 2a033dd02e3d408eadf277165d4c5b95: No bootstrap required, opened a new log
I20260809 12:13:28.648051 16941 ts_tablet_manager.cc:1397] T 72a95f55b65240db982c2c443ebb0862 P 1204c82bd36a468fbdf532c7aad13159: Time spent bootstrapping tablet: real 0.014s user 0.002s sys 0.010s
I20260809 12:13:28.648272 16942 ts_tablet_manager.cc:1397] T 72a95f55b65240db982c2c443ebb0862 P 2a033dd02e3d408eadf277165d4c5b95: Time spent bootstrapping tablet: real 0.010s user 0.004s sys 0.004s
I20260809 12:13:28.650254 16946 tablet_bootstrap.cc:492] T 72a95f55b65240db982c2c443ebb0862 P 5c1e1fc7cc4b4f59b9ca7298b6929b5c: Bootstrap starting.
I20260809 12:13:28.654276 16946 tablet_bootstrap.cc:654] T 72a95f55b65240db982c2c443ebb0862 P 5c1e1fc7cc4b4f59b9ca7298b6929b5c: Neither blocks nor log segments found. Creating new log.
I20260809 12:13:28.655611 16946 log.cc:824] T 72a95f55b65240db982c2c443ebb0862 P 5c1e1fc7cc4b4f59b9ca7298b6929b5c: Log is configured to *not* fsync() on all Append() calls
I20260809 12:13:28.656268 16942 raft_consensus.cc:348] T 72a95f55b65240db982c2c443ebb0862 P 2a033dd02e3d408eadf277165d4c5b95 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1204c82bd36a468fbdf532c7aad13159" member_type: VOTER last_known_addr { host: "127.0.222.3" port: 46283 } } peers { permanent_uuid: "5c1e1fc7cc4b4f59b9ca7298b6929b5c" member_type: VOTER last_known_addr { host: "127.0.222.2" port: 41313 } } peers { permanent_uuid: "2a033dd02e3d408eadf277165d4c5b95" member_type: VOTER last_known_addr { host: "127.0.222.1" port: 40557 } }
I20260809 12:13:28.656797 16942 raft_consensus.cc:374] T 72a95f55b65240db982c2c443ebb0862 P 2a033dd02e3d408eadf277165d4c5b95 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260809 12:13:28.656924 16942 raft_consensus.cc:729] T 72a95f55b65240db982c2c443ebb0862 P 2a033dd02e3d408eadf277165d4c5b95 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2a033dd02e3d408eadf277165d4c5b95, State: Initialized, Role: FOLLOWER
I20260809 12:13:28.657615 16942 consensus_queue.cc:260] T 72a95f55b65240db982c2c443ebb0862 P 2a033dd02e3d408eadf277165d4c5b95 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1204c82bd36a468fbdf532c7aad13159" member_type: VOTER last_known_addr { host: "127.0.222.3" port: 46283 } } peers { permanent_uuid: "5c1e1fc7cc4b4f59b9ca7298b6929b5c" member_type: VOTER last_known_addr { host: "127.0.222.2" port: 41313 } } peers { permanent_uuid: "2a033dd02e3d408eadf277165d4c5b95" member_type: VOTER last_known_addr { host: "127.0.222.1" port: 40557 } }
I20260809 12:13:28.658605 16941 raft_consensus.cc:348] T 72a95f55b65240db982c2c443ebb0862 P 1204c82bd36a468fbdf532c7aad13159 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1204c82bd36a468fbdf532c7aad13159" member_type: VOTER last_known_addr { host: "127.0.222.3" port: 46283 } } peers { permanent_uuid: "5c1e1fc7cc4b4f59b9ca7298b6929b5c" member_type: VOTER last_known_addr { host: "127.0.222.2" port: 41313 } } peers { permanent_uuid: "2a033dd02e3d408eadf277165d4c5b95" member_type: VOTER last_known_addr { host: "127.0.222.1" port: 40557 } }
I20260809 12:13:28.659051 16941 raft_consensus.cc:374] T 72a95f55b65240db982c2c443ebb0862 P 1204c82bd36a468fbdf532c7aad13159 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260809 12:13:28.659207 16941 raft_consensus.cc:729] T 72a95f55b65240db982c2c443ebb0862 P 1204c82bd36a468fbdf532c7aad13159 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1204c82bd36a468fbdf532c7aad13159, State: Initialized, Role: FOLLOWER
I20260809 12:13:28.659363 16942 ts_tablet_manager.cc:1428] T 72a95f55b65240db982c2c443ebb0862 P 2a033dd02e3d408eadf277165d4c5b95: Time spent starting tablet: real 0.011s user 0.010s sys 0.000s
I20260809 12:13:28.659863 16941 consensus_queue.cc:260] T 72a95f55b65240db982c2c443ebb0862 P 1204c82bd36a468fbdf532c7aad13159 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1204c82bd36a468fbdf532c7aad13159" member_type: VOTER last_known_addr { host: "127.0.222.3" port: 46283 } } peers { permanent_uuid: "5c1e1fc7cc4b4f59b9ca7298b6929b5c" member_type: VOTER last_known_addr { host: "127.0.222.2" port: 41313 } } peers { permanent_uuid: "2a033dd02e3d408eadf277165d4c5b95" member_type: VOTER last_known_addr { host: "127.0.222.1" port: 40557 } }
I20260809 12:13:28.660398 16946 tablet_bootstrap.cc:492] T 72a95f55b65240db982c2c443ebb0862 P 5c1e1fc7cc4b4f59b9ca7298b6929b5c: No bootstrap required, opened a new log
I20260809 12:13:28.660681 16946 ts_tablet_manager.cc:1397] T 72a95f55b65240db982c2c443ebb0862 P 5c1e1fc7cc4b4f59b9ca7298b6929b5c: Time spent bootstrapping tablet: real 0.011s user 0.003s sys 0.006s
I20260809 12:13:28.661319 16924 heartbeater.cc:503] Master 127.0.222.62:44147 was elected leader, sending a full tablet report...
I20260809 12:13:28.661618 16941 ts_tablet_manager.cc:1428] T 72a95f55b65240db982c2c443ebb0862 P 1204c82bd36a468fbdf532c7aad13159: Time spent starting tablet: real 0.013s user 0.013s sys 0.000s
I20260809 12:13:28.669533 16946 raft_consensus.cc:348] T 72a95f55b65240db982c2c443ebb0862 P 5c1e1fc7cc4b4f59b9ca7298b6929b5c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1204c82bd36a468fbdf532c7aad13159" member_type: VOTER last_known_addr { host: "127.0.222.3" port: 46283 } } peers { permanent_uuid: "5c1e1fc7cc4b4f59b9ca7298b6929b5c" member_type: VOTER last_known_addr { host: "127.0.222.2" port: 41313 } } peers { permanent_uuid: "2a033dd02e3d408eadf277165d4c5b95" member_type: VOTER last_known_addr { host: "127.0.222.1" port: 40557 } }
I20260809 12:13:28.669939 16946 raft_consensus.cc:374] T 72a95f55b65240db982c2c443ebb0862 P 5c1e1fc7cc4b4f59b9ca7298b6929b5c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260809 12:13:28.670051 16946 raft_consensus.cc:729] T 72a95f55b65240db982c2c443ebb0862 P 5c1e1fc7cc4b4f59b9ca7298b6929b5c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5c1e1fc7cc4b4f59b9ca7298b6929b5c, State: Initialized, Role: FOLLOWER
I20260809 12:13:28.670665 16946 consensus_queue.cc:260] T 72a95f55b65240db982c2c443ebb0862 P 5c1e1fc7cc4b4f59b9ca7298b6929b5c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1204c82bd36a468fbdf532c7aad13159" member_type: VOTER last_known_addr { host: "127.0.222.3" port: 46283 } } peers { permanent_uuid: "5c1e1fc7cc4b4f59b9ca7298b6929b5c" member_type: VOTER last_known_addr { host: "127.0.222.2" port: 41313 } } peers { permanent_uuid: "2a033dd02e3d408eadf277165d4c5b95" member_type: VOTER last_known_addr { host: "127.0.222.1" port: 40557 } }
I20260809 12:13:28.671814 16787 heartbeater.cc:503] Master 127.0.222.62:44147 was elected leader, sending a full tablet report...
I20260809 12:13:28.672394 16946 ts_tablet_manager.cc:1428] T 72a95f55b65240db982c2c443ebb0862 P 5c1e1fc7cc4b4f59b9ca7298b6929b5c: Time spent starting tablet: real 0.012s user 0.006s sys 0.004s
W20260809 12:13:28.777352 16925 tablet.cc:2369] T 72a95f55b65240db982c2c443ebb0862 P 1204c82bd36a468fbdf532c7aad13159: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260809 12:13:28.806097 16788 tablet.cc:2369] T 72a95f55b65240db982c2c443ebb0862 P 5c1e1fc7cc4b4f59b9ca7298b6929b5c: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260809 12:13:28.823906 16646 tablet.cc:2369] T 72a95f55b65240db982c2c443ebb0862 P 2a033dd02e3d408eadf277165d4c5b95: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260809 12:13:28.877266 16949 raft_consensus.cc:482] T 72a95f55b65240db982c2c443ebb0862 P 1204c82bd36a468fbdf532c7aad13159 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260809 12:13:28.877640 16949 raft_consensus.cc:504] T 72a95f55b65240db982c2c443ebb0862 P 1204c82bd36a468fbdf532c7aad13159 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1204c82bd36a468fbdf532c7aad13159" member_type: VOTER last_known_addr { host: "127.0.222.3" port: 46283 } } peers { permanent_uuid: "5c1e1fc7cc4b4f59b9ca7298b6929b5c" member_type: VOTER last_known_addr { host: "127.0.222.2" port: 41313 } } peers { permanent_uuid: "2a033dd02e3d408eadf277165d4c5b95" member_type: VOTER last_known_addr { host: "127.0.222.1" port: 40557 } }
I20260809 12:13:28.879405 16949 leader_election.cc:288] T 72a95f55b65240db982c2c443ebb0862 P 1204c82bd36a468fbdf532c7aad13159 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 5c1e1fc7cc4b4f59b9ca7298b6929b5c (127.0.222.2:41313), 2a033dd02e3d408eadf277165d4c5b95 (127.0.222.1:40557)
I20260809 12:13:28.886189 16937 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 12:13:28.879974 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.222.3:40783 (local address 127.0.222.1:40557)
0809 12:13:28.880192 (+ 218us) server_negotiation.cc:206] Beginning negotiation
0809 12:13:28.880198 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0809 12:13:28.880223 (+ 25us) server_negotiation.cc:407] Connection header received
0809 12:13:28.880516 (+ 293us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 12:13:28.880522 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 12:13:28.880603 (+ 81us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 12:13:28.880723 (+ 120us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0809 12:13:28.881252 (+ 529us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 12:13:28.882199 (+ 947us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 12:13:28.884137 (+ 1938us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 12:13:28.885196 (+ 1059us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 12:13:28.885303 (+ 107us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 12:13:28.885381 (+ 78us) server_negotiation.cc:1035] Waiting for connection context
0809 12:13:28.885770 (+ 389us) server_negotiation.cc:299] Negotiation successful
0809 12:13:28.885861 (+ 91us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":71}
I20260809 12:13:28.886201 16926 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 12:13:28.879778 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.222.1:40557 (local address 127.0.222.3:40783)
0809 12:13:28.880111 (+ 333us) negotiation.cc:107] Waiting for socket to connect
0809 12:13:28.880134 (+ 23us) client_negotiation.cc:174] Beginning negotiation
0809 12:13:28.880321 (+ 187us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 12:13:28.880831 (+ 510us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 12:13:28.880836 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 12:13:28.880868 (+ 32us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0809 12:13:28.881093 (+ 225us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 12:13:28.881107 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 12:13:28.882350 (+ 1243us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 12:13:28.882356 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 12:13:28.883979 (+ 1623us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 12:13:28.883991 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 12:13:28.885380 (+ 1389us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 12:13:28.885386 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 12:13:28.885595 (+ 209us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 12:13:28.885633 (+ 38us) client_negotiation.cc:769] Sending connection context
0809 12:13:28.885741 (+ 108us) client_negotiation.cc:240] Negotiation successful
0809 12:13:28.885879 (+ 138us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":179,"mutex_wait_us":24}
I20260809 12:13:28.886996 16940 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 12:13:28.879975 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.222.3:36721 (local address 127.0.222.2:41313)
0809 12:13:28.880192 (+ 217us) server_negotiation.cc:206] Beginning negotiation
0809 12:13:28.880199 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0809 12:13:28.880833 (+ 634us) server_negotiation.cc:407] Connection header received
0809 12:13:28.880970 (+ 137us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 12:13:28.880975 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 12:13:28.881074 (+ 99us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 12:13:28.881179 (+ 105us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0809 12:13:28.881787 (+ 608us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 12:13:28.882700 (+ 913us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 12:13:28.884837 (+ 2137us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 12:13:28.885866 (+ 1029us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 12:13:28.885992 (+ 126us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 12:13:28.886089 (+ 97us) server_negotiation.cc:1035] Waiting for connection context
0809 12:13:28.886572 (+ 483us) server_negotiation.cc:299] Negotiation successful
0809 12:13:28.886704 (+ 132us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":80}
I20260809 12:13:28.887071 16951 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 12:13:28.879885 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.222.2:41313 (local address 127.0.222.3:36721)
0809 12:13:28.880685 (+ 800us) negotiation.cc:107] Waiting for socket to connect
0809 12:13:28.880703 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0809 12:13:28.880822 (+ 119us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 12:13:28.881219 (+ 397us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 12:13:28.881224 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 12:13:28.881250 (+ 26us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0809 12:13:28.881628 (+ 378us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 12:13:28.881640 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 12:13:28.882854 (+ 1214us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 12:13:28.882858 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 12:13:28.884625 (+ 1767us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 12:13:28.884639 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 12:13:28.886101 (+ 1462us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 12:13:28.886107 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 12:13:28.886301 (+ 194us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 12:13:28.886338 (+ 37us) client_negotiation.cc:769] Sending connection context
0809 12:13:28.886537 (+ 199us) client_negotiation.cc:240] Negotiation successful
0809 12:13:28.886665 (+ 128us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":676,"mutex_wait_us":56,"thread_start_us":479,"threads_started":1}
I20260809 12:13:28.887485 16588 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "72a95f55b65240db982c2c443ebb0862" candidate_uuid: "1204c82bd36a468fbdf532c7aad13159" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2a033dd02e3d408eadf277165d4c5b95" is_pre_election: true
I20260809 12:13:28.888070 16588 raft_consensus.cc:2454] T 72a95f55b65240db982c2c443ebb0862 P 2a033dd02e3d408eadf277165d4c5b95 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1204c82bd36a468fbdf532c7aad13159 in term 0.
I20260809 12:13:28.888072 16738 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "72a95f55b65240db982c2c443ebb0862" candidate_uuid: "1204c82bd36a468fbdf532c7aad13159" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5c1e1fc7cc4b4f59b9ca7298b6929b5c" is_pre_election: true
I20260809 12:13:28.888521 16738 raft_consensus.cc:2454] T 72a95f55b65240db982c2c443ebb0862 P 5c1e1fc7cc4b4f59b9ca7298b6929b5c [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1204c82bd36a468fbdf532c7aad13159 in term 0.
I20260809 12:13:28.888834 16808 leader_election.cc:302] T 72a95f55b65240db982c2c443ebb0862 P 1204c82bd36a468fbdf532c7aad13159 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1204c82bd36a468fbdf532c7aad13159, 2a033dd02e3d408eadf277165d4c5b95; no voters:
I20260809 12:13:28.889268 16949 raft_consensus.cc:2781] T 72a95f55b65240db982c2c443ebb0862 P 1204c82bd36a468fbdf532c7aad13159 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260809 12:13:28.889425 16949 raft_consensus.cc:482] T 72a95f55b65240db982c2c443ebb0862 P 1204c82bd36a468fbdf532c7aad13159 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260809 12:13:28.889549 16949 raft_consensus.cc:3037] T 72a95f55b65240db982c2c443ebb0862 P 1204c82bd36a468fbdf532c7aad13159 [term 0 FOLLOWER]: Advancing to term 1
I20260809 12:13:28.891808 16949 raft_consensus.cc:504] T 72a95f55b65240db982c2c443ebb0862 P 1204c82bd36a468fbdf532c7aad13159 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1204c82bd36a468fbdf532c7aad13159" member_type: VOTER last_known_addr { host: "127.0.222.3" port: 46283 } } peers { permanent_uuid: "5c1e1fc7cc4b4f59b9ca7298b6929b5c" member_type: VOTER last_known_addr { host: "127.0.222.2" port: 41313 } } peers { permanent_uuid: "2a033dd02e3d408eadf277165d4c5b95" member_type: VOTER last_known_addr { host: "127.0.222.1" port: 40557 } }
I20260809 12:13:28.892391 16949 leader_election.cc:288] T 72a95f55b65240db982c2c443ebb0862 P 1204c82bd36a468fbdf532c7aad13159 [CANDIDATE]: Term 1 election: Requested vote from peers 5c1e1fc7cc4b4f59b9ca7298b6929b5c (127.0.222.2:41313), 2a033dd02e3d408eadf277165d4c5b95 (127.0.222.1:40557)
I20260809 12:13:28.892853 16738 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "72a95f55b65240db982c2c443ebb0862" candidate_uuid: "1204c82bd36a468fbdf532c7aad13159" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5c1e1fc7cc4b4f59b9ca7298b6929b5c"
I20260809 12:13:28.892946 16588 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "72a95f55b65240db982c2c443ebb0862" candidate_uuid: "1204c82bd36a468fbdf532c7aad13159" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2a033dd02e3d408eadf277165d4c5b95"
I20260809 12:13:28.893049 16738 raft_consensus.cc:3037] T 72a95f55b65240db982c2c443ebb0862 P 5c1e1fc7cc4b4f59b9ca7298b6929b5c [term 0 FOLLOWER]: Advancing to term 1
I20260809 12:13:28.893196 16588 raft_consensus.cc:3037] T 72a95f55b65240db982c2c443ebb0862 P 2a033dd02e3d408eadf277165d4c5b95 [term 0 FOLLOWER]: Advancing to term 1
I20260809 12:13:28.895448 16738 raft_consensus.cc:2454] T 72a95f55b65240db982c2c443ebb0862 P 5c1e1fc7cc4b4f59b9ca7298b6929b5c [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1204c82bd36a468fbdf532c7aad13159 in term 1.
I20260809 12:13:28.895524 16588 raft_consensus.cc:2454] T 72a95f55b65240db982c2c443ebb0862 P 2a033dd02e3d408eadf277165d4c5b95 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1204c82bd36a468fbdf532c7aad13159 in term 1.
I20260809 12:13:28.895946 16806 leader_election.cc:302] T 72a95f55b65240db982c2c443ebb0862 P 1204c82bd36a468fbdf532c7aad13159 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1204c82bd36a468fbdf532c7aad13159, 5c1e1fc7cc4b4f59b9ca7298b6929b5c; no voters:
I20260809 12:13:28.896288 16949 raft_consensus.cc:2781] T 72a95f55b65240db982c2c443ebb0862 P 1204c82bd36a468fbdf532c7aad13159 [term 1 FOLLOWER]: Leader election won for term 1
I20260809 12:13:28.896760 16949 raft_consensus.cc:686] T 72a95f55b65240db982c2c443ebb0862 P 1204c82bd36a468fbdf532c7aad13159 [term 1 LEADER]: Becoming Leader. State: Replica: 1204c82bd36a468fbdf532c7aad13159, State: Running, Role: LEADER
I20260809 12:13:28.897300 16949 consensus_queue.cc:237] T 72a95f55b65240db982c2c443ebb0862 P 1204c82bd36a468fbdf532c7aad13159 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1204c82bd36a468fbdf532c7aad13159" member_type: VOTER last_known_addr { host: "127.0.222.3" port: 46283 } } peers { permanent_uuid: "5c1e1fc7cc4b4f59b9ca7298b6929b5c" member_type: VOTER last_known_addr { host: "127.0.222.2" port: 41313 } } peers { permanent_uuid: "2a033dd02e3d408eadf277165d4c5b95" member_type: VOTER last_known_addr { host: "127.0.222.1" port: 40557 } }
I20260809 12:13:28.902312 16436 catalog_manager.cc:5495] T 72a95f55b65240db982c2c443ebb0862 P 1204c82bd36a468fbdf532c7aad13159 reported cstate change: term changed from 0 to 1, leader changed from <none> to 1204c82bd36a468fbdf532c7aad13159 (127.0.222.3). New cstate: current_term: 1 leader_uuid: "1204c82bd36a468fbdf532c7aad13159" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1204c82bd36a468fbdf532c7aad13159" member_type: VOTER last_known_addr { host: "127.0.222.3" port: 46283 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "5c1e1fc7cc4b4f59b9ca7298b6929b5c" member_type: VOTER last_known_addr { host: "127.0.222.2" port: 41313 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "2a033dd02e3d408eadf277165d4c5b95" member_type: VOTER last_known_addr { host: "127.0.222.1" port: 40557 } health_report { overall_health: UNKNOWN } } }
I20260809 12:13:28.924585 16936 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 12:13:28.918953 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56642 (local address 127.0.222.3:46283)
0809 12:13:28.919210 (+ 257us) server_negotiation.cc:206] Beginning negotiation
0809 12:13:28.919215 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0809 12:13:28.919311 (+ 96us) server_negotiation.cc:407] Connection header received
0809 12:13:28.919566 (+ 255us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 12:13:28.919571 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 12:13:28.919659 (+ 88us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 12:13:28.919831 (+ 172us) server_negotiation.cc:226] Negotiated authn=TOKEN
0809 12:13:28.920237 (+ 406us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 12:13:28.920997 (+ 760us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 12:13:28.922524 (+ 1527us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 12:13:28.922942 (+ 418us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 12:13:28.923075 (+ 133us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 12:13:28.923565 (+ 490us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0809 12:13:28.924114 (+ 549us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0809 12:13:28.924229 (+ 115us) server_negotiation.cc:1035] Waiting for connection context
0809 12:13:28.924308 (+ 79us) server_negotiation.cc:299] Negotiation successful
0809 12:13:28.924380 (+ 72us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":64}
I20260809 12:13:28.933609 16588 raft_consensus.cc:1261] T 72a95f55b65240db982c2c443ebb0862 P 2a033dd02e3d408eadf277165d4c5b95 [term 1 FOLLOWER]: Refusing update from remote peer 1204c82bd36a468fbdf532c7aad13159: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260809 12:13:28.933609 16738 raft_consensus.cc:1261] T 72a95f55b65240db982c2c443ebb0862 P 5c1e1fc7cc4b4f59b9ca7298b6929b5c [term 1 FOLLOWER]: Refusing update from remote peer 1204c82bd36a468fbdf532c7aad13159: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260809 12:13:28.934840 16953 consensus_queue.cc:1035] T 72a95f55b65240db982c2c443ebb0862 P 1204c82bd36a468fbdf532c7aad13159 [LEADER]: Connected to new peer: Peer: permanent_uuid: "5c1e1fc7cc4b4f59b9ca7298b6929b5c" member_type: VOTER last_known_addr { host: "127.0.222.2" port: 41313 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260809 12:13:28.935335 16949 consensus_queue.cc:1035] T 72a95f55b65240db982c2c443ebb0862 P 1204c82bd36a468fbdf532c7aad13159 [LEADER]: Connected to new peer: Peer: permanent_uuid: "2a033dd02e3d408eadf277165d4c5b95" member_type: VOTER last_known_addr { host: "127.0.222.1" port: 40557 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260809 12:13:28.949295 16955 mvcc.cc:205] Tried to move back new op lower bound from 7316593086179352576 to 7316593086047760384. Current Snapshot: MvccSnapshot[applied={T|T < 7316593086179352576}]
I20260809 12:13:28.955274 16957 mvcc.cc:205] Tried to move back new op lower bound from 7316593086179352576 to 7316593086047760384. Current Snapshot: MvccSnapshot[applied={T|T < 7316593086179352576}]
I20260809 12:13:28.966826 16437 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin'} at 127.0.0.1:45496:
table { table_name: "test-table" } modify_external_catalogs: true
I20260809 12:13:28.967188 16437 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin'} at 127.0.0.1:45496:
table { table_name: "test-table" } modify_external_catalogs: true
I20260809 12:13:28.971153 16437 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 6d7f8671e34d46459506bc2cba79c82c: Sending DeleteTablet for 3 replicas of tablet 72a95f55b65240db982c2c443ebb0862
I20260809 12:13:28.972594 16568 tablet_service.cc:1514] Processing DeleteTablet for tablet 72a95f55b65240db982c2c443ebb0862 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-09 12:13:28 UTC) from {username='kudu', principal='kudu/127.0.222.62@KRBTEST.COM'} at 127.0.0.1:51386
I20260809 12:13:28.972623 16857 tablet_service.cc:1514] Processing DeleteTablet for tablet 72a95f55b65240db982c2c443ebb0862 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-09 12:13:28 UTC) from {username='kudu', principal='kudu/127.0.222.62@KRBTEST.COM'} at 127.0.0.1:56626
I20260809 12:13:28.973377 16716 tablet_service.cc:1514] Processing DeleteTablet for tablet 72a95f55b65240db982c2c443ebb0862 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-09 12:13:28 UTC) from {username='kudu', principal='kudu/127.0.222.62@KRBTEST.COM'} at 127.0.0.1:53672
I20260809 12:13:28.973572 16965 tablet_replica.cc:330] T 72a95f55b65240db982c2c443ebb0862 P 2a033dd02e3d408eadf277165d4c5b95: stopping tablet replica
I20260809 12:13:28.973573 16966 tablet_replica.cc:330] T 72a95f55b65240db982c2c443ebb0862 P 1204c82bd36a468fbdf532c7aad13159: stopping tablet replica
I20260809 12:13:28.974028 16966 raft_consensus.cc:2229] T 72a95f55b65240db982c2c443ebb0862 P 1204c82bd36a468fbdf532c7aad13159 [term 1 LEADER]: Raft consensus shutting down.
I20260809 12:13:28.974021 16965 raft_consensus.cc:2229] T 72a95f55b65240db982c2c443ebb0862 P 2a033dd02e3d408eadf277165d4c5b95 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260809 12:13:28.974570 888 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskgH5Pjm/build/asan/bin/kudu pid 16478
I20260809 12:13:28.974610 16965 raft_consensus.cc:2258] T 72a95f55b65240db982c2c443ebb0862 P 2a033dd02e3d408eadf277165d4c5b95 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260809 12:13:28.974788 16966 raft_consensus.cc:2258] T 72a95f55b65240db982c2c443ebb0862 P 1204c82bd36a468fbdf532c7aad13159 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260809 12:13:28.977278 16966 ts_tablet_manager.cc:1905] T 72a95f55b65240db982c2c443ebb0862 P 1204c82bd36a468fbdf532c7aad13159: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260809 12:13:28.983507 16965 ts_tablet_manager.cc:1905] T 72a95f55b65240db982c2c443ebb0862 P 2a033dd02e3d408eadf277165d4c5b95: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260809 12:13:28.985827 16966 ts_tablet_manager.cc:1918] T 72a95f55b65240db982c2c443ebb0862 P 1204c82bd36a468fbdf532c7aad13159: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260809 12:13:28.986078 16966 log.cc:1196] T 72a95f55b65240db982c2c443ebb0862 P 1204c82bd36a468fbdf532c7aad13159: Deleting WAL directory at /tmp/dist-test-taskgH5Pjm/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786277382073938-888-0/minicluster-data/ts-2/wal/wals/72a95f55b65240db982c2c443ebb0862
I20260809 12:13:28.986796 16966 ts_tablet_manager.cc:1939] T 72a95f55b65240db982c2c443ebb0862 P 1204c82bd36a468fbdf532c7aad13159: Deleting consensus metadata
I20260809 12:13:28.988982 16420 catalog_manager.cc:4841] TS 1204c82bd36a468fbdf532c7aad13159 (127.0.222.3:46283): tablet 72a95f55b65240db982c2c443ebb0862 (table test-table [id=fe0aaad699bf4737978fae44f8b05436]) successfully deleted
I20260809 12:13:28.994026 16967 tablet_replica.cc:330] T 72a95f55b65240db982c2c443ebb0862 P 5c1e1fc7cc4b4f59b9ca7298b6929b5c: stopping tablet replica
I20260809 12:13:28.994597 16967 raft_consensus.cc:2229] T 72a95f55b65240db982c2c443ebb0862 P 5c1e1fc7cc4b4f59b9ca7298b6929b5c [term 1 FOLLOWER]: Raft consensus shutting down.
I20260809 12:13:28.995368 16967 raft_consensus.cc:2258] T 72a95f55b65240db982c2c443ebb0862 P 5c1e1fc7cc4b4f59b9ca7298b6929b5c [term 1 FOLLOWER]: Raft consensus is shut down!
I20260809 12:13:28.997589 16967 ts_tablet_manager.cc:1905] T 72a95f55b65240db982c2c443ebb0862 P 5c1e1fc7cc4b4f59b9ca7298b6929b5c: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260809 12:13:28.998458 16937 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 12:13:28.975811 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51400 (local address 127.0.222.1:40557)
0809 12:13:28.975999 (+ 188us) server_negotiation.cc:206] Beginning negotiation
0809 12:13:28.976006 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0809 12:13:28.976058 (+ 52us) server_negotiation.cc:407] Connection header received
0809 12:13:28.976267 (+ 209us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 12:13:28.976275 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 12:13:28.976360 (+ 85us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 12:13:28.976474 (+ 114us) server_negotiation.cc:226] Negotiated authn=SASL
0809 12:13:28.979150 (+ 2676us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 12:13:28.980283 (+ 1133us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 12:13:28.982940 (+ 2657us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 12:13:28.983596 (+ 656us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 12:13:28.983734 (+ 138us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 12:13:28.988404 (+ 4670us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 12:13:28.988416 (+ 12us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 12:13:28.988433 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0809 12:13:28.988476 (+ 43us) server_negotiation.cc:922] calling sasl_server_start()
0809 12:13:28.993275 (+ 4799us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 12:13:28.994267 (+ 992us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 12:13:28.994278 (+ 11us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 12:13:28.994282 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0809 12:13:28.994459 (+ 177us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 12:13:28.995121 (+ 662us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 12:13:28.995129 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 12:13:28.995135 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0809 12:13:28.995720 (+ 585us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 12:13:28.996373 (+ 653us) server_negotiation.cc:1035] Waiting for connection context
0809 12:13:28.997771 (+ 1398us) server_negotiation.cc:299] Negotiation successful
0809 12:13:28.998045 (+ 274us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":57}
I20260809 12:13:28.999274 16965 ts_tablet_manager.cc:1918] T 72a95f55b65240db982c2c443ebb0862 P 2a033dd02e3d408eadf277165d4c5b95: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260809 12:13:28.999503 16965 log.cc:1196] T 72a95f55b65240db982c2c443ebb0862 P 2a033dd02e3d408eadf277165d4c5b95: Deleting WAL directory at /tmp/dist-test-taskgH5Pjm/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786277382073938-888-0/minicluster-data/ts-0/wal/wals/72a95f55b65240db982c2c443ebb0862
I20260809 12:13:28.999636 16640 generic_service.cc:196] Checking for leaks (request via RPC)
I20260809 12:13:29.005507 16967 ts_tablet_manager.cc:1918] T 72a95f55b65240db982c2c443ebb0862 P 5c1e1fc7cc4b4f59b9ca7298b6929b5c: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260809 12:13:29.005719 16967 log.cc:1196] T 72a95f55b65240db982c2c443ebb0862 P 5c1e1fc7cc4b4f59b9ca7298b6929b5c: Deleting WAL directory at /tmp/dist-test-taskgH5Pjm/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786277382073938-888-0/minicluster-data/ts-1/wal/wals/72a95f55b65240db982c2c443ebb0862
I20260809 12:13:29.006268 16967 ts_tablet_manager.cc:1939] T 72a95f55b65240db982c2c443ebb0862 P 5c1e1fc7cc4b4f59b9ca7298b6929b5c: Deleting consensus metadata
I20260809 12:13:29.008142 16420 catalog_manager.cc:4841] TS 5c1e1fc7cc4b4f59b9ca7298b6929b5c (127.0.222.2:41313): tablet 72a95f55b65240db982c2c443ebb0862 (table test-table [id=fe0aaad699bf4737978fae44f8b05436]) successfully deleted
I20260809 12:13:29.124064 16965 ts_tablet_manager.cc:1939] T 72a95f55b65240db982c2c443ebb0862 P 2a033dd02e3d408eadf277165d4c5b95: Deleting consensus metadata
I20260809 12:13:29.124815 888 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskgH5Pjm/build/asan/bin/kudu with pid 16478
W20260809 12:13:29.146241 16422 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.0.222.1:40557 (error 108)
I20260809 12:13:29.146481 888 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskgH5Pjm/build/asan/bin/kudu pid 16649
I20260809 12:13:29.148552 16970 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 12:13:29.147821 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.222.1:40557 (local address 127.0.0.1:51410)
0809 12:13:29.148293 (+ 472us) negotiation.cc:107] Waiting for socket to connect
0809 12:13:29.148393 (+ 100us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.222.1:40557: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":348,"thread_start_us":252,"threads_started":1}
W20260809 12:13:29.148870 16422 catalog_manager.cc:4584] TS 2a033dd02e3d408eadf277165d4c5b95 (127.0.222.1:40557): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 72a95f55b65240db982c2c443ebb0862: Network error: Client connection negotiation failed: client connection to 127.0.222.1:40557: connect: Connection refused (error 111)
I20260809 12:13:29.163254 16940 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 12:13:29.147336 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53684 (local address 127.0.222.2:41313)
0809 12:13:29.147537 (+ 201us) server_negotiation.cc:206] Beginning negotiation
0809 12:13:29.147544 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0809 12:13:29.147654 (+ 110us) server_negotiation.cc:407] Connection header received
0809 12:13:29.147855 (+ 201us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 12:13:29.147861 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 12:13:29.147951 (+ 90us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 12:13:29.148057 (+ 106us) server_negotiation.cc:226] Negotiated authn=SASL
0809 12:13:29.150019 (+ 1962us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 12:13:29.151330 (+ 1311us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 12:13:29.152634 (+ 1304us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 12:13:29.153272 (+ 638us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 12:13:29.153400 (+ 128us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 12:13:29.156220 (+ 2820us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 12:13:29.156246 (+ 26us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 12:13:29.156292 (+ 46us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0809 12:13:29.156341 (+ 49us) server_negotiation.cc:922] calling sasl_server_start()
0809 12:13:29.160099 (+ 3758us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 12:13:29.160830 (+ 731us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 12:13:29.160839 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 12:13:29.160842 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0809 12:13:29.160987 (+ 145us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 12:13:29.161425 (+ 438us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 12:13:29.161431 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 12:13:29.161434 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0809 12:13:29.161938 (+ 504us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 12:13:29.162207 (+ 269us) server_negotiation.cc:1035] Waiting for connection context
0809 12:13:29.162586 (+ 379us) server_negotiation.cc:299] Negotiation successful
0809 12:13:29.162825 (+ 239us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":70}
I20260809 12:13:29.164026 16782 generic_service.cc:196] Checking for leaks (request via RPC)
I20260809 12:13:29.202702 16970 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 12:13:29.202091 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.222.1:40557 (local address 127.0.0.1:51412)
0809 12:13:29.202377 (+ 286us) negotiation.cc:107] Waiting for socket to connect
0809 12:13:29.202502 (+ 125us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.222.1:40557: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":101}
I20260809 12:13:29.204015 16970 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 12:13:29.203567 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.222.1:40557 (local address 127.0.0.1:51426)
0809 12:13:29.203756 (+ 189us) negotiation.cc:107] Waiting for socket to connect
0809 12:13:29.203850 (+ 94us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.222.1:40557: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":57}
I20260809 12:13:29.257299 888 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskgH5Pjm/build/asan/bin/kudu with pid 16649
I20260809 12:13:29.265010 16970 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 12:13:29.264374 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.222.1:40557 (local address 127.0.0.1:51442)
0809 12:13:29.264639 (+ 265us) negotiation.cc:107] Waiting for socket to connect
0809 12:13:29.264759 (+ 120us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.222.1:40557: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":88}
I20260809 12:13:29.266378 16970 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 12:13:29.265890 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.222.1:40557 (local address 127.0.0.1:51456)
0809 12:13:29.266080 (+ 190us) negotiation.cc:107] Waiting for socket to connect
0809 12:13:29.266179 (+ 99us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.222.1:40557: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":58}
I20260809 12:13:29.281101 888 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskgH5Pjm/build/asan/bin/kudu pid 16790
I20260809 12:13:29.298662 16936 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 12:13:29.282105 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56644 (local address 127.0.222.3:46283)
0809 12:13:29.282315 (+ 210us) server_negotiation.cc:206] Beginning negotiation
0809 12:13:29.282320 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0809 12:13:29.282338 (+ 18us) server_negotiation.cc:407] Connection header received
0809 12:13:29.282392 (+ 54us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 12:13:29.282396 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 12:13:29.282455 (+ 59us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 12:13:29.282541 (+ 86us) server_negotiation.cc:226] Negotiated authn=SASL
0809 12:13:29.284716 (+ 2175us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 12:13:29.285607 (+ 891us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 12:13:29.287746 (+ 2139us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 12:13:29.288218 (+ 472us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 12:13:29.288330 (+ 112us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 12:13:29.291932 (+ 3602us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 12:13:29.291945 (+ 13us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 12:13:29.291961 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0809 12:13:29.292013 (+ 52us) server_negotiation.cc:922] calling sasl_server_start()
0809 12:13:29.295373 (+ 3360us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 12:13:29.296394 (+ 1021us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 12:13:29.296402 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 12:13:29.296404 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0809 12:13:29.296587 (+ 183us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 12:13:29.297087 (+ 500us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 12:13:29.297090 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 12:13:29.297092 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0809 12:13:29.297473 (+ 381us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 12:13:29.297705 (+ 232us) server_negotiation.cc:1035] Waiting for connection context
0809 12:13:29.298098 (+ 393us) server_negotiation.cc:299] Negotiation successful
0809 12:13:29.298267 (+ 169us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":65}
I20260809 12:13:29.299578 16919 generic_service.cc:196] Checking for leaks (request via RPC)
I20260809 12:13:29.347129 16970 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 12:13:29.346554 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.222.1:40557 (local address 127.0.0.1:51458)
0809 12:13:29.346802 (+ 248us) negotiation.cc:107] Waiting for socket to connect
0809 12:13:29.346893 (+ 91us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.222.1:40557: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":97}
I20260809 12:13:29.348229 16970 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 12:13:29.347806 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.222.1:40557 (local address 127.0.0.1:51468)
0809 12:13:29.348010 (+ 204us) negotiation.cc:107] Waiting for socket to connect
0809 12:13:29.348077 (+ 67us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.222.1:40557: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":98}
I20260809 12:13:29.427203 888 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskgH5Pjm/build/asan/bin/kudu with pid 16790
I20260809 12:13:29.448552 888 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskgH5Pjm/build/asan/bin/kudu pid 16402
I20260809 12:13:29.449744 16471 generic_service.cc:196] Checking for leaks (request via RPC)
I20260809 12:13:29.566816 16970 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 12:13:29.566104 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.222.1:40557 (local address 127.0.0.1:51474)
0809 12:13:29.566366 (+ 262us) negotiation.cc:107] Waiting for socket to connect
0809 12:13:29.566529 (+ 163us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.222.1:40557: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":69}
I20260809 12:13:29.567060 888 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskgH5Pjm/build/asan/bin/kudu with pid 16402
I20260809 12:13:29.591569 888 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskgH5Pjm/build/asan/bin/kudu
/tmp/dist-test-taskgH5Pjm/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskgH5Pjm/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786277382073938-888-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskgH5Pjm/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786277382073938-888-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskgH5Pjm/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786277382073938-888-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskgH5Pjm/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786277382073938-888-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.222.62:44147
--webserver_interface=127.0.222.62
--webserver_port=42133
--keytab_file=/tmp/dist-test-taskgH5Pjm/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786277382073938-888-0/krb5kdc/kudu_127.0.222.62.keytab
--principal=kudu/127.0.222.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.222.20:41797
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.222.62:44147
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskgH5Pjm/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786277382073938-888-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskgH5Pjm/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786277382073938-888-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskgH5Pjm/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786277382073938-888-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260809 12:13:29.954548 16974 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260809 12:13:29.954947 16974 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260809 12:13:29.955132 16974 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260809 12:13:29.963115 16974 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260809 12:13:29.963271 16974 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260809 12:13:29.963376 16974 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260809 12:13:29.963491 16974 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260809 12:13:29.963584 16974 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260809 12:13:29.974129 16974 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.222.20:41797
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskgH5Pjm/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786277382073938-888-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskgH5Pjm/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786277382073938-888-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.222.62:44147
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskgH5Pjm/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786277382073938-888-0/krb5kdc/kudu_127.0.222.62.keytab
--principal=kudu/127.0.222.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.222.62:44147
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskgH5Pjm/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786277382073938-888-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.222.62
--webserver_port=42133
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskgH5Pjm/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786277382073938-888-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 09 Aug 2026 11:06:41 UTC on ccycloud.quasar-bwoogy.root.comops.site
build id 4575991
ASAN enabled
I20260809 12:13:29.975765 16974 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260809 12:13:29.977137 16974 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260809 12:13:29.989403 16983 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 12:13:29.989370 16979 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 12:13:29.989454 16980 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260809 12:13:29.990691 16974 server_base.cc:1034] running on GCE node
I20260809 12:13:29.991583 16974 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260809 12:13:29.993110 16974 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260809 12:13:29.994323 16974 hybrid_clock.cc:648] HybridClock initialized: now 1786277609994273 us; error 36 us; skew 500 ppm
I20260809 12:13:29.994699 16974 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 09 12:13:29 dist-test-slave-2w3w krb5kdc[15823](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786277609, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.222.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260809 12:13:29.999280 16974 init.cc:377] Logged in from keytab as kudu/127.0.222.62@KRBTEST.COM (short username kudu)
I20260809 12:13:30.001484 16974 webserver.cc:460] Webserver started at http://127.0.222.62:42133/ using document root <none> and password file <none>
I20260809 12:13:30.002535 16974 fs_manager.cc:359] Metadata directory not provided
I20260809 12:13:30.002704 16974 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260809 12:13:30.011189 16974 fs_manager.cc:709] Time spent opening directory manager: real 0.005s user 0.006s sys 0.001s
I20260809 12:13:30.015184 16990 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260809 12:13:30.017012 16974 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.004s sys 0.000s
I20260809 12:13:30.017176 16974 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskgH5Pjm/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786277382073938-888-0/minicluster-data/master-0/data,/tmp/dist-test-taskgH5Pjm/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786277382073938-888-0/minicluster-data/master-0/wal
uuid: "6d7f8671e34d46459506bc2cba79c82c"
format_stamp: "Formatted at 2026-08-09 12:13:23 on dist-test-slave-2w3w"
server_key: "5cd736e973b6ffe5de54c3107192d76b"
server_key_iv: "493043018edc30ecf9a63340ec4f2759"
server_key_version: "encryptionkey@0"
I20260809 12:13:30.018190 16974 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskgH5Pjm/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786277382073938-888-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskgH5Pjm/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786277382073938-888-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskgH5Pjm/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786277382073938-888-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260809 12:13:30.035758 16974 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260809 12:13:30.040117 16974 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260809 12:13:30.040365 16974 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260809 12:13:30.060045 16974 rpc_server.cc:307] RPC server started. Bound to: 127.0.222.62:44147
I20260809 12:13:30.060065 17043 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.222.62:44147 every 8 connection(s)
I20260809 12:13:30.061702 16974 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskgH5Pjm/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786277382073938-888-0/minicluster-data/master-0/data/info.pb
I20260809 12:13:30.064189 888 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskgH5Pjm/build/asan/bin/kudu as pid 16974
I20260809 12:13:30.065258 888 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskgH5Pjm/build/asan/bin/kudu
/tmp/dist-test-taskgH5Pjm/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskgH5Pjm/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786277382073938-888-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskgH5Pjm/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786277382073938-888-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskgH5Pjm/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786277382073938-888-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskgH5Pjm/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786277382073938-888-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.222.1:40557
--local_ip_for_outbound_sockets=127.0.222.1
--tserver_master_addrs=127.0.222.62:44147
--webserver_port=44545
--webserver_interface=127.0.222.1
--keytab_file=/tmp/dist-test-taskgH5Pjm/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786277382073938-888-0/krb5kdc/kudu_127.0.222.1.keytab
--principal=kudu/127.0.222.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.222.20:41797
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskgH5Pjm/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786277382073938-888-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskgH5Pjm/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786277382073938-888-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskgH5Pjm/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786277382073938-888-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260809 12:13:30.072067 17044 sys_catalog.cc:263] Verifying existing consensus state
I20260809 12:13:30.077601 17044 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 6d7f8671e34d46459506bc2cba79c82c: Bootstrap starting.
I20260809 12:13:30.099223 17044 log.cc:824] T 00000000000000000000000000000000 P 6d7f8671e34d46459506bc2cba79c82c: Log is configured to *not* fsync() on all Append() calls
I20260809 12:13:30.116204 17044 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 6d7f8671e34d46459506bc2cba79c82c: Bootstrap replayed 1/1 log segments. Stats: ops{read=14 overwritten=0 applied=14 ignored=0} inserts{seen=7 ignored=0} mutations{seen=9 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260809 12:13:30.116820 17044 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 6d7f8671e34d46459506bc2cba79c82c: Bootstrap complete.
I20260809 12:13:30.126924 17044 raft_consensus.cc:348] T 00000000000000000000000000000000 P 6d7f8671e34d46459506bc2cba79c82c [term 2 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6d7f8671e34d46459506bc2cba79c82c" member_type: VOTER last_known_addr { host: "127.0.222.62" port: 44147 } }
I20260809 12:13:30.128013 17044 raft_consensus.cc:729] T 00000000000000000000000000000000 P 6d7f8671e34d46459506bc2cba79c82c [term 2 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6d7f8671e34d46459506bc2cba79c82c, State: Initialized, Role: FOLLOWER
I20260809 12:13:30.128712 17044 consensus_queue.cc:260] T 00000000000000000000000000000000 P 6d7f8671e34d46459506bc2cba79c82c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 14, Last appended: 2.14, Last appended by leader: 14, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6d7f8671e34d46459506bc2cba79c82c" member_type: VOTER last_known_addr { host: "127.0.222.62" port: 44147 } }
I20260809 12:13:30.128960 17044 raft_consensus.cc:388] T 00000000000000000000000000000000 P 6d7f8671e34d46459506bc2cba79c82c [term 2 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260809 12:13:30.129084 17044 raft_consensus.cc:482] T 00000000000000000000000000000000 P 6d7f8671e34d46459506bc2cba79c82c [term 2 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260809 12:13:30.129298 17044 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 6d7f8671e34d46459506bc2cba79c82c [term 2 FOLLOWER]: Advancing to term 3
I20260809 12:13:30.133394 17044 raft_consensus.cc:504] T 00000000000000000000000000000000 P 6d7f8671e34d46459506bc2cba79c82c [term 3 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6d7f8671e34d46459506bc2cba79c82c" member_type: VOTER last_known_addr { host: "127.0.222.62" port: 44147 } }
I20260809 12:13:30.133927 17044 leader_election.cc:302] T 00000000000000000000000000000000 P 6d7f8671e34d46459506bc2cba79c82c [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 6d7f8671e34d46459506bc2cba79c82c; no voters:
I20260809 12:13:30.134663 17044 leader_election.cc:288] T 00000000000000000000000000000000 P 6d7f8671e34d46459506bc2cba79c82c [CANDIDATE]: Term 3 election: Requested vote from peers
I20260809 12:13:30.134820 17048 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 6d7f8671e34d46459506bc2cba79c82c [term 3 FOLLOWER]: Leader election won for term 3
I20260809 12:13:30.136024 17048 raft_consensus.cc:686] T 00000000000000000000000000000000 P 6d7f8671e34d46459506bc2cba79c82c [term 3 LEADER]: Becoming Leader. State: Replica: 6d7f8671e34d46459506bc2cba79c82c, State: Running, Role: LEADER
I20260809 12:13:30.136729 17048 consensus_queue.cc:237] T 00000000000000000000000000000000 P 6d7f8671e34d46459506bc2cba79c82c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 14, Committed index: 14, Last appended: 2.14, Last appended by leader: 14, Current term: 3, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6d7f8671e34d46459506bc2cba79c82c" member_type: VOTER last_known_addr { host: "127.0.222.62" port: 44147 } }
I20260809 12:13:30.137089 17044 sys_catalog.cc:564] T 00000000000000000000000000000000 P 6d7f8671e34d46459506bc2cba79c82c [sys.catalog]: configured and running, proceeding with master startup.
I20260809 12:13:30.139384 17050 sys_catalog.cc:455] T 00000000000000000000000000000000 P 6d7f8671e34d46459506bc2cba79c82c [sys.catalog]: SysCatalogTable state changed. Reason: New leader 6d7f8671e34d46459506bc2cba79c82c. Latest consensus state: current_term: 3 leader_uuid: "6d7f8671e34d46459506bc2cba79c82c" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6d7f8671e34d46459506bc2cba79c82c" member_type: VOTER last_known_addr { host: "127.0.222.62" port: 44147 } } }
I20260809 12:13:30.139802 17050 sys_catalog.cc:458] T 00000000000000000000000000000000 P 6d7f8671e34d46459506bc2cba79c82c [sys.catalog]: This master's current role is: LEADER
I20260809 12:13:30.140506 17049 sys_catalog.cc:455] T 00000000000000000000000000000000 P 6d7f8671e34d46459506bc2cba79c82c [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 3 leader_uuid: "6d7f8671e34d46459506bc2cba79c82c" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6d7f8671e34d46459506bc2cba79c82c" member_type: VOTER last_known_addr { host: "127.0.222.62" port: 44147 } } }
I20260809 12:13:30.140786 17049 sys_catalog.cc:458] T 00000000000000000000000000000000 P 6d7f8671e34d46459506bc2cba79c82c [sys.catalog]: This master's current role is: LEADER
I20260809 12:13:30.147872 17057 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260809 12:13:30.159394 17057 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260809 12:13:30.161635 17057 catalog_manager.cc:1259] Loaded cluster ID: 71a354fc623b4b8f864d7005c8295487
I20260809 12:13:30.161846 17057 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
F20260809 12:13:30.165417 17057 catalog_manager.cc:1466] Initializing Kudu internal certificate authority failed: Runtime error: unable to load data from memory: error:0D0680A8:asn1 encoding routines:ASN1_CHECK_TLEN:wrong tag:tasn_dec.c:1220 error:0D06C03A:asn1 encoding routines:ASN1_D2I_EX_PRIMITIVE:nested asn1 error:tasn_dec.c:788 error:0D08303A:asn1 encoding routines:ASN1_TEMPLATE_NOEXP_D2I:nested asn1 error:tasn_dec.c:720:Field=version, Type=RSA error:04093004:rsa routines:OLD_RSA_PRIV_DECODE:RSA lib:rsa_ameth.c:121 error:0D0680A8:asn1 encoding routines:ASN1_CHECK_TLEN:wrong tag:tasn_dec.c:1220 error:0D06C03A:asn1 encoding routines:ASN1_D2I_EX_PRIMITIVE:nested asn1 error:tasn_dec.c:788 error:0D08303A:asn1 encoding routines:ASN1_TEMPLATE_NOEXP_D2I:nested asn1 error:tasn_dec.c:720:Field=version, Type=PKCS8_PRIV_KEY_INFO
*** Check failure stack trace: ***
@ 0x7fd32678a9ad google::LogMessage::Fail() at ??:0
@ 0x7fd32678d88a google::LogMessage::SendToLog() at ??:0
@ 0x7fd32678a4de google::LogMessage::Flush() at ??:0
@ 0x7fd32678ba19 google::LogMessageFatal::~LogMessageFatal() at ??:0
I20260809 12:13:30.187595 17067 catalog_manager.cc:795] Waiting for catalog manager background task thread to start: Service unavailable: Catalog manager is not initialized. State: Starting
@ 0x7fd33c1bd512 kudu::master::CatalogManager::PrepareForLeadershipTask()::$_6::operator()() at ??:0
@ 0x7fd33c1ba976 kudu::master::CatalogManager::PrepareForLeadershipTask() at ??:0
@ 0x7fd33c242610 kudu::master::CatalogManager::ElectedAsLeaderCb()::$_2::operator()() at ??:0
@ 0x7fd33c242439 std::_Function_handler<>::_M_invoke() at ??:0
@ 0x9edb7d std::function<>::operator()() at ??:?
@ 0x7fd3293d6345 kudu::ThreadPool::DispatchThread() at ??:0
@ 0x7fd3293d991c kudu::ThreadPool::CreateThread()::$_2::operator()() at ??:0
@ 0x7fd3293d9799 std::_Function_handler<>::_M_invoke() at ??:0
@ 0x9edb7d std::function<>::operator()() at ??:?
@ 0x7fd3293b4eaf kudu::Thread::SuperviseThread() at ??:0
@ 0x7fd32c68e6db start_thread at ??:0
@ 0x7fd32562461f clone at ??:0
W20260809 12:13:30.511389 17046 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260809 12:13:30.511708 17046 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260809 12:13:30.511902 17046 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260809 12:13:30.519366 17046 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260809 12:13:30.519526 17046 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260809 12:13:30.519726 17046 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.222.1
I20260809 12:13:30.529922 17046 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.222.20:41797
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskgH5Pjm/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786277382073938-888-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskgH5Pjm/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786277382073938-888-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskgH5Pjm/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786277382073938-888-0/krb5kdc/kudu_127.0.222.1.keytab
--principal=kudu/127.0.222.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.222.1:40557
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskgH5Pjm/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786277382073938-888-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.222.1
--webserver_port=44545
--tserver_master_addrs=127.0.222.62:44147
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.222.1
--log_dir=/tmp/dist-test-taskgH5Pjm/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786277382073938-888-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 09 Aug 2026 11:06:41 UTC on ccycloud.quasar-bwoogy.root.comops.site
build id 4575991
ASAN enabled
I20260809 12:13:30.531375 17046 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260809 12:13:30.532784 17046 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260809 12:13:30.546999 17088 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 12:13:30.547133 17087 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 12:13:30.547103 17090 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260809 12:13:30.548074 17046 server_base.cc:1034] running on GCE node
I20260809 12:13:30.548729 17046 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260809 12:13:30.549702 17046 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260809 12:13:30.550908 17046 hybrid_clock.cc:648] HybridClock initialized: now 1786277610550868 us; error 30 us; skew 500 ppm
I20260809 12:13:30.551254 17046 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 09 12:13:30 dist-test-slave-2w3w krb5kdc[15823](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786277610, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.222.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260809 12:13:30.554512 17046 init.cc:377] Logged in from keytab as kudu/127.0.222.1@KRBTEST.COM (short username kudu)
I20260809 12:13:30.556035 17046 webserver.cc:460] Webserver started at http://127.0.222.1:44545/ using document root <none> and password file <none>
I20260809 12:13:30.556758 17046 fs_manager.cc:359] Metadata directory not provided
I20260809 12:13:30.556917 17046 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260809 12:13:30.562438 17046 fs_manager.cc:709] Time spent opening directory manager: real 0.004s user 0.005s sys 0.000s
I20260809 12:13:30.565465 17098 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260809 12:13:30.566860 17046 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.000s sys 0.002s
I20260809 12:13:30.567071 17046 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskgH5Pjm/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786277382073938-888-0/minicluster-data/ts-0/data,/tmp/dist-test-taskgH5Pjm/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786277382073938-888-0/minicluster-data/ts-0/wal
uuid: "2a033dd02e3d408eadf277165d4c5b95"
format_stamp: "Formatted at 2026-08-09 12:13:24 on dist-test-slave-2w3w"
server_key: "cbad234163032fa77b32d4e26b9f9057"
server_key_iv: "bbf5477b04df91cc882d7c584ab72de6"
server_key_version: "encryptionkey@0"
I20260809 12:13:30.568074 17046 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskgH5Pjm/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786277382073938-888-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskgH5Pjm/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786277382073938-888-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskgH5Pjm/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786277382073938-888-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260809 12:13:30.583513 17046 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260809 12:13:30.587760 17046 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260809 12:13:30.588009 17046 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260809 12:13:30.589028 17046 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260809 12:13:30.590325 17046 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260809 12:13:30.590425 17046 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260809 12:13:30.590517 17046 ts_tablet_manager.cc:610] Registered 0 tablets
I20260809 12:13:30.590595 17046 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260809 12:13:30.626133 17046 rpc_server.cc:307] RPC server started. Bound to: 127.0.222.1:40557
I20260809 12:13:30.626227 17222 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.222.1:40557 every 8 connection(s)
I20260809 12:13:30.627735 17046 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskgH5Pjm/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786277382073938-888-0/minicluster-data/ts-0/data/info.pb
I20260809 12:13:30.630247 888 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskgH5Pjm/build/asan/bin/kudu as pid 17046
I20260809 12:13:30.631163 888 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskgH5Pjm/build/asan/bin/kudu
/tmp/dist-test-taskgH5Pjm/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskgH5Pjm/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786277382073938-888-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskgH5Pjm/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786277382073938-888-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskgH5Pjm/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786277382073938-888-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskgH5Pjm/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786277382073938-888-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.222.2:41313
--local_ip_for_outbound_sockets=127.0.222.2
--tserver_master_addrs=127.0.222.62:44147
--webserver_port=46335
--webserver_interface=127.0.222.2
--keytab_file=/tmp/dist-test-taskgH5Pjm/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786277382073938-888-0/krb5kdc/kudu_127.0.222.2.keytab
--principal=kudu/127.0.222.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.222.20:41797
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskgH5Pjm/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786277382073938-888-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskgH5Pjm/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786277382073938-888-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskgH5Pjm/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786277382073938-888-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260809 12:13:30.633325 17225 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 12:13:30.631278 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.222.62:44147 (local address 127.0.222.1:33057)
0809 12:13:30.632221 (+ 943us) negotiation.cc:107] Waiting for socket to connect
0809 12:13:30.632434 (+ 213us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.222.62:44147: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":656,"thread_start_us":431,"threads_started":1}
W20260809 12:13:30.634093 17223 heartbeater.cc:644] Failed to heartbeat to 127.0.222.62:44147 (0 consecutive failures): Network error: Failed to ping master at 127.0.222.62:44147: Client connection negotiation failed: client connection to 127.0.222.62:44147: connect: Connection refused (error 111)
I20260809 12:13:30.635116 17225 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 12:13:30.634702 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.222.62:44147 (local address 127.0.222.1:57127)
0809 12:13:30.634883 (+ 181us) negotiation.cc:107] Waiting for socket to connect
0809 12:13:30.634952 (+ 69us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.222.62:44147: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":60}
I20260809 12:13:30.636049 17225 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 12:13:30.635628 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.222.62:44147 (local address 127.0.222.1:48167)
0809 12:13:30.635770 (+ 142us) negotiation.cc:107] Waiting for socket to connect
0809 12:13:30.635882 (+ 112us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.222.62:44147: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":51}
W20260809 12:13:30.636279 17223 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
W20260809 12:13:30.980726 17226 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260809 12:13:30.981029 17226 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260809 12:13:30.981210 17226 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260809 12:13:30.988569 17226 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260809 12:13:30.988685 17226 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260809 12:13:30.988907 17226 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.222.2
I20260809 12:13:30.998770 17226 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.222.20:41797
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskgH5Pjm/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786277382073938-888-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskgH5Pjm/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786277382073938-888-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskgH5Pjm/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786277382073938-888-0/krb5kdc/kudu_127.0.222.2.keytab
--principal=kudu/127.0.222.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.222.2:41313
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskgH5Pjm/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786277382073938-888-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.222.2
--webserver_port=46335
--tserver_master_addrs=127.0.222.62:44147
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.222.2
--log_dir=/tmp/dist-test-taskgH5Pjm/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786277382073938-888-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 09 Aug 2026 11:06:41 UTC on ccycloud.quasar-bwoogy.root.comops.site
build id 4575991
ASAN enabled
I20260809 12:13:31.000237 17226 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260809 12:13:31.001590 17226 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260809 12:13:31.015206 17234 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 12:13:31.015161 17231 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 12:13:31.015252 17232 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260809 12:13:31.016160 17226 server_base.cc:1034] running on GCE node
I20260809 12:13:31.016790 17226 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260809 12:13:31.017782 17226 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260809 12:13:31.019068 17226 hybrid_clock.cc:648] HybridClock initialized: now 1786277611019012 us; error 44 us; skew 500 ppm
I20260809 12:13:31.019420 17226 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 09 12:13:31 dist-test-slave-2w3w krb5kdc[15823](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786277611, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.222.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260809 12:13:31.022631 17226 init.cc:377] Logged in from keytab as kudu/127.0.222.2@KRBTEST.COM (short username kudu)
I20260809 12:13:31.024230 17226 webserver.cc:460] Webserver started at http://127.0.222.2:46335/ using document root <none> and password file <none>
I20260809 12:13:31.024941 17226 fs_manager.cc:359] Metadata directory not provided
I20260809 12:13:31.025094 17226 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260809 12:13:31.030591 17226 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.002s sys 0.000s
I20260809 12:13:31.033419 17242 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260809 12:13:31.034778 17226 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.000s sys 0.002s
I20260809 12:13:31.035056 17226 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskgH5Pjm/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786277382073938-888-0/minicluster-data/ts-1/data,/tmp/dist-test-taskgH5Pjm/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786277382073938-888-0/minicluster-data/ts-1/wal
uuid: "5c1e1fc7cc4b4f59b9ca7298b6929b5c"
format_stamp: "Formatted at 2026-08-09 12:13:24 on dist-test-slave-2w3w"
server_key: "659ef2a30fb36e2536ff973fe9e8e5b4"
server_key_iv: "ca75656d9f5532d4adc03ec74619056d"
server_key_version: "encryptionkey@0"
I20260809 12:13:31.036136 17226 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskgH5Pjm/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786277382073938-888-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskgH5Pjm/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786277382073938-888-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskgH5Pjm/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786277382073938-888-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260809 12:13:31.061002 17226 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260809 12:13:31.065234 17226 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260809 12:13:31.065528 17226 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260809 12:13:31.066568 17226 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260809 12:13:31.068027 17226 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260809 12:13:31.068161 17226 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260809 12:13:31.068282 17226 ts_tablet_manager.cc:610] Registered 0 tablets
I20260809 12:13:31.068383 17226 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260809 12:13:31.103163 17226 rpc_server.cc:307] RPC server started. Bound to: 127.0.222.2:41313
I20260809 12:13:31.103250 17358 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.222.2:41313 every 8 connection(s)
I20260809 12:13:31.104643 17226 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskgH5Pjm/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786277382073938-888-0/minicluster-data/ts-1/data/info.pb
I20260809 12:13:31.109520 17362 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 12:13:31.107672 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.222.62:44147 (local address 127.0.222.2:45837)
0809 12:13:31.108426 (+ 754us) negotiation.cc:107] Waiting for socket to connect
0809 12:13:31.108655 (+ 229us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.222.62:44147: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":506,"thread_start_us":306,"threads_started":1}
W20260809 12:13:31.110227 17359 heartbeater.cc:644] Failed to heartbeat to 127.0.222.62:44147 (0 consecutive failures): Network error: Failed to ping master at 127.0.222.62:44147: Client connection negotiation failed: client connection to 127.0.222.62:44147: connect: Connection refused (error 111)
I20260809 12:13:31.111119 17362 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 12:13:31.110745 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.222.62:44147 (local address 127.0.222.2:56795)
0809 12:13:31.110899 (+ 154us) negotiation.cc:107] Waiting for socket to connect
0809 12:13:31.111013 (+ 114us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.222.62:44147: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":52}
I20260809 12:13:31.112016 17362 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 12:13:31.111691 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.222.62:44147 (local address 127.0.222.2:36297)
0809 12:13:31.111813 (+ 122us) negotiation.cc:107] Waiting for socket to connect
0809 12:13:31.111907 (+ 94us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.222.62:44147: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":48}
W20260809 12:13:31.112267 17359 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
I20260809 12:13:31.112514 888 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskgH5Pjm/build/asan/bin/kudu as pid 17226
I20260809 12:13:31.113189 888 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskgH5Pjm/build/asan/bin/kudu
/tmp/dist-test-taskgH5Pjm/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskgH5Pjm/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786277382073938-888-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskgH5Pjm/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786277382073938-888-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskgH5Pjm/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786277382073938-888-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskgH5Pjm/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786277382073938-888-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.222.3:46283
--local_ip_for_outbound_sockets=127.0.222.3
--tserver_master_addrs=127.0.222.62:44147
--webserver_port=36801
--webserver_interface=127.0.222.3
--keytab_file=/tmp/dist-test-taskgH5Pjm/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786277382073938-888-0/krb5kdc/kudu_127.0.222.3.keytab
--principal=kudu/127.0.222.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.222.20:41797
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskgH5Pjm/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786277382073938-888-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskgH5Pjm/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786277382073938-888-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskgH5Pjm/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786277382073938-888-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260809 12:13:31.455710 17363 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260809 12:13:31.456015 17363 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260809 12:13:31.456110 17363 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260809 12:13:31.463562 17363 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260809 12:13:31.463709 17363 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260809 12:13:31.463876 17363 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.222.3
I20260809 12:13:31.473830 17363 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.222.20:41797
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskgH5Pjm/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786277382073938-888-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskgH5Pjm/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786277382073938-888-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskgH5Pjm/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786277382073938-888-0/krb5kdc/kudu_127.0.222.3.keytab
--principal=kudu/127.0.222.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.222.3:46283
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskgH5Pjm/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786277382073938-888-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.222.3
--webserver_port=36801
--tserver_master_addrs=127.0.222.62:44147
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.222.3
--log_dir=/tmp/dist-test-taskgH5Pjm/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786277382073938-888-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 09 Aug 2026 11:06:41 UTC on ccycloud.quasar-bwoogy.root.comops.site
build id 4575991
ASAN enabled
I20260809 12:13:31.475126 17363 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260809 12:13:31.476449 17363 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260809 12:13:31.490037 17368 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 12:13:31.489989 17369 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 12:13:31.490067 17371 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260809 12:13:31.490887 17363 server_base.cc:1034] running on GCE node
I20260809 12:13:31.491549 17363 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260809 12:13:31.492523 17363 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260809 12:13:31.493767 17363 hybrid_clock.cc:648] HybridClock initialized: now 1786277611493728 us; error 47 us; skew 500 ppm
I20260809 12:13:31.494136 17363 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 09 12:13:31 dist-test-slave-2w3w krb5kdc[15823](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786277611, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.222.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260809 12:13:31.497496 17363 init.cc:377] Logged in from keytab as kudu/127.0.222.3@KRBTEST.COM (short username kudu)
I20260809 12:13:31.499069 17363 webserver.cc:460] Webserver started at http://127.0.222.3:36801/ using document root <none> and password file <none>
I20260809 12:13:31.499737 17363 fs_manager.cc:359] Metadata directory not provided
I20260809 12:13:31.499855 17363 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260809 12:13:31.505237 17363 fs_manager.cc:709] Time spent opening directory manager: real 0.004s user 0.001s sys 0.004s
I20260809 12:13:31.507898 17378 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260809 12:13:31.509219 17363 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.001s
I20260809 12:13:31.509361 17363 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskgH5Pjm/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786277382073938-888-0/minicluster-data/ts-2/data,/tmp/dist-test-taskgH5Pjm/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786277382073938-888-0/minicluster-data/ts-2/wal
uuid: "1204c82bd36a468fbdf532c7aad13159"
format_stamp: "Formatted at 2026-08-09 12:13:25 on dist-test-slave-2w3w"
server_key: "69332dd618b10d8ba467b56a6ab50e51"
server_key_iv: "02d88d015dd0bb3411fe617ba0481989"
server_key_version: "encryptionkey@0"
I20260809 12:13:31.510321 17363 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskgH5Pjm/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786277382073938-888-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskgH5Pjm/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786277382073938-888-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskgH5Pjm/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786277382073938-888-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260809 12:13:31.525718 17363 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260809 12:13:31.530290 17363 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260809 12:13:31.530536 17363 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260809 12:13:31.531622 17363 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260809 12:13:31.533041 17363 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260809 12:13:31.533150 17363 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260809 12:13:31.533238 17363 ts_tablet_manager.cc:610] Registered 0 tablets
I20260809 12:13:31.533298 17363 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260809 12:13:31.567740 17363 rpc_server.cc:307] RPC server started. Bound to: 127.0.222.3:46283
I20260809 12:13:31.567835 17500 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.222.3:46283 every 8 connection(s)
I20260809 12:13:31.569235 17363 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskgH5Pjm/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786277382073938-888-0/minicluster-data/ts-2/data/info.pb
I20260809 12:13:31.573748 888 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskgH5Pjm/build/asan/bin/kudu as pid 17363
I20260809 12:13:31.574275 17504 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 12:13:31.572379 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.222.62:44147 (local address 127.0.222.3:55343)
0809 12:13:31.573143 (+ 764us) negotiation.cc:107] Waiting for socket to connect
0809 12:13:31.573368 (+ 225us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.222.62:44147: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":512,"thread_start_us":312,"threads_started":1}
W20260809 12:13:31.575073 17501 heartbeater.cc:644] Failed to heartbeat to 127.0.222.62:44147 (0 consecutive failures): Network error: Failed to ping master at 127.0.222.62:44147: Client connection negotiation failed: client connection to 127.0.222.62:44147: connect: Connection refused (error 111)
I20260809 12:13:31.575946 17504 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 12:13:31.575633 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.222.62:44147 (local address 127.0.222.3:54577)
0809 12:13:31.575773 (+ 140us) negotiation.cc:107] Waiting for socket to connect
0809 12:13:31.575840 (+ 67us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.222.62:44147: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":54}
I20260809 12:13:31.576752 17504 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 12:13:31.576427 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.222.62:44147 (local address 127.0.222.3:42049)
0809 12:13:31.576569 (+ 142us) negotiation.cc:107] Waiting for socket to connect
0809 12:13:31.576648 (+ 79us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.222.62:44147: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":57}
W20260809 12:13:31.576978 17501 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
I20260809 12:13:31.637657 17507 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 12:13:31.636845 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.222.62:44147 (local address 127.0.222.1:50207)
0809 12:13:31.637321 (+ 476us) negotiation.cc:107] Waiting for socket to connect
0809 12:13:31.637427 (+ 106us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.222.62:44147: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":360,"thread_start_us":226,"threads_started":1}
I20260809 12:13:32.113595 17508 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 12:13:32.112797 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.222.62:44147 (local address 127.0.222.2:50727)
0809 12:13:32.113251 (+ 454us) negotiation.cc:107] Waiting for socket to connect
0809 12:13:32.113337 (+ 86us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.222.62:44147: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":345,"thread_start_us":217,"threads_started":1}
I20260809 12:13:32.578466 17509 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 12:13:32.577540 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.222.62:44147 (local address 127.0.222.3:55873)
0809 12:13:32.578065 (+ 525us) negotiation.cc:107] Waiting for socket to connect
0809 12:13:32.578203 (+ 138us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.222.62:44147: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":408,"thread_start_us":279,"threads_started":1}
I20260809 12:13:32.639705 17510 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 12:13:32.638907 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.222.62:44147 (local address 127.0.222.1:34099)
0809 12:13:32.639444 (+ 537us) negotiation.cc:107] Waiting for socket to connect
0809 12:13:32.639535 (+ 91us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.222.62:44147: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":438,"thread_start_us":302,"threads_started":1}
I20260809 12:13:33.115463 17511 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 12:13:33.114538 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.222.62:44147 (local address 127.0.222.2:36299)
0809 12:13:33.115186 (+ 648us) negotiation.cc:107] Waiting for socket to connect
0809 12:13:33.115282 (+ 96us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.222.62:44147: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":550,"thread_start_us":366,"threads_started":1}
I20260809 12:13:33.580232 17512 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 12:13:33.579500 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.222.62:44147 (local address 127.0.222.3:53911)
0809 12:13:33.579971 (+ 471us) negotiation.cc:107] Waiting for socket to connect
0809 12:13:33.580057 (+ 86us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.222.62:44147: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":370,"thread_start_us":228,"threads_started":1}
I20260809 12:13:33.641451 17513 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 12:13:33.640691 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.222.62:44147 (local address 127.0.222.1:45075)
0809 12:13:33.641158 (+ 467us) negotiation.cc:107] Waiting for socket to connect
0809 12:13:33.641244 (+ 86us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.222.62:44147: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":366,"thread_start_us":240,"threads_started":1}
I20260809 12:13:34.117079 17514 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 12:13:34.116389 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.222.62:44147 (local address 127.0.222.2:34355)
0809 12:13:34.116822 (+ 433us) negotiation.cc:107] Waiting for socket to connect
0809 12:13:34.116902 (+ 80us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.222.62:44147: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":335,"thread_start_us":223,"threads_started":1}
I20260809 12:13:34.582098 17516 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 12:13:34.581252 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.222.62:44147 (local address 127.0.222.3:36925)
0809 12:13:34.581783 (+ 531us) negotiation.cc:107] Waiting for socket to connect
0809 12:13:34.581866 (+ 83us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.222.62:44147: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":420,"thread_start_us":316,"threads_started":1}
I20260809 12:13:34.643249 17517 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 12:13:34.642429 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.222.62:44147 (local address 127.0.222.1:34887)
0809 12:13:34.642906 (+ 477us) negotiation.cc:107] Waiting for socket to connect
0809 12:13:34.643047 (+ 141us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.222.62:44147: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":373,"thread_start_us":248,"threads_started":1}
I20260809 12:13:35.118734 17518 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 12:13:35.118031 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.222.62:44147 (local address 127.0.222.2:42089)
0809 12:13:35.118487 (+ 456us) negotiation.cc:107] Waiting for socket to connect
0809 12:13:35.118565 (+ 78us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.222.62:44147: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":358,"thread_start_us":241,"threads_started":1}
I20260809 12:13:35.591749 17519 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 12:13:35.583078 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.222.62:44147 (local address 127.0.222.3:49737)
0809 12:13:35.591440 (+ 8362us) negotiation.cc:107] Waiting for socket to connect
0809 12:13:35.591536 (+ 96us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.222.62:44147: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":8219,"thread_start_us":8102,"threads_started":1}
I20260809 12:13:35.644972 17520 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 12:13:35.644226 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.222.62:44147 (local address 127.0.222.1:51885)
0809 12:13:35.644696 (+ 470us) negotiation.cc:107] Waiting for socket to connect
0809 12:13:35.644778 (+ 82us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.222.62:44147: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":375,"thread_start_us":243,"threads_started":1}
I20260809 12:13:36.120448 17521 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 12:13:36.119679 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.222.62:44147 (local address 127.0.222.2:35393)
0809 12:13:36.120155 (+ 476us) negotiation.cc:107] Waiting for socket to connect
0809 12:13:36.120235 (+ 80us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.222.62:44147: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":375,"thread_start_us":235,"threads_started":1}
I20260809 12:13:36.593545 17522 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 12:13:36.592735 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.222.62:44147 (local address 127.0.222.3:34569)
0809 12:13:36.593274 (+ 539us) negotiation.cc:107] Waiting for socket to connect
0809 12:13:36.593366 (+ 92us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.222.62:44147: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":397,"thread_start_us":256,"threads_started":1}
I20260809 12:13:36.646570 17523 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 12:13:36.645843 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.222.62:44147 (local address 127.0.222.1:60867)
0809 12:13:36.646302 (+ 459us) negotiation.cc:107] Waiting for socket to connect
0809 12:13:36.646404 (+ 102us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.222.62:44147: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":364,"thread_start_us":249,"threads_started":1}
I20260809 12:13:37.122340 17524 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 12:13:37.121379 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.222.62:44147 (local address 127.0.222.2:48465)
0809 12:13:37.122060 (+ 681us) negotiation.cc:107] Waiting for socket to connect
0809 12:13:37.122170 (+ 110us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.222.62:44147: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":434,"thread_start_us":311,"threads_started":1}
I20260809 12:13:37.595472 17525 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 12:13:37.594583 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.222.62:44147 (local address 127.0.222.3:51945)
0809 12:13:37.595143 (+ 560us) negotiation.cc:107] Waiting for socket to connect
0809 12:13:37.595244 (+ 101us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.222.62:44147: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":453,"thread_start_us":273,"threads_started":1}
I20260809 12:13:37.648324 17526 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 12:13:37.647505 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.222.62:44147 (local address 127.0.222.1:50031)
0809 12:13:37.648018 (+ 513us) negotiation.cc:107] Waiting for socket to connect
0809 12:13:37.648100 (+ 82us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.222.62:44147: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":410,"thread_start_us":229,"threads_started":1}
I20260809 12:13:38.124219 17527 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 12:13:38.123374 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.222.62:44147 (local address 127.0.222.2:39489)
0809 12:13:38.123921 (+ 547us) negotiation.cc:107] Waiting for socket to connect
0809 12:13:38.124003 (+ 82us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.222.62:44147: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":424,"thread_start_us":231,"threads_started":1}
I20260809 12:13:38.597337 17529 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 12:13:38.596510 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.222.62:44147 (local address 127.0.222.3:41077)
0809 12:13:38.597019 (+ 509us) negotiation.cc:107] Waiting for socket to connect
0809 12:13:38.597108 (+ 89us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.222.62:44147: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":389,"thread_start_us":220,"threads_started":1}
I20260809 12:13:38.650051 17530 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 12:13:38.649273 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.222.62:44147 (local address 127.0.222.1:35659)
0809 12:13:38.649788 (+ 515us) negotiation.cc:107] Waiting for socket to connect
0809 12:13:38.649882 (+ 94us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.222.62:44147: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":414,"thread_start_us":276,"threads_started":1}
I20260809 12:13:39.130488 17531 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 12:13:39.129424 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.222.62:44147 (local address 127.0.222.2:52839)
0809 12:13:39.130042 (+ 618us) negotiation.cc:107] Waiting for socket to connect
0809 12:13:39.130193 (+ 151us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.222.62:44147: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":446,"thread_start_us":802,"threads_started":1}
I20260809 12:13:39.599399 17532 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 12:13:39.598531 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.222.62:44147 (local address 127.0.222.3:44963)
0809 12:13:39.599092 (+ 561us) negotiation.cc:107] Waiting for socket to connect
0809 12:13:39.599194 (+ 102us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.222.62:44147: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":391,"thread_start_us":260,"threads_started":1}
I20260809 12:13:39.651973 17534 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 12:13:39.651066 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.222.62:44147 (local address 127.0.222.1:35071)
0809 12:13:39.651685 (+ 619us) negotiation.cc:107] Waiting for socket to connect
0809 12:13:39.651785 (+ 100us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.222.62:44147: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":470,"thread_start_us":323,"threads_started":1}
I20260809 12:13:40.132395 17535 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 12:13:40.131490 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.222.62:44147 (local address 127.0.222.2:45607)
0809 12:13:40.132081 (+ 591us) negotiation.cc:107] Waiting for socket to connect
0809 12:13:40.132208 (+ 127us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.222.62:44147: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":453,"thread_start_us":261,"threads_started":1}
I20260809 12:13:40.601114 17536 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 12:13:40.600360 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.222.62:44147 (local address 127.0.222.3:50641)
0809 12:13:40.600850 (+ 490us) negotiation.cc:107] Waiting for socket to connect
0809 12:13:40.600935 (+ 85us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.222.62:44147: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":388,"thread_start_us":250,"threads_started":1}
I20260809 12:13:40.653666 17537 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 12:13:40.652949 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.222.62:44147 (local address 127.0.222.1:34387)
0809 12:13:40.653417 (+ 468us) negotiation.cc:107] Waiting for socket to connect
0809 12:13:40.653502 (+ 85us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.222.62:44147: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":371,"thread_start_us":249,"threads_started":1}
I20260809 12:13:41.134126 17538 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 12:13:41.133391 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.222.62:44147 (local address 127.0.222.2:37997)
0809 12:13:41.133863 (+ 472us) negotiation.cc:107] Waiting for socket to connect
0809 12:13:41.133946 (+ 83us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.222.62:44147: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":378,"thread_start_us":265,"threads_started":1}
I20260809 12:13:41.602885 17539 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 12:13:41.602150 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.222.62:44147 (local address 127.0.222.3:42601)
0809 12:13:41.602631 (+ 481us) negotiation.cc:107] Waiting for socket to connect
0809 12:13:41.602716 (+ 85us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.222.62:44147: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":385,"thread_start_us":278,"threads_started":1}
I20260809 12:13:41.655385 17540 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 12:13:41.654554 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.222.62:44147 (local address 127.0.222.1:39487)
0809 12:13:41.655077 (+ 523us) negotiation.cc:107] Waiting for socket to connect
0809 12:13:41.655174 (+ 97us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.222.62:44147: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":372,"thread_start_us":234,"threads_started":1}
I20260809 12:13:42.135991 17541 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 12:13:42.135185 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.222.62:44147 (local address 127.0.222.2:33759)
0809 12:13:42.135661 (+ 476us) negotiation.cc:107] Waiting for socket to connect
0809 12:13:42.135764 (+ 103us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.222.62:44147: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":374,"thread_start_us":238,"threads_started":1}
I20260809 12:13:42.604612 17542 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 12:13:42.603852 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.222.62:44147 (local address 127.0.222.3:49347)
0809 12:13:42.604333 (+ 481us) negotiation.cc:107] Waiting for socket to connect
0809 12:13:42.604428 (+ 95us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.222.62:44147: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":372,"thread_start_us":248,"threads_started":1}
I20260809 12:13:42.657065 17543 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 12:13:42.656383 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.222.62:44147 (local address 127.0.222.1:59133)
0809 12:13:42.656826 (+ 443us) negotiation.cc:107] Waiting for socket to connect
0809 12:13:42.656903 (+ 77us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.222.62:44147: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":339,"thread_start_us":229,"threads_started":1}
I20260809 12:13:43.137670 17544 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 12:13:43.136951 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.222.62:44147 (local address 127.0.222.2:55511)
0809 12:13:43.137423 (+ 472us) negotiation.cc:107] Waiting for socket to connect
0809 12:13:43.137505 (+ 82us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.222.62:44147: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":378,"thread_start_us":242,"threads_started":1}
I20260809 12:13:43.606374 17546 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 12:13:43.605665 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.222.62:44147 (local address 127.0.222.3:58799)
0809 12:13:43.606106 (+ 441us) negotiation.cc:107] Waiting for socket to connect
0809 12:13:43.606190 (+ 84us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.222.62:44147: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":335,"thread_start_us":227,"threads_started":1}
I20260809 12:13:43.658737 17547 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 12:13:43.657984 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.222.62:44147 (local address 127.0.222.1:56703)
0809 12:13:43.658442 (+ 458us) negotiation.cc:107] Waiting for socket to connect
0809 12:13:43.658546 (+ 104us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.222.62:44147: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":367,"thread_start_us":227,"threads_started":1}
I20260809 12:13:44.139624 17548 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 12:13:44.138668 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.222.62:44147 (local address 127.0.222.2:50937)
0809 12:13:44.139292 (+ 624us) negotiation.cc:107] Waiting for socket to connect
0809 12:13:44.139427 (+ 135us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.222.62:44147: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":473,"thread_start_us":333,"threads_started":1}
I20260809 12:13:44.608225 17549 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 12:13:44.607375 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.222.62:44147 (local address 127.0.222.3:58805)
0809 12:13:44.607923 (+ 548us) negotiation.cc:107] Waiting for socket to connect
0809 12:13:44.608022 (+ 99us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.222.62:44147: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":447,"thread_start_us":262,"threads_started":1}
I20260809 12:13:44.660454 17550 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 12:13:44.659720 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.222.62:44147 (local address 127.0.222.1:48209)
0809 12:13:44.660184 (+ 464us) negotiation.cc:107] Waiting for socket to connect
0809 12:13:44.660268 (+ 84us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.222.62:44147: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":361,"thread_start_us":230,"threads_started":1}
I20260809 12:13:45.141286 17551 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 12:13:45.140474 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.222.62:44147 (local address 127.0.222.2:47141)
0809 12:13:45.141020 (+ 546us) negotiation.cc:107] Waiting for socket to connect
0809 12:13:45.141099 (+ 79us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.222.62:44147: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":451,"thread_start_us":333,"threads_started":1}
I20260809 12:13:45.609895 17552 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 12:13:45.609159 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.222.62:44147 (local address 127.0.222.3:41971)
0809 12:13:45.609618 (+ 459us) negotiation.cc:107] Waiting for socket to connect
0809 12:13:45.609702 (+ 84us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.222.62:44147: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":357,"thread_start_us":236,"threads_started":1}
I20260809 12:13:45.662436 17553 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 12:13:45.661547 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.222.62:44147 (local address 127.0.222.1:59845)
0809 12:13:45.662092 (+ 545us) negotiation.cc:107] Waiting for socket to connect
0809 12:13:45.662181 (+ 89us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.222.62:44147: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":424,"thread_start_us":324,"threads_started":1}
I20260809 12:13:46.143206 17554 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 12:13:46.142320 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.222.62:44147 (local address 127.0.222.2:47101)
0809 12:13:46.142842 (+ 522us) negotiation.cc:107] Waiting for socket to connect
0809 12:13:46.142982 (+ 140us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.222.62:44147: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":391,"thread_start_us":264,"threads_started":1}
I20260809 12:13:46.583065 888 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskgH5Pjm/build/asan/bin/kudu pid 17046
I20260809 12:13:46.601575 17555 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 12:13:46.584169 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50856 (local address 127.0.222.1:40557)
0809 12:13:46.584736 (+ 567us) server_negotiation.cc:206] Beginning negotiation
0809 12:13:46.584752 (+ 16us) server_negotiation.cc:399] Waiting for connection header
0809 12:13:46.584794 (+ 42us) server_negotiation.cc:407] Connection header received
0809 12:13:46.586625 (+ 1831us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 12:13:46.586639 (+ 14us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 12:13:46.587135 (+ 496us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 12:13:46.587436 (+ 301us) server_negotiation.cc:226] Negotiated authn=SASL
0809 12:13:46.588861 (+ 1425us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 12:13:46.590166 (+ 1305us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 12:13:46.591639 (+ 1473us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 12:13:46.592358 (+ 719us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 12:13:46.592503 (+ 145us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 12:13:46.595512 (+ 3009us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 12:13:46.595521 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 12:13:46.595537 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0809 12:13:46.595583 (+ 46us) server_negotiation.cc:922] calling sasl_server_start()
0809 12:13:46.598493 (+ 2910us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 12:13:46.599249 (+ 756us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 12:13:46.599256 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 12:13:46.599263 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0809 12:13:46.599440 (+ 177us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 12:13:46.599801 (+ 361us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 12:13:46.599804 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 12:13:46.599806 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0809 12:13:46.600344 (+ 538us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 12:13:46.600593 (+ 249us) server_negotiation.cc:1035] Waiting for connection context
0809 12:13:46.600953 (+ 360us) server_negotiation.cc:299] Negotiation successful
0809 12:13:46.601287 (+ 334us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":388,"thread_start_us":266,"threads_started":1}
I20260809 12:13:46.603137 17218 generic_service.cc:196] Checking for leaks (request via RPC)
I20260809 12:13:46.612141 17557 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 12:13:46.611282 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.222.62:44147 (local address 127.0.222.3:39121)
0809 12:13:46.611838 (+ 556us) negotiation.cc:107] Waiting for socket to connect
0809 12:13:46.611947 (+ 109us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.222.62:44147: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":422,"thread_start_us":311,"threads_started":1}
I20260809 12:13:46.717684 888 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskgH5Pjm/build/asan/bin/kudu with pid 17046
I20260809 12:13:46.741274 888 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskgH5Pjm/build/asan/bin/kudu pid 17226
I20260809 12:13:46.759958 17560 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 12:13:46.742308 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45584 (local address 127.0.222.2:41313)
0809 12:13:46.742840 (+ 532us) server_negotiation.cc:206] Beginning negotiation
0809 12:13:46.742854 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0809 12:13:46.742891 (+ 37us) server_negotiation.cc:407] Connection header received
0809 12:13:46.744581 (+ 1690us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 12:13:46.744590 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 12:13:46.745008 (+ 418us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 12:13:46.745363 (+ 355us) server_negotiation.cc:226] Negotiated authn=SASL
0809 12:13:46.746878 (+ 1515us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 12:13:46.748388 (+ 1510us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 12:13:46.749707 (+ 1319us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 12:13:46.750405 (+ 698us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 12:13:46.750604 (+ 199us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 12:13:46.753488 (+ 2884us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 12:13:46.753496 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 12:13:46.753511 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0809 12:13:46.753552 (+ 41us) server_negotiation.cc:922] calling sasl_server_start()
0809 12:13:46.756450 (+ 2898us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 12:13:46.757193 (+ 743us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 12:13:46.757204 (+ 11us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 12:13:46.757216 (+ 12us) server_negotiation.cc:967] calling sasl_server_step()
0809 12:13:46.757380 (+ 164us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 12:13:46.757816 (+ 436us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 12:13:46.757821 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 12:13:46.757824 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0809 12:13:46.758330 (+ 506us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 12:13:46.758652 (+ 322us) server_negotiation.cc:1035] Waiting for connection context
0809 12:13:46.759273 (+ 621us) server_negotiation.cc:299] Negotiation successful
0809 12:13:46.759656 (+ 383us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":387,"thread_start_us":252,"threads_started":1}
I20260809 12:13:46.761418 17354 generic_service.cc:196] Checking for leaks (request via RPC)
I20260809 12:13:46.847815 888 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskgH5Pjm/build/asan/bin/kudu with pid 17226
I20260809 12:13:46.869191 888 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskgH5Pjm/build/asan/bin/kudu pid 17363
I20260809 12:13:46.886734 17562 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 12:13:46.870234 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34402 (local address 127.0.222.3:46283)
0809 12:13:46.870845 (+ 611us) server_negotiation.cc:206] Beginning negotiation
0809 12:13:46.870859 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0809 12:13:46.870903 (+ 44us) server_negotiation.cc:407] Connection header received
0809 12:13:46.872712 (+ 1809us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 12:13:46.872723 (+ 11us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 12:13:46.873162 (+ 439us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 12:13:46.873452 (+ 290us) server_negotiation.cc:226] Negotiated authn=SASL
0809 12:13:46.874794 (+ 1342us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 12:13:46.876123 (+ 1329us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 12:13:46.877311 (+ 1188us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 12:13:46.877944 (+ 633us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 12:13:46.878088 (+ 144us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 12:13:46.880677 (+ 2589us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 12:13:46.880685 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 12:13:46.880700 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0809 12:13:46.880741 (+ 41us) server_negotiation.cc:922] calling sasl_server_start()
0809 12:13:46.883681 (+ 2940us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 12:13:46.884372 (+ 691us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 12:13:46.884378 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 12:13:46.884384 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0809 12:13:46.884552 (+ 168us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 12:13:46.884878 (+ 326us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 12:13:46.884882 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 12:13:46.884884 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0809 12:13:46.885383 (+ 499us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 12:13:46.885601 (+ 218us) server_negotiation.cc:1035] Waiting for connection context
0809 12:13:46.886000 (+ 399us) server_negotiation.cc:299] Negotiation successful
0809 12:13:46.886424 (+ 424us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":449,"thread_start_us":308,"threads_started":1}
I20260809 12:13:46.888212 17496 generic_service.cc:196] Checking for leaks (request via RPC)
I20260809 12:13:46.973186 888 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskgH5Pjm/build/asan/bin/kudu with pid 17363
2026-08-09T12:13:46Z chronyd exiting
[ OK ] SecurityITest.TestEnableAndDisableEncryptedIPKICert (24986 ms)
[ RUN ] SecurityITest.TestEnableAndDisableEncryptedTSK
Loading random data
Initializing database '/tmp/dist-test-taskgH5Pjm/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786277382073938-888-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 09 12:13:47 dist-test-slave-2w3w krb5kdc[17567](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 09 12:13:47 dist-test-slave-2w3w krb5kdc[17567](info): set up 2 sockets
Aug 09 12:13:47 dist-test-slave-2w3w krb5kdc[17567](info): commencing operation
krb5kdc: starting...
W20260809 12:13:48.163017 888 mini_kdc.cc:121] Time spent starting KDC: real 1.144s user 0.009s sys 0.032s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskgH5Pjm/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786277382073938-888-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskgH5Pjm/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786277382073938-888-0/krb5kdc/test-user.keytab.
Aug 09 12:13:48 dist-test-slave-2w3w krb5kdc[17567](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786277628, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-09T12:13:48Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-09T12:13:48Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.222.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.222.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.222.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskgH5Pjm/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786277382073938-888-0/krb5kdc/kudu_127.0.222.62.keytab.
Entry for principal kudu/127.0.222.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskgH5Pjm/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786277382073938-888-0/krb5kdc/kudu_127.0.222.62.keytab.
I20260809 12:13:48.463234 888 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskgH5Pjm/build/asan/bin/kudu
/tmp/dist-test-taskgH5Pjm/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskgH5Pjm/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786277382073938-888-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskgH5Pjm/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786277382073938-888-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskgH5Pjm/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786277382073938-888-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskgH5Pjm/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786277382073938-888-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.222.62:45505
--webserver_interface=127.0.222.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskgH5Pjm/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786277382073938-888-0/krb5kdc/kudu_127.0.222.62.keytab
--principal=kudu/127.0.222.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.222.20:41481
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.222.62:45505
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--authn_token_validity_seconds=20
--authz_token_validity_seconds=20
--tsk_rotation_seconds=20 with env {KRB5CCNAME=/tmp/dist-test-taskgH5Pjm/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786277382073938-888-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskgH5Pjm/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786277382073938-888-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskgH5Pjm/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786277382073938-888-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260809 12:13:48.811290 17582 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260809 12:13:48.811609 17582 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260809 12:13:48.811774 17582 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260809 12:13:48.819660 17582 flags.cc:425] Enabled experimental flag: --authn_token_validity_seconds=20
W20260809 12:13:48.819761 17582 flags.cc:425] Enabled experimental flag: --authz_token_validity_seconds=20
W20260809 12:13:48.819864 17582 flags.cc:425] Enabled experimental flag: --tsk_rotation_seconds=20
W20260809 12:13:48.819943 17582 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260809 12:13:48.820025 17582 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260809 12:13:48.820132 17582 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260809 12:13:48.820225 17582 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260809 12:13:48.820330 17582 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260809 12:13:48.830214 17582 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.222.20:41481
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskgH5Pjm/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786277382073938-888-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskgH5Pjm/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786277382073938-888-0/minicluster-data/master-0/wal
--authn_token_validity_seconds=20
--authz_token_validity_seconds=20
--tsk_rotation_seconds=20
--ipki_ca_key_size=768
--master_addresses=127.0.222.62:45505
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskgH5Pjm/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786277382073938-888-0/krb5kdc/kudu_127.0.222.62.keytab
--principal=kudu/127.0.222.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.222.62:45505
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskgH5Pjm/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786277382073938-888-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.222.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskgH5Pjm/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786277382073938-888-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 09 Aug 2026 11:06:41 UTC on ccycloud.quasar-bwoogy.root.comops.site
build id 4575991
ASAN enabled
I20260809 12:13:48.831848 17582 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260809 12:13:48.833261 17582 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260809 12:13:48.845546 17587 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 12:13:48.845481 17588 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 12:13:48.845587 17590 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260809 12:13:48.846172 17582 server_base.cc:1034] running on GCE node
I20260809 12:13:48.847107 17582 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260809 12:13:48.848444 17582 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260809 12:13:48.849665 17582 hybrid_clock.cc:648] HybridClock initialized: now 1786277628849608 us; error 26 us; skew 500 ppm
I20260809 12:13:48.850005 17582 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 09 12:13:48 dist-test-slave-2w3w krb5kdc[17567](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786277628, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.222.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260809 12:13:48.858143 17582 init.cc:377] Logged in from keytab as kudu/127.0.222.62@KRBTEST.COM (short username kudu)
I20260809 12:13:48.859750 17582 webserver.cc:460] Webserver started at http://127.0.222.62:35835/ using document root <none> and password file <none>
I20260809 12:13:48.860502 17582 fs_manager.cc:359] Metadata directory not provided
I20260809 12:13:48.860641 17582 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260809 12:13:48.860982 17582 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260809 12:13:48.864161 17582 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskgH5Pjm/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786277382073938-888-0/minicluster-data/master-0/data/instance:
uuid: "976efcee2bd94ed6b5c102cc07ac7ab1"
format_stamp: "Formatted at 2026-08-09 12:13:48 on dist-test-slave-2w3w"
server_key: "5f75dd8c4f7b73a87d0db18621478cff"
server_key_iv: "fdfbac5a7eab9ac2b3d7ff0e02c44b99"
server_key_version: "encryptionkey@0"
I20260809 12:13:48.865046 17582 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskgH5Pjm/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786277382073938-888-0/minicluster-data/master-0/wal/instance:
uuid: "976efcee2bd94ed6b5c102cc07ac7ab1"
format_stamp: "Formatted at 2026-08-09 12:13:48 on dist-test-slave-2w3w"
server_key: "5f75dd8c4f7b73a87d0db18621478cff"
server_key_iv: "fdfbac5a7eab9ac2b3d7ff0e02c44b99"
server_key_version: "encryptionkey@0"
I20260809 12:13:48.870620 17582 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.000s
I20260809 12:13:48.874295 17597 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260809 12:13:48.875715 17582 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260809 12:13:48.875909 17582 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskgH5Pjm/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786277382073938-888-0/minicluster-data/master-0/data,/tmp/dist-test-taskgH5Pjm/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786277382073938-888-0/minicluster-data/master-0/wal
uuid: "976efcee2bd94ed6b5c102cc07ac7ab1"
format_stamp: "Formatted at 2026-08-09 12:13:48 on dist-test-slave-2w3w"
server_key: "5f75dd8c4f7b73a87d0db18621478cff"
server_key_iv: "fdfbac5a7eab9ac2b3d7ff0e02c44b99"
server_key_version: "encryptionkey@0"
I20260809 12:13:48.876152 17582 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskgH5Pjm/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786277382073938-888-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskgH5Pjm/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786277382073938-888-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskgH5Pjm/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786277382073938-888-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260809 12:13:48.898738 17582 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260809 12:13:48.903399 17582 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260809 12:13:48.903683 17582 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260809 12:13:48.923175 17582 rpc_server.cc:307] RPC server started. Bound to: 127.0.222.62:45505
I20260809 12:13:48.923199 17653 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.222.62:45505 every 8 connection(s)
I20260809 12:13:48.924625 17582 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskgH5Pjm/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786277382073938-888-0/minicluster-data/master-0/data/info.pb
I20260809 12:13:48.928453 17655 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260809 12:13:48.933192 888 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskgH5Pjm/build/asan/bin/kudu as pid 17582
I20260809 12:13:48.933451 888 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskgH5Pjm/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786277382073938-888-0/minicluster-data/master-0/wal/instance
I20260809 12:13:48.934226 888 external_mini_cluster.cc:1428] Setting key 755ff7a66551598257279bac0b6da6d5
I20260809 12:13:48.939673 17655 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 976efcee2bd94ed6b5c102cc07ac7ab1: Bootstrap starting.
I20260809 12:13:48.943516 17655 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 976efcee2bd94ed6b5c102cc07ac7ab1: Neither blocks nor log segments found. Creating new log.
I20260809 12:13:48.944783 17655 log.cc:824] T 00000000000000000000000000000000 P 976efcee2bd94ed6b5c102cc07ac7ab1: Log is configured to *not* fsync() on all Append() calls
I20260809 12:13:48.948419 17655 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 976efcee2bd94ed6b5c102cc07ac7ab1: No bootstrap required, opened a new log
Aug 09 12:13:48 dist-test-slave-2w3w krb5kdc[17567](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786277628, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.222.62@KRBTEST.COM
I20260809 12:13:48.956606 17655 raft_consensus.cc:348] T 00000000000000000000000000000000 P 976efcee2bd94ed6b5c102cc07ac7ab1 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "976efcee2bd94ed6b5c102cc07ac7ab1" member_type: VOTER last_known_addr { host: "127.0.222.62" port: 45505 } }
I20260809 12:13:48.957011 17655 raft_consensus.cc:374] T 00000000000000000000000000000000 P 976efcee2bd94ed6b5c102cc07ac7ab1 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260809 12:13:48.957149 17655 raft_consensus.cc:729] T 00000000000000000000000000000000 P 976efcee2bd94ed6b5c102cc07ac7ab1 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 976efcee2bd94ed6b5c102cc07ac7ab1, State: Initialized, Role: FOLLOWER
I20260809 12:13:48.957878 17655 consensus_queue.cc:260] T 00000000000000000000000000000000 P 976efcee2bd94ed6b5c102cc07ac7ab1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "976efcee2bd94ed6b5c102cc07ac7ab1" member_type: VOTER last_known_addr { host: "127.0.222.62" port: 45505 } }
I20260809 12:13:48.958173 17655 raft_consensus.cc:388] T 00000000000000000000000000000000 P 976efcee2bd94ed6b5c102cc07ac7ab1 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260809 12:13:48.958317 17655 raft_consensus.cc:482] T 00000000000000000000000000000000 P 976efcee2bd94ed6b5c102cc07ac7ab1 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260809 12:13:48.958523 17655 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 976efcee2bd94ed6b5c102cc07ac7ab1 [term 0 FOLLOWER]: Advancing to term 1
I20260809 12:13:48.959467 17658 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 12:13:48.936331 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41810 (local address 127.0.222.62:45505)
0809 12:13:48.937202 (+ 871us) server_negotiation.cc:206] Beginning negotiation
0809 12:13:48.937213 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0809 12:13:48.937259 (+ 46us) server_negotiation.cc:407] Connection header received
0809 12:13:48.938929 (+ 1670us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 12:13:48.938950 (+ 21us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 12:13:48.939408 (+ 458us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 12:13:48.939716 (+ 308us) server_negotiation.cc:226] Negotiated authn=SASL
0809 12:13:48.941684 (+ 1968us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 12:13:48.943115 (+ 1431us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 12:13:48.945100 (+ 1985us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 12:13:48.946202 (+ 1102us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 12:13:48.946364 (+ 162us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 12:13:48.951425 (+ 5061us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 12:13:48.951432 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 12:13:48.951450 (+ 18us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0809 12:13:48.951490 (+ 40us) server_negotiation.cc:922] calling sasl_server_start()
0809 12:13:48.954756 (+ 3266us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 12:13:48.955582 (+ 826us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 12:13:48.955588 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 12:13:48.955595 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0809 12:13:48.955774 (+ 179us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 12:13:48.956195 (+ 421us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 12:13:48.956201 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 12:13:48.956204 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0809 12:13:48.956736 (+ 532us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 12:13:48.957063 (+ 327us) server_negotiation.cc:1035] Waiting for connection context
0809 12:13:48.957527 (+ 464us) server_negotiation.cc:299] Negotiation successful
0809 12:13:48.957993 (+ 466us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":636,"thread_start_us":402,"threads_started":1}
I20260809 12:13:48.961791 17655 raft_consensus.cc:504] T 00000000000000000000000000000000 P 976efcee2bd94ed6b5c102cc07ac7ab1 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "976efcee2bd94ed6b5c102cc07ac7ab1" member_type: VOTER last_known_addr { host: "127.0.222.62" port: 45505 } }
I20260809 12:13:48.962392 17655 leader_election.cc:302] T 00000000000000000000000000000000 P 976efcee2bd94ed6b5c102cc07ac7ab1 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 976efcee2bd94ed6b5c102cc07ac7ab1; no voters:
I20260809 12:13:48.963153 17655 leader_election.cc:288] T 00000000000000000000000000000000 P 976efcee2bd94ed6b5c102cc07ac7ab1 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260809 12:13:48.963310 17660 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 976efcee2bd94ed6b5c102cc07ac7ab1 [term 1 FOLLOWER]: Leader election won for term 1
I20260809 12:13:48.964022 17660 raft_consensus.cc:686] T 00000000000000000000000000000000 P 976efcee2bd94ed6b5c102cc07ac7ab1 [term 1 LEADER]: Becoming Leader. State: Replica: 976efcee2bd94ed6b5c102cc07ac7ab1, State: Running, Role: LEADER
I20260809 12:13:48.964597 17660 consensus_queue.cc:237] T 00000000000000000000000000000000 P 976efcee2bd94ed6b5c102cc07ac7ab1 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "976efcee2bd94ed6b5c102cc07ac7ab1" member_type: VOTER last_known_addr { host: "127.0.222.62" port: 45505 } }
I20260809 12:13:48.965301 17655 sys_catalog.cc:564] T 00000000000000000000000000000000 P 976efcee2bd94ed6b5c102cc07ac7ab1 [sys.catalog]: configured and running, proceeding with master startup.
I20260809 12:13:48.967309 17661 sys_catalog.cc:455] T 00000000000000000000000000000000 P 976efcee2bd94ed6b5c102cc07ac7ab1 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "976efcee2bd94ed6b5c102cc07ac7ab1" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "976efcee2bd94ed6b5c102cc07ac7ab1" member_type: VOTER last_known_addr { host: "127.0.222.62" port: 45505 } } }
I20260809 12:13:48.967739 17661 sys_catalog.cc:458] T 00000000000000000000000000000000 P 976efcee2bd94ed6b5c102cc07ac7ab1 [sys.catalog]: This master's current role is: LEADER
I20260809 12:13:48.971345 17662 sys_catalog.cc:455] T 00000000000000000000000000000000 P 976efcee2bd94ed6b5c102cc07ac7ab1 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 976efcee2bd94ed6b5c102cc07ac7ab1. Latest consensus state: current_term: 1 leader_uuid: "976efcee2bd94ed6b5c102cc07ac7ab1" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "976efcee2bd94ed6b5c102cc07ac7ab1" member_type: VOTER last_known_addr { host: "127.0.222.62" port: 45505 } } }
I20260809 12:13:48.972007 17662 sys_catalog.cc:458] T 00000000000000000000000000000000 P 976efcee2bd94ed6b5c102cc07ac7ab1 [sys.catalog]: This master's current role is: LEADER
I20260809 12:13:48.972831 17666 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260809 12:13:48.977855 17666 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260809 12:13:48.986764 17666 catalog_manager.cc:1347] Generated new cluster ID: d8085938efb84ddc8c174e7e9bb18f14
I20260809 12:13:48.986917 17666 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260809 12:13:48.993909 17666 catalog_manager.cc:1370] Generated new certificate authority record
I20260809 12:13:48.995251 17666 catalog_manager.cc:1504] Loading token signing keys...
I20260809 12:13:49.010336 17666 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 976efcee2bd94ed6b5c102cc07ac7ab1: Generated new TSK 0
I20260809 12:13:49.011296 17666 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.0.222.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.222.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.222.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskgH5Pjm/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786277382073938-888-0/krb5kdc/kudu_127.0.222.1.keytab.
Entry for principal kudu/127.0.222.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskgH5Pjm/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786277382073938-888-0/krb5kdc/kudu_127.0.222.1.keytab.
I20260809 12:13:49.095475 888 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskgH5Pjm/build/asan/bin/kudu
/tmp/dist-test-taskgH5Pjm/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskgH5Pjm/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786277382073938-888-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskgH5Pjm/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786277382073938-888-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskgH5Pjm/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786277382073938-888-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskgH5Pjm/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786277382073938-888-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.222.1:0
--local_ip_for_outbound_sockets=127.0.222.1
--webserver_interface=127.0.222.1
--webserver_port=0
--tserver_master_addrs=127.0.222.62:45505
--keytab_file=/tmp/dist-test-taskgH5Pjm/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786277382073938-888-0/krb5kdc/kudu_127.0.222.1.keytab
--principal=kudu/127.0.222.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.222.20:41481
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskgH5Pjm/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786277382073938-888-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskgH5Pjm/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786277382073938-888-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskgH5Pjm/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786277382073938-888-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260809 12:13:49.438054 17681 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260809 12:13:49.438338 17681 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260809 12:13:49.438490 17681 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260809 12:13:49.446489 17681 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260809 12:13:49.446619 17681 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260809 12:13:49.446815 17681 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.222.1
I20260809 12:13:49.456629 17681 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.222.20:41481
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskgH5Pjm/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786277382073938-888-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskgH5Pjm/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786277382073938-888-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskgH5Pjm/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786277382073938-888-0/krb5kdc/kudu_127.0.222.1.keytab
--principal=kudu/127.0.222.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.222.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskgH5Pjm/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786277382073938-888-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.222.1
--webserver_port=0
--tserver_master_addrs=127.0.222.62:45505
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.222.1
--log_dir=/tmp/dist-test-taskgH5Pjm/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786277382073938-888-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 09 Aug 2026 11:06:41 UTC on ccycloud.quasar-bwoogy.root.comops.site
build id 4575991
ASAN enabled
I20260809 12:13:49.457976 17681 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260809 12:13:49.459350 17681 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260809 12:13:49.472879 17689 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 12:13:49.472980 17686 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 12:13:49.473502 17687 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260809 12:13:49.473783 17681 server_base.cc:1034] running on GCE node
I20260809 12:13:49.474417 17681 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260809 12:13:49.475468 17681 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260809 12:13:49.476686 17681 hybrid_clock.cc:648] HybridClock initialized: now 1786277629476634 us; error 50 us; skew 500 ppm
I20260809 12:13:49.476999 17681 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 09 12:13:49 dist-test-slave-2w3w krb5kdc[17567](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786277629, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.222.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260809 12:13:49.480652 17681 init.cc:377] Logged in from keytab as kudu/127.0.222.1@KRBTEST.COM (short username kudu)
I20260809 12:13:49.482208 17681 webserver.cc:460] Webserver started at http://127.0.222.1:38689/ using document root <none> and password file <none>
I20260809 12:13:49.482898 17681 fs_manager.cc:359] Metadata directory not provided
I20260809 12:13:49.483067 17681 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260809 12:13:49.483385 17681 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260809 12:13:49.486248 17681 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskgH5Pjm/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786277382073938-888-0/minicluster-data/ts-0/data/instance:
uuid: "bd233f0a216e42e7858d2eb736a785bb"
format_stamp: "Formatted at 2026-08-09 12:13:49 on dist-test-slave-2w3w"
server_key: "9f4432a335e7b85d34e07fd2f3696829"
server_key_iv: "1d7e63d4b85cc01a747aca61fa912fc3"
server_key_version: "encryptionkey@0"
I20260809 12:13:49.487052 17681 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskgH5Pjm/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786277382073938-888-0/minicluster-data/ts-0/wal/instance:
uuid: "bd233f0a216e42e7858d2eb736a785bb"
format_stamp: "Formatted at 2026-08-09 12:13:49 on dist-test-slave-2w3w"
server_key: "9f4432a335e7b85d34e07fd2f3696829"
server_key_iv: "1d7e63d4b85cc01a747aca61fa912fc3"
server_key_version: "encryptionkey@0"
I20260809 12:13:49.492157 17681 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.003s sys 0.001s
I20260809 12:13:49.495625 17697 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260809 12:13:49.496932 17681 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260809 12:13:49.497110 17681 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskgH5Pjm/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786277382073938-888-0/minicluster-data/ts-0/data,/tmp/dist-test-taskgH5Pjm/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786277382073938-888-0/minicluster-data/ts-0/wal
uuid: "bd233f0a216e42e7858d2eb736a785bb"
format_stamp: "Formatted at 2026-08-09 12:13:49 on dist-test-slave-2w3w"
server_key: "9f4432a335e7b85d34e07fd2f3696829"
server_key_iv: "1d7e63d4b85cc01a747aca61fa912fc3"
server_key_version: "encryptionkey@0"
I20260809 12:13:49.497321 17681 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskgH5Pjm/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786277382073938-888-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskgH5Pjm/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786277382073938-888-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskgH5Pjm/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786277382073938-888-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260809 12:13:49.510713 17681 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260809 12:13:49.515296 17681 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260809 12:13:49.515596 17681 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260809 12:13:49.516554 17681 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260809 12:13:49.517864 17681 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260809 12:13:49.517943 17681 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260809 12:13:49.518124 17681 ts_tablet_manager.cc:610] Registered 0 tablets
I20260809 12:13:49.518187 17681 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260809 12:13:49.552626 17681 rpc_server.cc:307] RPC server started. Bound to: 127.0.222.1:41049
I20260809 12:13:49.552716 17816 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.222.1:41049 every 8 connection(s)
I20260809 12:13:49.553956 17681 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskgH5Pjm/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786277382073938-888-0/minicluster-data/ts-0/data/info.pb
I20260809 12:13:49.556399 888 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskgH5Pjm/build/asan/bin/kudu as pid 17681
I20260809 12:13:49.556629 888 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskgH5Pjm/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786277382073938-888-0/minicluster-data/ts-0/wal/instance
I20260809 12:13:49.557456 888 external_mini_cluster.cc:1428] Setting key b56e18891fcd92771eca55f8d9434203
Aug 09 12:13:49 dist-test-slave-2w3w krb5kdc[17567](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786277629, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.222.1@KRBTEST.COM for kudu/127.0.222.62@KRBTEST.COM
I20260809 12:13:49.576248 17819 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 12:13:49.556547 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.222.1:55921 (local address 127.0.222.62:45505)
0809 12:13:49.557095 (+ 548us) server_negotiation.cc:206] Beginning negotiation
0809 12:13:49.557103 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0809 12:13:49.558097 (+ 994us) server_negotiation.cc:407] Connection header received
0809 12:13:49.559977 (+ 1880us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 12:13:49.559991 (+ 14us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 12:13:49.560090 (+ 99us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 12:13:49.560214 (+ 124us) server_negotiation.cc:226] Negotiated authn=SASL
0809 12:13:49.561648 (+ 1434us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 12:13:49.562812 (+ 1164us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 12:13:49.564379 (+ 1567us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 12:13:49.565110 (+ 731us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 12:13:49.565259 (+ 149us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 12:13:49.568893 (+ 3634us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 12:13:49.568903 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 12:13:49.568910 (+ 7us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0809 12:13:49.568956 (+ 46us) server_negotiation.cc:922] calling sasl_server_start()
0809 12:13:49.572512 (+ 3556us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 12:13:49.573547 (+ 1035us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 12:13:49.573551 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 12:13:49.573554 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0809 12:13:49.573673 (+ 119us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 12:13:49.574150 (+ 477us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 12:13:49.574158 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 12:13:49.574163 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0809 12:13:49.574553 (+ 390us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 12:13:49.574711 (+ 158us) server_negotiation.cc:1035] Waiting for connection context
0809 12:13:49.575497 (+ 786us) server_negotiation.cc:299] Negotiation successful
0809 12:13:49.575772 (+ 275us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":391,"thread_start_us":261,"threads_started":1}
I20260809 12:13:49.576833 17820 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 12:13:49.557075 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.222.62:45505 (local address 127.0.222.1:55921)
0809 12:13:49.557888 (+ 813us) negotiation.cc:107] Waiting for socket to connect
0809 12:13:49.557951 (+ 63us) client_negotiation.cc:174] Beginning negotiation
0809 12:13:49.559720 (+ 1769us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 12:13:49.560423 (+ 703us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 12:13:49.560431 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 12:13:49.560942 (+ 511us) client_negotiation.cc:189] Negotiated authn=SASL
0809 12:13:49.561472 (+ 530us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 12:13:49.561488 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 12:13:49.563025 (+ 1537us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 12:13:49.563029 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 12:13:49.564232 (+ 1203us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 12:13:49.564244 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 12:13:49.565359 (+ 1115us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 12:13:49.565365 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 12:13:49.565638 (+ 273us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 12:13:49.566388 (+ 750us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0809 12:13:49.566413 (+ 25us) client_negotiation.cc:656] calling sasl_client_start()
0809 12:13:49.568672 (+ 2259us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0809 12:13:49.572689 (+ 4017us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0809 12:13:49.572699 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0809 12:13:49.572718 (+ 19us) client_negotiation.cc:760] calling sasl_client_step()
0809 12:13:49.573426 (+ 708us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0809 12:13:49.573797 (+ 371us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0809 12:13:49.573800 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0809 12:13:49.573803 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0809 12:13:49.574022 (+ 219us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0809 12:13:49.574674 (+ 652us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0809 12:13:49.574680 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0809 12:13:49.575081 (+ 401us) client_negotiation.cc:769] Sending connection context
0809 12:13:49.575307 (+ 226us) client_negotiation.cc:240] Negotiation successful
0809 12:13:49.575696 (+ 389us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":553,"thread_start_us":358,"threads_started":1}
I20260809 12:13:49.578513 17817 heartbeater.cc:347] Connected to a master server at 127.0.222.62:45505
I20260809 12:13:49.578835 17817 heartbeater.cc:464] Registering TS with master...
I20260809 12:13:49.579605 17817 heartbeater.cc:511] Master 127.0.222.62:45505 requested a full tablet report, sending...
I20260809 12:13:49.581653 17615 ts_manager.cc:194] Registered new tserver with Master: bd233f0a216e42e7858d2eb736a785bb (127.0.222.1:41049)
I20260809 12:13:49.583170 17615 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.222.1@KRBTEST.COM'} at 127.0.222.1:55921
WARNING: no policy specified for kudu/127.0.222.2@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.222.2@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.222.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskgH5Pjm/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786277382073938-888-0/krb5kdc/kudu_127.0.222.2.keytab.
Entry for principal kudu/127.0.222.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskgH5Pjm/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786277382073938-888-0/krb5kdc/kudu_127.0.222.2.keytab.
I20260809 12:13:49.626350 888 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskgH5Pjm/build/asan/bin/kudu
/tmp/dist-test-taskgH5Pjm/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskgH5Pjm/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786277382073938-888-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskgH5Pjm/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786277382073938-888-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskgH5Pjm/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786277382073938-888-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskgH5Pjm/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786277382073938-888-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.222.2:0
--local_ip_for_outbound_sockets=127.0.222.2
--webserver_interface=127.0.222.2
--webserver_port=0
--tserver_master_addrs=127.0.222.62:45505
--keytab_file=/tmp/dist-test-taskgH5Pjm/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786277382073938-888-0/krb5kdc/kudu_127.0.222.2.keytab
--principal=kudu/127.0.222.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.222.20:41481
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskgH5Pjm/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786277382073938-888-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskgH5Pjm/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786277382073938-888-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskgH5Pjm/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786277382073938-888-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260809 12:13:49.959573 17823 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260809 12:13:49.959859 17823 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260809 12:13:49.960022 17823 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260809 12:13:49.967322 17823 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260809 12:13:49.967423 17823 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260809 12:13:49.967629 17823 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.222.2
I20260809 12:13:49.977556 17823 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.222.20:41481
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskgH5Pjm/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786277382073938-888-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskgH5Pjm/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786277382073938-888-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskgH5Pjm/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786277382073938-888-0/krb5kdc/kudu_127.0.222.2.keytab
--principal=kudu/127.0.222.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.222.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskgH5Pjm/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786277382073938-888-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.222.2
--webserver_port=0
--tserver_master_addrs=127.0.222.62:45505
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.222.2
--log_dir=/tmp/dist-test-taskgH5Pjm/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786277382073938-888-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 09 Aug 2026 11:06:41 UTC on ccycloud.quasar-bwoogy.root.comops.site
build id 4575991
ASAN enabled
I20260809 12:13:49.979008 17823 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260809 12:13:49.980360 17823 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260809 12:13:49.993773 17829 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 12:13:49.993721 17831 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 12:13:49.993811 17828 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260809 12:13:49.994860 17823 server_base.cc:1034] running on GCE node
I20260809 12:13:49.995498 17823 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260809 12:13:49.996456 17823 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260809 12:13:49.997689 17823 hybrid_clock.cc:648] HybridClock initialized: now 1786277629997635 us; error 49 us; skew 500 ppm
I20260809 12:13:49.998010 17823 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 09 12:13:49 dist-test-slave-2w3w krb5kdc[17567](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786277629, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.222.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260809 12:13:50.001309 17823 init.cc:377] Logged in from keytab as kudu/127.0.222.2@KRBTEST.COM (short username kudu)
I20260809 12:13:50.002846 17823 webserver.cc:460] Webserver started at http://127.0.222.2:37553/ using document root <none> and password file <none>
I20260809 12:13:50.003563 17823 fs_manager.cc:359] Metadata directory not provided
I20260809 12:13:50.003670 17823 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260809 12:13:50.003945 17823 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260809 12:13:50.006829 17823 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskgH5Pjm/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786277382073938-888-0/minicluster-data/ts-1/data/instance:
uuid: "a83ae6a077a34a80985b4b65dfa7b6f6"
format_stamp: "Formatted at 2026-08-09 12:13:50 on dist-test-slave-2w3w"
server_key: "df2da0431ccce241d64ece7296ce9d25"
server_key_iv: "90e49b1f6082ddd27cd5c5b866344a61"
server_key_version: "encryptionkey@0"
I20260809 12:13:50.007611 17823 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskgH5Pjm/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786277382073938-888-0/minicluster-data/ts-1/wal/instance:
uuid: "a83ae6a077a34a80985b4b65dfa7b6f6"
format_stamp: "Formatted at 2026-08-09 12:13:50 on dist-test-slave-2w3w"
server_key: "df2da0431ccce241d64ece7296ce9d25"
server_key_iv: "90e49b1f6082ddd27cd5c5b866344a61"
server_key_version: "encryptionkey@0"
I20260809 12:13:50.012712 17823 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.004s sys 0.000s
I20260809 12:13:50.016067 17838 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260809 12:13:50.017427 17823 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260809 12:13:50.017560 17823 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskgH5Pjm/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786277382073938-888-0/minicluster-data/ts-1/data,/tmp/dist-test-taskgH5Pjm/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786277382073938-888-0/minicluster-data/ts-1/wal
uuid: "a83ae6a077a34a80985b4b65dfa7b6f6"
format_stamp: "Formatted at 2026-08-09 12:13:50 on dist-test-slave-2w3w"
server_key: "df2da0431ccce241d64ece7296ce9d25"
server_key_iv: "90e49b1f6082ddd27cd5c5b866344a61"
server_key_version: "encryptionkey@0"
I20260809 12:13:50.017720 17823 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskgH5Pjm/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786277382073938-888-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskgH5Pjm/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786277382073938-888-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskgH5Pjm/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786277382073938-888-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260809 12:13:50.037532 17823 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260809 12:13:50.041884 17823 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260809 12:13:50.042184 17823 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260809 12:13:50.043244 17823 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260809 12:13:50.044659 17823 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260809 12:13:50.044734 17823 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260809 12:13:50.044909 17823 ts_tablet_manager.cc:610] Registered 0 tablets
I20260809 12:13:50.044971 17823 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260809 12:13:50.079622 17823 rpc_server.cc:307] RPC server started. Bound to: 127.0.222.2:35933
I20260809 12:13:50.079718 17956 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.222.2:35933 every 8 connection(s)
I20260809 12:13:50.080935 17823 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskgH5Pjm/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786277382073938-888-0/minicluster-data/ts-1/data/info.pb
I20260809 12:13:50.085914 888 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskgH5Pjm/build/asan/bin/kudu as pid 17823
I20260809 12:13:50.086117 888 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskgH5Pjm/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786277382073938-888-0/minicluster-data/ts-1/wal/instance
I20260809 12:13:50.086719 888 external_mini_cluster.cc:1428] Setting key f5078a6936e6c86bfc64e458bce4b70f
Aug 09 12:13:50 dist-test-slave-2w3w krb5kdc[17567](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786277629, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.222.2@KRBTEST.COM for kudu/127.0.222.62@KRBTEST.COM
I20260809 12:13:50.103690 17959 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 12:13:50.083453 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.222.2:49873 (local address 127.0.222.62:45505)
0809 12:13:50.083922 (+ 469us) server_negotiation.cc:206] Beginning negotiation
0809 12:13:50.083926 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0809 12:13:50.084754 (+ 828us) server_negotiation.cc:407] Connection header received
0809 12:13:50.087296 (+ 2542us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 12:13:50.087301 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 12:13:50.087424 (+ 123us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 12:13:50.087553 (+ 129us) server_negotiation.cc:226] Negotiated authn=SASL
0809 12:13:50.089396 (+ 1843us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 12:13:50.090153 (+ 757us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 12:13:50.092704 (+ 2551us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 12:13:50.093173 (+ 469us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 12:13:50.093273 (+ 100us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 12:13:50.097567 (+ 4294us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 12:13:50.097573 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 12:13:50.097576 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0809 12:13:50.097610 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0809 12:13:50.100328 (+ 2718us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 12:13:50.101166 (+ 838us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 12:13:50.101178 (+ 12us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 12:13:50.101181 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0809 12:13:50.101317 (+ 136us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 12:13:50.101774 (+ 457us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 12:13:50.101781 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 12:13:50.101788 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0809 12:13:50.102123 (+ 335us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 12:13:50.102300 (+ 177us) server_negotiation.cc:1035] Waiting for connection context
0809 12:13:50.103040 (+ 740us) server_negotiation.cc:299] Negotiation successful
0809 12:13:50.103292 (+ 252us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":371,"thread_start_us":256,"threads_started":1}
I20260809 12:13:50.104673 17960 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 12:13:50.083870 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.222.62:45505 (local address 127.0.222.2:49873)
0809 12:13:50.084590 (+ 720us) negotiation.cc:107] Waiting for socket to connect
0809 12:13:50.084627 (+ 37us) client_negotiation.cc:174] Beginning negotiation
0809 12:13:50.086923 (+ 2296us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 12:13:50.087842 (+ 919us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 12:13:50.087852 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 12:13:50.088394 (+ 542us) client_negotiation.cc:189] Negotiated authn=SASL
0809 12:13:50.089207 (+ 813us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 12:13:50.089235 (+ 28us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 12:13:50.090327 (+ 1092us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 12:13:50.090335 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 12:13:50.092534 (+ 2199us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 12:13:50.092552 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 12:13:50.093412 (+ 860us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 12:13:50.093421 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 12:13:50.093740 (+ 319us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 12:13:50.094513 (+ 773us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0809 12:13:50.094555 (+ 42us) client_negotiation.cc:656] calling sasl_client_start()
0809 12:13:50.097322 (+ 2767us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0809 12:13:50.100478 (+ 3156us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0809 12:13:50.100487 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0809 12:13:50.100503 (+ 16us) client_negotiation.cc:760] calling sasl_client_step()
0809 12:13:50.101036 (+ 533us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0809 12:13:50.101441 (+ 405us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0809 12:13:50.101446 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0809 12:13:50.101450 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0809 12:13:50.101648 (+ 198us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0809 12:13:50.102262 (+ 614us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0809 12:13:50.102271 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0809 12:13:50.102632 (+ 361us) client_negotiation.cc:769] Sending connection context
0809 12:13:50.102919 (+ 287us) client_negotiation.cc:240] Negotiation successful
0809 12:13:50.103503 (+ 584us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":492,"thread_start_us":306,"threads_started":1}
I20260809 12:13:50.106289 17957 heartbeater.cc:347] Connected to a master server at 127.0.222.62:45505
I20260809 12:13:50.106654 17957 heartbeater.cc:464] Registering TS with master...
I20260809 12:13:50.107599 17957 heartbeater.cc:511] Master 127.0.222.62:45505 requested a full tablet report, sending...
I20260809 12:13:50.109324 17615 ts_manager.cc:194] Registered new tserver with Master: a83ae6a077a34a80985b4b65dfa7b6f6 (127.0.222.2:35933)
I20260809 12:13:50.110661 17615 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.222.2@KRBTEST.COM'} at 127.0.222.2:49873
WARNING: no policy specified for kudu/127.0.222.3@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.222.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.222.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskgH5Pjm/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786277382073938-888-0/krb5kdc/kudu_127.0.222.3.keytab.
Entry for principal kudu/127.0.222.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskgH5Pjm/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786277382073938-888-0/krb5kdc/kudu_127.0.222.3.keytab.
I20260809 12:13:50.158429 888 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskgH5Pjm/build/asan/bin/kudu
/tmp/dist-test-taskgH5Pjm/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskgH5Pjm/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786277382073938-888-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskgH5Pjm/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786277382073938-888-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskgH5Pjm/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786277382073938-888-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskgH5Pjm/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786277382073938-888-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.222.3:0
--local_ip_for_outbound_sockets=127.0.222.3
--webserver_interface=127.0.222.3
--webserver_port=0
--tserver_master_addrs=127.0.222.62:45505
--keytab_file=/tmp/dist-test-taskgH5Pjm/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786277382073938-888-0/krb5kdc/kudu_127.0.222.3.keytab
--principal=kudu/127.0.222.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.222.20:41481
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskgH5Pjm/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786277382073938-888-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskgH5Pjm/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786277382073938-888-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskgH5Pjm/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786277382073938-888-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260809 12:13:50.501133 17963 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260809 12:13:50.501459 17963 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260809 12:13:50.501643 17963 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260809 12:13:50.508857 17963 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260809 12:13:50.508963 17963 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260809 12:13:50.509162 17963 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.222.3
I20260809 12:13:50.519158 17963 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.222.20:41481
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskgH5Pjm/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786277382073938-888-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskgH5Pjm/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786277382073938-888-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskgH5Pjm/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786277382073938-888-0/krb5kdc/kudu_127.0.222.3.keytab
--principal=kudu/127.0.222.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.222.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskgH5Pjm/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786277382073938-888-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.222.3
--webserver_port=0
--tserver_master_addrs=127.0.222.62:45505
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.222.3
--log_dir=/tmp/dist-test-taskgH5Pjm/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786277382073938-888-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 09 Aug 2026 11:06:41 UTC on ccycloud.quasar-bwoogy.root.comops.site
build id 4575991
ASAN enabled
I20260809 12:13:50.520496 17963 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260809 12:13:50.521854 17963 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260809 12:13:50.535156 17968 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 12:13:50.535096 17971 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 12:13:50.535181 17969 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260809 12:13:50.535980 17963 server_base.cc:1034] running on GCE node
I20260809 12:13:50.536628 17963 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260809 12:13:50.537587 17963 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260809 12:13:50.538854 17963 hybrid_clock.cc:648] HybridClock initialized: now 1786277630538787 us; error 31 us; skew 500 ppm
I20260809 12:13:50.539300 17963 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 09 12:13:50 dist-test-slave-2w3w krb5kdc[17567](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786277630, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.222.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260809 12:13:50.542466 17963 init.cc:377] Logged in from keytab as kudu/127.0.222.3@KRBTEST.COM (short username kudu)
I20260809 12:13:50.543979 17963 webserver.cc:460] Webserver started at http://127.0.222.3:39089/ using document root <none> and password file <none>
I20260809 12:13:50.544692 17963 fs_manager.cc:359] Metadata directory not provided
I20260809 12:13:50.544847 17963 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260809 12:13:50.545198 17963 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260809 12:13:50.548179 17963 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskgH5Pjm/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786277382073938-888-0/minicluster-data/ts-2/data/instance:
uuid: "b97494c2f09f448e85a9497d47d415f6"
format_stamp: "Formatted at 2026-08-09 12:13:50 on dist-test-slave-2w3w"
server_key: "2fb234b39084d1c2b523aaaa022a2ee0"
server_key_iv: "1ee45beb051160c66da10ad87b022316"
server_key_version: "encryptionkey@0"
I20260809 12:13:50.548978 17963 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskgH5Pjm/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786277382073938-888-0/minicluster-data/ts-2/wal/instance:
uuid: "b97494c2f09f448e85a9497d47d415f6"
format_stamp: "Formatted at 2026-08-09 12:13:50 on dist-test-slave-2w3w"
server_key: "2fb234b39084d1c2b523aaaa022a2ee0"
server_key_iv: "1ee45beb051160c66da10ad87b022316"
server_key_version: "encryptionkey@0"
I20260809 12:13:50.554553 17963 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.001s
I20260809 12:13:50.558151 17978 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260809 12:13:50.559507 17963 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.000s sys 0.003s
I20260809 12:13:50.559686 17963 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskgH5Pjm/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786277382073938-888-0/minicluster-data/ts-2/data,/tmp/dist-test-taskgH5Pjm/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786277382073938-888-0/minicluster-data/ts-2/wal
uuid: "b97494c2f09f448e85a9497d47d415f6"
format_stamp: "Formatted at 2026-08-09 12:13:50 on dist-test-slave-2w3w"
server_key: "2fb234b39084d1c2b523aaaa022a2ee0"
server_key_iv: "1ee45beb051160c66da10ad87b022316"
server_key_version: "encryptionkey@0"
I20260809 12:13:50.559897 17963 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskgH5Pjm/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786277382073938-888-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskgH5Pjm/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786277382073938-888-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskgH5Pjm/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786277382073938-888-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260809 12:13:50.572108 17963 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260809 12:13:50.576393 17963 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260809 12:13:50.576695 17963 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260809 12:13:50.577749 17963 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260809 12:13:50.579257 17963 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260809 12:13:50.579363 17963 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260809 12:13:50.579521 17963 ts_tablet_manager.cc:610] Registered 0 tablets
I20260809 12:13:50.579602 17963 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260809 12:13:50.586086 17817 heartbeater.cc:503] Master 127.0.222.62:45505 was elected leader, sending a full tablet report...
I20260809 12:13:50.615226 17963 rpc_server.cc:307] RPC server started. Bound to: 127.0.222.3:43523
I20260809 12:13:50.615272 18099 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.222.3:43523 every 8 connection(s)
I20260809 12:13:50.616559 17963 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskgH5Pjm/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786277382073938-888-0/minicluster-data/ts-2/data/info.pb
I20260809 12:13:50.618690 888 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskgH5Pjm/build/asan/bin/kudu as pid 17963
I20260809 12:13:50.618875 888 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskgH5Pjm/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786277382073938-888-0/minicluster-data/ts-2/wal/instance
I20260809 12:13:50.619550 888 external_mini_cluster.cc:1428] Setting key 05981e99baaefbe89f098080280004ca
Aug 09 12:13:50 dist-test-slave-2w3w krb5kdc[17567](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786277630, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.222.3@KRBTEST.COM for kudu/127.0.222.62@KRBTEST.COM
I20260809 12:13:50.637235 18102 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 12:13:50.619200 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.222.3:57955 (local address 127.0.222.62:45505)
0809 12:13:50.619716 (+ 516us) server_negotiation.cc:206] Beginning negotiation
0809 12:13:50.619723 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0809 12:13:50.620779 (+ 1056us) server_negotiation.cc:407] Connection header received
0809 12:13:50.622884 (+ 2105us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 12:13:50.622888 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 12:13:50.623108 (+ 220us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 12:13:50.623224 (+ 116us) server_negotiation.cc:226] Negotiated authn=SASL
0809 12:13:50.624691 (+ 1467us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 12:13:50.625442 (+ 751us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 12:13:50.626956 (+ 1514us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 12:13:50.627420 (+ 464us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 12:13:50.627524 (+ 104us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 12:13:50.630809 (+ 3285us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 12:13:50.630813 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 12:13:50.630817 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0809 12:13:50.630849 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0809 12:13:50.633687 (+ 2838us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 12:13:50.634655 (+ 968us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 12:13:50.634663 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 12:13:50.634669 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0809 12:13:50.634817 (+ 148us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 12:13:50.635429 (+ 612us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 12:13:50.635435 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 12:13:50.635438 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0809 12:13:50.635724 (+ 286us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 12:13:50.635901 (+ 177us) server_negotiation.cc:1035] Waiting for connection context
0809 12:13:50.636626 (+ 725us) server_negotiation.cc:299] Negotiation successful
0809 12:13:50.636854 (+ 228us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":386,"thread_start_us":240,"threads_started":1}
I20260809 12:13:50.638161 18103 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 12:13:50.619773 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.222.62:45505 (local address 127.0.222.3:57955)
0809 12:13:50.620612 (+ 839us) negotiation.cc:107] Waiting for socket to connect
0809 12:13:50.620664 (+ 52us) client_negotiation.cc:174] Beginning negotiation
0809 12:13:50.622632 (+ 1968us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 12:13:50.623438 (+ 806us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 12:13:50.623447 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 12:13:50.623993 (+ 546us) client_negotiation.cc:189] Negotiated authn=SASL
0809 12:13:50.624543 (+ 550us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 12:13:50.624560 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 12:13:50.625583 (+ 1023us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 12:13:50.625587 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 12:13:50.626824 (+ 1237us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 12:13:50.626843 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 12:13:50.627612 (+ 769us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 12:13:50.627617 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 12:13:50.627838 (+ 221us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 12:13:50.628452 (+ 614us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0809 12:13:50.628475 (+ 23us) client_negotiation.cc:656] calling sasl_client_start()
0809 12:13:50.630609 (+ 2134us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0809 12:13:50.633897 (+ 3288us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0809 12:13:50.633906 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0809 12:13:50.633922 (+ 16us) client_negotiation.cc:760] calling sasl_client_step()
0809 12:13:50.634552 (+ 630us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0809 12:13:50.635030 (+ 478us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0809 12:13:50.635036 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0809 12:13:50.635040 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0809 12:13:50.635299 (+ 259us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0809 12:13:50.635881 (+ 582us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0809 12:13:50.635888 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0809 12:13:50.636248 (+ 360us) client_negotiation.cc:769] Sending connection context
0809 12:13:50.636524 (+ 276us) client_negotiation.cc:240] Negotiation successful
0809 12:13:50.637051 (+ 527us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":568,"thread_start_us":357,"threads_started":1}
I20260809 12:13:50.639451 18100 heartbeater.cc:347] Connected to a master server at 127.0.222.62:45505
I20260809 12:13:50.639775 18100 heartbeater.cc:464] Registering TS with master...
I20260809 12:13:50.640504 18100 heartbeater.cc:511] Master 127.0.222.62:45505 requested a full tablet report, sending...
I20260809 12:13:50.641903 17615 ts_manager.cc:194] Registered new tserver with Master: b97494c2f09f448e85a9497d47d415f6 (127.0.222.3:43523)
I20260809 12:13:50.642726 17615 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.222.3@KRBTEST.COM'} at 127.0.222.3:57955
I20260809 12:13:50.647446 888 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260809 12:13:50.664992 18102 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 12:13:50.652295 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41812 (local address 127.0.222.62:45505)
0809 12:13:50.652490 (+ 195us) server_negotiation.cc:206] Beginning negotiation
0809 12:13:50.652497 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0809 12:13:50.652900 (+ 403us) server_negotiation.cc:407] Connection header received
0809 12:13:50.653115 (+ 215us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 12:13:50.653121 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 12:13:50.653195 (+ 74us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 12:13:50.653298 (+ 103us) server_negotiation.cc:226] Negotiated authn=SASL
0809 12:13:50.654684 (+ 1386us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 12:13:50.655510 (+ 826us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 12:13:50.656822 (+ 1312us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 12:13:50.657227 (+ 405us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 12:13:50.657357 (+ 130us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 12:13:50.659934 (+ 2577us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 12:13:50.659940 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 12:13:50.659944 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0809 12:13:50.659982 (+ 38us) server_negotiation.cc:922] calling sasl_server_start()
0809 12:13:50.662287 (+ 2305us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 12:13:50.663017 (+ 730us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 12:13:50.663022 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 12:13:50.663026 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0809 12:13:50.663147 (+ 121us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 12:13:50.663524 (+ 377us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 12:13:50.663530 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 12:13:50.663533 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0809 12:13:50.663856 (+ 323us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 12:13:50.664111 (+ 255us) server_negotiation.cc:1035] Waiting for connection context
0809 12:13:50.664445 (+ 334us) server_negotiation.cc:299] Negotiation successful
0809 12:13:50.664674 (+ 229us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":62}
I20260809 12:13:50.669193 17615 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:41812:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260809 12:13:50.676364 17615 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260809 12:13:50.701895 18114 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 12:13:50.689155 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.222.3:43523 (local address 127.0.0.1:54026)
0809 12:13:50.692503 (+ 3348us) negotiation.cc:107] Waiting for socket to connect
0809 12:13:50.692516 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0809 12:13:50.692667 (+ 151us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 12:13:50.693367 (+ 700us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 12:13:50.693381 (+ 14us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 12:13:50.693421 (+ 40us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0809 12:13:50.693954 (+ 533us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 12:13:50.693972 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 12:13:50.696261 (+ 2289us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 12:13:50.696267 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 12:13:50.698626 (+ 2359us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 12:13:50.698639 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 12:13:50.700742 (+ 2103us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 12:13:50.700748 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 12:13:50.700976 (+ 228us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 12:13:50.701045 (+ 69us) client_negotiation.cc:769] Sending connection context
0809 12:13:50.701196 (+ 151us) client_negotiation.cc:240] Negotiation successful
0809 12:13:50.701416 (+ 220us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1243,"mutex_wait_us":3,"thread_start_us":1213,"threads_started":1}
I20260809 12:13:50.702035 18115 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 12:13:50.689391 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54026 (local address 127.0.222.3:43523)
0809 12:13:50.692369 (+ 2978us) server_negotiation.cc:206] Beginning negotiation
0809 12:13:50.692375 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0809 12:13:50.692621 (+ 246us) server_negotiation.cc:407] Connection header received
0809 12:13:50.692857 (+ 236us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 12:13:50.692867 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 12:13:50.693109 (+ 242us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 12:13:50.694076 (+ 967us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0809 12:13:50.694428 (+ 352us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 12:13:50.696096 (+ 1668us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 12:13:50.698810 (+ 2714us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 12:13:50.700189 (+ 1379us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 12:13:50.700322 (+ 133us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 12:13:50.700580 (+ 258us) server_negotiation.cc:1035] Waiting for connection context
0809 12:13:50.701266 (+ 686us) server_negotiation.cc:299] Negotiation successful
0809 12:13:50.701457 (+ 191us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":2820,"thread_start_us":2671,"threads_started":1}
I20260809 12:13:50.702601 18119 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 12:13:50.690817 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49042 (local address 127.0.222.2:35933)
0809 12:13:50.693161 (+ 2344us) server_negotiation.cc:206] Beginning negotiation
0809 12:13:50.693169 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0809 12:13:50.694019 (+ 850us) server_negotiation.cc:407] Connection header received
0809 12:13:50.694137 (+ 118us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 12:13:50.694145 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 12:13:50.694358 (+ 213us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 12:13:50.694560 (+ 202us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0809 12:13:50.695178 (+ 618us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 12:13:50.696365 (+ 1187us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 12:13:50.699495 (+ 3130us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 12:13:50.700969 (+ 1474us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 12:13:50.701122 (+ 153us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 12:13:50.701330 (+ 208us) server_negotiation.cc:1035] Waiting for connection context
0809 12:13:50.702035 (+ 705us) server_negotiation.cc:299] Negotiation successful
0809 12:13:50.702234 (+ 199us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":2190,"thread_start_us":2098,"threads_started":1}
I20260809 12:13:50.702840 18116 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 12:13:50.690359 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.222.2:35933 (local address 127.0.0.1:49042)
0809 12:13:50.693731 (+ 3372us) negotiation.cc:107] Waiting for socket to connect
0809 12:13:50.693750 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0809 12:13:50.693890 (+ 140us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 12:13:50.694556 (+ 666us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 12:13:50.694560 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 12:13:50.694584 (+ 24us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0809 12:13:50.694960 (+ 376us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 12:13:50.695022 (+ 62us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 12:13:50.696563 (+ 1541us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 12:13:50.696567 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 12:13:50.699319 (+ 2752us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 12:13:50.699335 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 12:13:50.701218 (+ 1883us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 12:13:50.701225 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 12:13:50.701422 (+ 197us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 12:13:50.701456 (+ 34us) client_negotiation.cc:769] Sending connection context
0809 12:13:50.701549 (+ 93us) client_negotiation.cc:240] Negotiation successful
0809 12:13:50.701690 (+ 141us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":3245,"mutex_wait_us":188,"thread_start_us":1236,"threads_started":1}
I20260809 12:13:50.704166 18118 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 12:13:50.689511 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.222.1:41049 (local address 127.0.0.1:60294)
0809 12:13:50.692110 (+ 2599us) negotiation.cc:107] Waiting for socket to connect
0809 12:13:50.692153 (+ 43us) client_negotiation.cc:174] Beginning negotiation
0809 12:13:50.692368 (+ 215us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 12:13:50.693553 (+ 1185us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 12:13:50.693561 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 12:13:50.693591 (+ 30us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0809 12:13:50.694121 (+ 530us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 12:13:50.694135 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 12:13:50.696261 (+ 2126us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 12:13:50.696267 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 12:13:50.699319 (+ 3052us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 12:13:50.699335 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 12:13:50.703274 (+ 3939us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 12:13:50.703281 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 12:13:50.703444 (+ 163us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 12:13:50.703474 (+ 30us) client_negotiation.cc:769] Sending connection context
0809 12:13:50.703629 (+ 155us) client_negotiation.cc:240] Negotiation successful
0809 12:13:50.703778 (+ 149us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":2316,"mutex_wait_us":186,"thread_start_us":3887,"threads_started":1}
I20260809 12:13:50.704221 18117 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 12:13:50.690246 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60294 (local address 127.0.222.1:41049)
0809 12:13:50.691483 (+ 1237us) server_negotiation.cc:206] Beginning negotiation
0809 12:13:50.691493 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0809 12:13:50.692588 (+ 1095us) server_negotiation.cc:407] Connection header received
0809 12:13:50.692712 (+ 124us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 12:13:50.692723 (+ 11us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 12:13:50.693014 (+ 291us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 12:13:50.693221 (+ 207us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0809 12:13:50.694314 (+ 1093us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 12:13:50.696096 (+ 1782us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 12:13:50.701439 (+ 5343us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 12:13:50.703043 (+ 1604us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 12:13:50.703188 (+ 145us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 12:13:50.703463 (+ 275us) server_negotiation.cc:1035] Waiting for connection context
0809 12:13:50.703717 (+ 254us) server_negotiation.cc:299] Negotiation successful
0809 12:13:50.703889 (+ 172us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1032,"thread_start_us":888,"threads_started":1}
I20260809 12:13:50.706727 18028 tablet_service.cc:1467] Processing CreateTablet for tablet c56b5b890310479c9ea3b7ed3d43e63c (DEFAULT_TABLE table=test-table [id=0684479eb30f4e14bb2db8d41c3af017]), partition=RANGE (key) PARTITION UNBOUNDED
I20260809 12:13:50.706938 17888 tablet_service.cc:1467] Processing CreateTablet for tablet c56b5b890310479c9ea3b7ed3d43e63c (DEFAULT_TABLE table=test-table [id=0684479eb30f4e14bb2db8d41c3af017]), partition=RANGE (key) PARTITION UNBOUNDED
I20260809 12:13:50.708230 18028 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c56b5b890310479c9ea3b7ed3d43e63c. 1 dirs total, 0 dirs full, 0 dirs failed
I20260809 12:13:50.708362 17888 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c56b5b890310479c9ea3b7ed3d43e63c. 1 dirs total, 0 dirs full, 0 dirs failed
I20260809 12:13:50.708252 17747 tablet_service.cc:1467] Processing CreateTablet for tablet c56b5b890310479c9ea3b7ed3d43e63c (DEFAULT_TABLE table=test-table [id=0684479eb30f4e14bb2db8d41c3af017]), partition=RANGE (key) PARTITION UNBOUNDED
I20260809 12:13:50.709645 17747 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c56b5b890310479c9ea3b7ed3d43e63c. 1 dirs total, 0 dirs full, 0 dirs failed
I20260809 12:13:50.720207 18120 tablet_bootstrap.cc:492] T c56b5b890310479c9ea3b7ed3d43e63c P b97494c2f09f448e85a9497d47d415f6: Bootstrap starting.
I20260809 12:13:50.723282 18121 tablet_bootstrap.cc:492] T c56b5b890310479c9ea3b7ed3d43e63c P a83ae6a077a34a80985b4b65dfa7b6f6: Bootstrap starting.
I20260809 12:13:50.723971 18120 tablet_bootstrap.cc:654] T c56b5b890310479c9ea3b7ed3d43e63c P b97494c2f09f448e85a9497d47d415f6: Neither blocks nor log segments found. Creating new log.
I20260809 12:13:50.724958 18122 tablet_bootstrap.cc:492] T c56b5b890310479c9ea3b7ed3d43e63c P bd233f0a216e42e7858d2eb736a785bb: Bootstrap starting.
I20260809 12:13:50.726356 18120 log.cc:824] T c56b5b890310479c9ea3b7ed3d43e63c P b97494c2f09f448e85a9497d47d415f6: Log is configured to *not* fsync() on all Append() calls
I20260809 12:13:50.727321 18121 tablet_bootstrap.cc:654] T c56b5b890310479c9ea3b7ed3d43e63c P a83ae6a077a34a80985b4b65dfa7b6f6: Neither blocks nor log segments found. Creating new log.
I20260809 12:13:50.728235 18122 tablet_bootstrap.cc:654] T c56b5b890310479c9ea3b7ed3d43e63c P bd233f0a216e42e7858d2eb736a785bb: Neither blocks nor log segments found. Creating new log.
I20260809 12:13:50.728471 18121 log.cc:824] T c56b5b890310479c9ea3b7ed3d43e63c P a83ae6a077a34a80985b4b65dfa7b6f6: Log is configured to *not* fsync() on all Append() calls
I20260809 12:13:50.729489 18122 log.cc:824] T c56b5b890310479c9ea3b7ed3d43e63c P bd233f0a216e42e7858d2eb736a785bb: Log is configured to *not* fsync() on all Append() calls
I20260809 12:13:50.730067 18120 tablet_bootstrap.cc:492] T c56b5b890310479c9ea3b7ed3d43e63c P b97494c2f09f448e85a9497d47d415f6: No bootstrap required, opened a new log
I20260809 12:13:50.730501 18120 ts_tablet_manager.cc:1397] T c56b5b890310479c9ea3b7ed3d43e63c P b97494c2f09f448e85a9497d47d415f6: Time spent bootstrapping tablet: real 0.010s user 0.004s sys 0.004s
I20260809 12:13:50.732102 18122 tablet_bootstrap.cc:492] T c56b5b890310479c9ea3b7ed3d43e63c P bd233f0a216e42e7858d2eb736a785bb: No bootstrap required, opened a new log
I20260809 12:13:50.732394 18122 ts_tablet_manager.cc:1397] T c56b5b890310479c9ea3b7ed3d43e63c P bd233f0a216e42e7858d2eb736a785bb: Time spent bootstrapping tablet: real 0.008s user 0.006s sys 0.000s
I20260809 12:13:50.734867 18121 tablet_bootstrap.cc:492] T c56b5b890310479c9ea3b7ed3d43e63c P a83ae6a077a34a80985b4b65dfa7b6f6: No bootstrap required, opened a new log
I20260809 12:13:50.735245 18121 ts_tablet_manager.cc:1397] T c56b5b890310479c9ea3b7ed3d43e63c P a83ae6a077a34a80985b4b65dfa7b6f6: Time spent bootstrapping tablet: real 0.012s user 0.009s sys 0.002s
I20260809 12:13:50.740046 18122 raft_consensus.cc:348] T c56b5b890310479c9ea3b7ed3d43e63c P bd233f0a216e42e7858d2eb736a785bb [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b97494c2f09f448e85a9497d47d415f6" member_type: VOTER last_known_addr { host: "127.0.222.3" port: 43523 } } peers { permanent_uuid: "bd233f0a216e42e7858d2eb736a785bb" member_type: VOTER last_known_addr { host: "127.0.222.1" port: 41049 } } peers { permanent_uuid: "a83ae6a077a34a80985b4b65dfa7b6f6" member_type: VOTER last_known_addr { host: "127.0.222.2" port: 35933 } }
I20260809 12:13:50.740432 18122 raft_consensus.cc:374] T c56b5b890310479c9ea3b7ed3d43e63c P bd233f0a216e42e7858d2eb736a785bb [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260809 12:13:50.740546 18122 raft_consensus.cc:729] T c56b5b890310479c9ea3b7ed3d43e63c P bd233f0a216e42e7858d2eb736a785bb [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: bd233f0a216e42e7858d2eb736a785bb, State: Initialized, Role: FOLLOWER
I20260809 12:13:50.741186 18122 consensus_queue.cc:260] T c56b5b890310479c9ea3b7ed3d43e63c P bd233f0a216e42e7858d2eb736a785bb [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b97494c2f09f448e85a9497d47d415f6" member_type: VOTER last_known_addr { host: "127.0.222.3" port: 43523 } } peers { permanent_uuid: "bd233f0a216e42e7858d2eb736a785bb" member_type: VOTER last_known_addr { host: "127.0.222.1" port: 41049 } } peers { permanent_uuid: "a83ae6a077a34a80985b4b65dfa7b6f6" member_type: VOTER last_known_addr { host: "127.0.222.2" port: 35933 } }
I20260809 12:13:50.741997 18120 raft_consensus.cc:348] T c56b5b890310479c9ea3b7ed3d43e63c P b97494c2f09f448e85a9497d47d415f6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b97494c2f09f448e85a9497d47d415f6" member_type: VOTER last_known_addr { host: "127.0.222.3" port: 43523 } } peers { permanent_uuid: "bd233f0a216e42e7858d2eb736a785bb" member_type: VOTER last_known_addr { host: "127.0.222.1" port: 41049 } } peers { permanent_uuid: "a83ae6a077a34a80985b4b65dfa7b6f6" member_type: VOTER last_known_addr { host: "127.0.222.2" port: 35933 } }
I20260809 12:13:50.742506 18120 raft_consensus.cc:374] T c56b5b890310479c9ea3b7ed3d43e63c P b97494c2f09f448e85a9497d47d415f6 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260809 12:13:50.742652 18120 raft_consensus.cc:729] T c56b5b890310479c9ea3b7ed3d43e63c P b97494c2f09f448e85a9497d47d415f6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b97494c2f09f448e85a9497d47d415f6, State: Initialized, Role: FOLLOWER
I20260809 12:13:50.743057 18122 ts_tablet_manager.cc:1428] T c56b5b890310479c9ea3b7ed3d43e63c P bd233f0a216e42e7858d2eb736a785bb: Time spent starting tablet: real 0.010s user 0.009s sys 0.001s
I20260809 12:13:50.743337 18120 consensus_queue.cc:260] T c56b5b890310479c9ea3b7ed3d43e63c P b97494c2f09f448e85a9497d47d415f6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b97494c2f09f448e85a9497d47d415f6" member_type: VOTER last_known_addr { host: "127.0.222.3" port: 43523 } } peers { permanent_uuid: "bd233f0a216e42e7858d2eb736a785bb" member_type: VOTER last_known_addr { host: "127.0.222.1" port: 41049 } } peers { permanent_uuid: "a83ae6a077a34a80985b4b65dfa7b6f6" member_type: VOTER last_known_addr { host: "127.0.222.2" port: 35933 } }
I20260809 12:13:50.744472 18100 heartbeater.cc:503] Master 127.0.222.62:45505 was elected leader, sending a full tablet report...
I20260809 12:13:50.745097 18120 ts_tablet_manager.cc:1428] T c56b5b890310479c9ea3b7ed3d43e63c P b97494c2f09f448e85a9497d47d415f6: Time spent starting tablet: real 0.014s user 0.013s sys 0.000s
I20260809 12:13:50.747080 18121 raft_consensus.cc:348] T c56b5b890310479c9ea3b7ed3d43e63c P a83ae6a077a34a80985b4b65dfa7b6f6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b97494c2f09f448e85a9497d47d415f6" member_type: VOTER last_known_addr { host: "127.0.222.3" port: 43523 } } peers { permanent_uuid: "bd233f0a216e42e7858d2eb736a785bb" member_type: VOTER last_known_addr { host: "127.0.222.1" port: 41049 } } peers { permanent_uuid: "a83ae6a077a34a80985b4b65dfa7b6f6" member_type: VOTER last_known_addr { host: "127.0.222.2" port: 35933 } }
I20260809 12:13:50.747653 18121 raft_consensus.cc:374] T c56b5b890310479c9ea3b7ed3d43e63c P a83ae6a077a34a80985b4b65dfa7b6f6 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260809 12:13:50.747771 18121 raft_consensus.cc:729] T c56b5b890310479c9ea3b7ed3d43e63c P a83ae6a077a34a80985b4b65dfa7b6f6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a83ae6a077a34a80985b4b65dfa7b6f6, State: Initialized, Role: FOLLOWER
I20260809 12:13:50.748355 18121 consensus_queue.cc:260] T c56b5b890310479c9ea3b7ed3d43e63c P a83ae6a077a34a80985b4b65dfa7b6f6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b97494c2f09f448e85a9497d47d415f6" member_type: VOTER last_known_addr { host: "127.0.222.3" port: 43523 } } peers { permanent_uuid: "bd233f0a216e42e7858d2eb736a785bb" member_type: VOTER last_known_addr { host: "127.0.222.1" port: 41049 } } peers { permanent_uuid: "a83ae6a077a34a80985b4b65dfa7b6f6" member_type: VOTER last_known_addr { host: "127.0.222.2" port: 35933 } }
I20260809 12:13:50.749639 17957 heartbeater.cc:503] Master 127.0.222.62:45505 was elected leader, sending a full tablet report...
I20260809 12:13:50.750252 18121 ts_tablet_manager.cc:1428] T c56b5b890310479c9ea3b7ed3d43e63c P a83ae6a077a34a80985b4b65dfa7b6f6: Time spent starting tablet: real 0.015s user 0.011s sys 0.002s
I20260809 12:13:50.797282 18127 raft_consensus.cc:482] T c56b5b890310479c9ea3b7ed3d43e63c P b97494c2f09f448e85a9497d47d415f6 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260809 12:13:50.797616 18127 raft_consensus.cc:504] T c56b5b890310479c9ea3b7ed3d43e63c P b97494c2f09f448e85a9497d47d415f6 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b97494c2f09f448e85a9497d47d415f6" member_type: VOTER last_known_addr { host: "127.0.222.3" port: 43523 } } peers { permanent_uuid: "bd233f0a216e42e7858d2eb736a785bb" member_type: VOTER last_known_addr { host: "127.0.222.1" port: 41049 } } peers { permanent_uuid: "a83ae6a077a34a80985b4b65dfa7b6f6" member_type: VOTER last_known_addr { host: "127.0.222.2" port: 35933 } }
I20260809 12:13:50.799432 18127 leader_election.cc:288] T c56b5b890310479c9ea3b7ed3d43e63c P b97494c2f09f448e85a9497d47d415f6 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers bd233f0a216e42e7858d2eb736a785bb (127.0.222.1:41049), a83ae6a077a34a80985b4b65dfa7b6f6 (127.0.222.2:35933)
I20260809 12:13:50.806131 18103 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 12:13:50.799890 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.222.2:35933 (local address 127.0.222.3:42173)
0809 12:13:50.800078 (+ 188us) negotiation.cc:107] Waiting for socket to connect
0809 12:13:50.800102 (+ 24us) client_negotiation.cc:174] Beginning negotiation
0809 12:13:50.800239 (+ 137us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 12:13:50.800665 (+ 426us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 12:13:50.800671 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 12:13:50.800711 (+ 40us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0809 12:13:50.800980 (+ 269us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 12:13:50.800990 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 12:13:50.802321 (+ 1331us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 12:13:50.802324 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 12:13:50.804187 (+ 1863us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 12:13:50.804203 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 12:13:50.805552 (+ 1349us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 12:13:50.805556 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 12:13:50.805684 (+ 128us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 12:13:50.805708 (+ 24us) client_negotiation.cc:769] Sending connection context
0809 12:13:50.805782 (+ 74us) client_negotiation.cc:240] Negotiation successful
0809 12:13:50.805874 (+ 92us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":34,"mutex_wait_us":1}
I20260809 12:13:50.806263 18119 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 12:13:50.799926 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.222.3:42173 (local address 127.0.222.2:35933)
0809 12:13:50.800144 (+ 218us) server_negotiation.cc:206] Beginning negotiation
0809 12:13:50.800152 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0809 12:13:50.800174 (+ 22us) server_negotiation.cc:407] Connection header received
0809 12:13:50.800392 (+ 218us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 12:13:50.800396 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 12:13:50.800456 (+ 60us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 12:13:50.800610 (+ 154us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0809 12:13:50.801134 (+ 524us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 12:13:50.802173 (+ 1039us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 12:13:50.804369 (+ 2196us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 12:13:50.805385 (+ 1016us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 12:13:50.805524 (+ 139us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 12:13:50.805620 (+ 96us) server_negotiation.cc:1035] Waiting for connection context
0809 12:13:50.805834 (+ 214us) server_negotiation.cc:299] Negotiation successful
0809 12:13:50.805978 (+ 144us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":77}
W20260809 12:13:50.806468 17818 tablet.cc:2369] T c56b5b890310479c9ea3b7ed3d43e63c P bd233f0a216e42e7858d2eb736a785bb: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260809 12:13:50.807322 17908 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c56b5b890310479c9ea3b7ed3d43e63c" candidate_uuid: "b97494c2f09f448e85a9497d47d415f6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a83ae6a077a34a80985b4b65dfa7b6f6" is_pre_election: true
I20260809 12:13:50.807703 18117 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 12:13:50.799991 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.222.3:50611 (local address 127.0.222.1:41049)
0809 12:13:50.800186 (+ 195us) server_negotiation.cc:206] Beginning negotiation
0809 12:13:50.800193 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0809 12:13:50.801138 (+ 945us) server_negotiation.cc:407] Connection header received
0809 12:13:50.801382 (+ 244us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 12:13:50.801389 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 12:13:50.801468 (+ 79us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 12:13:50.801567 (+ 99us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0809 12:13:50.802095 (+ 528us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 12:13:50.802890 (+ 795us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 12:13:50.804974 (+ 2084us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 12:13:50.806277 (+ 1303us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 12:13:50.806439 (+ 162us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 12:13:50.806555 (+ 116us) server_negotiation.cc:1035] Waiting for connection context
0809 12:13:50.807316 (+ 761us) server_negotiation.cc:299] Negotiation successful
0809 12:13:50.807453 (+ 137us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":69}
I20260809 12:13:50.807931 17908 raft_consensus.cc:2454] T c56b5b890310479c9ea3b7ed3d43e63c P a83ae6a077a34a80985b4b65dfa7b6f6 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b97494c2f09f448e85a9497d47d415f6 in term 0.
I20260809 12:13:50.808311 18129 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 12:13:50.799890 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.222.1:41049 (local address 127.0.222.3:50611)
0809 12:13:50.801027 (+ 1137us) negotiation.cc:107] Waiting for socket to connect
0809 12:13:50.801042 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0809 12:13:50.801236 (+ 194us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 12:13:50.801585 (+ 349us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 12:13:50.801588 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 12:13:50.801605 (+ 17us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0809 12:13:50.801938 (+ 333us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 12:13:50.801947 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 12:13:50.803073 (+ 1126us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 12:13:50.803079 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 12:13:50.804804 (+ 1725us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 12:13:50.804815 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 12:13:50.806900 (+ 2085us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 12:13:50.806906 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 12:13:50.807104 (+ 198us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 12:13:50.807143 (+ 39us) client_negotiation.cc:769] Sending connection context
0809 12:13:50.807880 (+ 737us) client_negotiation.cc:240] Negotiation successful
0809 12:13:50.807994 (+ 114us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1013,"mutex_wait_us":234,"thread_start_us":500,"threads_started":1}
I20260809 12:13:50.808766 17981 leader_election.cc:302] T c56b5b890310479c9ea3b7ed3d43e63c P b97494c2f09f448e85a9497d47d415f6 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: a83ae6a077a34a80985b4b65dfa7b6f6, b97494c2f09f448e85a9497d47d415f6; no voters:
I20260809 12:13:50.809195 18127 raft_consensus.cc:2781] T c56b5b890310479c9ea3b7ed3d43e63c P b97494c2f09f448e85a9497d47d415f6 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260809 12:13:50.809369 18127 raft_consensus.cc:482] T c56b5b890310479c9ea3b7ed3d43e63c P b97494c2f09f448e85a9497d47d415f6 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260809 12:13:50.809489 18127 raft_consensus.cc:3037] T c56b5b890310479c9ea3b7ed3d43e63c P b97494c2f09f448e85a9497d47d415f6 [term 0 FOLLOWER]: Advancing to term 1
I20260809 12:13:50.809420 17767 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c56b5b890310479c9ea3b7ed3d43e63c" candidate_uuid: "b97494c2f09f448e85a9497d47d415f6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "bd233f0a216e42e7858d2eb736a785bb" is_pre_election: true
I20260809 12:13:50.809952 17767 raft_consensus.cc:2454] T c56b5b890310479c9ea3b7ed3d43e63c P bd233f0a216e42e7858d2eb736a785bb [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b97494c2f09f448e85a9497d47d415f6 in term 0.
I20260809 12:13:50.811964 18127 raft_consensus.cc:504] T c56b5b890310479c9ea3b7ed3d43e63c P b97494c2f09f448e85a9497d47d415f6 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b97494c2f09f448e85a9497d47d415f6" member_type: VOTER last_known_addr { host: "127.0.222.3" port: 43523 } } peers { permanent_uuid: "bd233f0a216e42e7858d2eb736a785bb" member_type: VOTER last_known_addr { host: "127.0.222.1" port: 41049 } } peers { permanent_uuid: "a83ae6a077a34a80985b4b65dfa7b6f6" member_type: VOTER last_known_addr { host: "127.0.222.2" port: 35933 } }
I20260809 12:13:50.812619 18127 leader_election.cc:288] T c56b5b890310479c9ea3b7ed3d43e63c P b97494c2f09f448e85a9497d47d415f6 [CANDIDATE]: Term 1 election: Requested vote from peers bd233f0a216e42e7858d2eb736a785bb (127.0.222.1:41049), a83ae6a077a34a80985b4b65dfa7b6f6 (127.0.222.2:35933)
I20260809 12:13:50.813088 17767 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c56b5b890310479c9ea3b7ed3d43e63c" candidate_uuid: "b97494c2f09f448e85a9497d47d415f6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "bd233f0a216e42e7858d2eb736a785bb"
I20260809 12:13:50.813143 17908 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c56b5b890310479c9ea3b7ed3d43e63c" candidate_uuid: "b97494c2f09f448e85a9497d47d415f6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a83ae6a077a34a80985b4b65dfa7b6f6"
I20260809 12:13:50.813311 17767 raft_consensus.cc:3037] T c56b5b890310479c9ea3b7ed3d43e63c P bd233f0a216e42e7858d2eb736a785bb [term 0 FOLLOWER]: Advancing to term 1
I20260809 12:13:50.813323 17908 raft_consensus.cc:3037] T c56b5b890310479c9ea3b7ed3d43e63c P a83ae6a077a34a80985b4b65dfa7b6f6 [term 0 FOLLOWER]: Advancing to term 1
I20260809 12:13:50.816768 17767 raft_consensus.cc:2454] T c56b5b890310479c9ea3b7ed3d43e63c P bd233f0a216e42e7858d2eb736a785bb [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b97494c2f09f448e85a9497d47d415f6 in term 1.
I20260809 12:13:50.816772 17908 raft_consensus.cc:2454] T c56b5b890310479c9ea3b7ed3d43e63c P a83ae6a077a34a80985b4b65dfa7b6f6 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b97494c2f09f448e85a9497d47d415f6 in term 1.
I20260809 12:13:50.817200 17980 leader_election.cc:302] T c56b5b890310479c9ea3b7ed3d43e63c P b97494c2f09f448e85a9497d47d415f6 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: b97494c2f09f448e85a9497d47d415f6, bd233f0a216e42e7858d2eb736a785bb; no voters:
I20260809 12:13:50.817528 18127 raft_consensus.cc:2781] T c56b5b890310479c9ea3b7ed3d43e63c P b97494c2f09f448e85a9497d47d415f6 [term 1 FOLLOWER]: Leader election won for term 1
I20260809 12:13:50.818150 18127 raft_consensus.cc:686] T c56b5b890310479c9ea3b7ed3d43e63c P b97494c2f09f448e85a9497d47d415f6 [term 1 LEADER]: Becoming Leader. State: Replica: b97494c2f09f448e85a9497d47d415f6, State: Running, Role: LEADER
I20260809 12:13:50.818724 18127 consensus_queue.cc:237] T c56b5b890310479c9ea3b7ed3d43e63c P b97494c2f09f448e85a9497d47d415f6 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b97494c2f09f448e85a9497d47d415f6" member_type: VOTER last_known_addr { host: "127.0.222.3" port: 43523 } } peers { permanent_uuid: "bd233f0a216e42e7858d2eb736a785bb" member_type: VOTER last_known_addr { host: "127.0.222.1" port: 41049 } } peers { permanent_uuid: "a83ae6a077a34a80985b4b65dfa7b6f6" member_type: VOTER last_known_addr { host: "127.0.222.2" port: 35933 } }
I20260809 12:13:50.823688 17615 catalog_manager.cc:5495] T c56b5b890310479c9ea3b7ed3d43e63c P b97494c2f09f448e85a9497d47d415f6 reported cstate change: term changed from 0 to 1, leader changed from <none> to b97494c2f09f448e85a9497d47d415f6 (127.0.222.3). New cstate: current_term: 1 leader_uuid: "b97494c2f09f448e85a9497d47d415f6" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b97494c2f09f448e85a9497d47d415f6" member_type: VOTER last_known_addr { host: "127.0.222.3" port: 43523 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "bd233f0a216e42e7858d2eb736a785bb" member_type: VOTER last_known_addr { host: "127.0.222.1" port: 41049 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "a83ae6a077a34a80985b4b65dfa7b6f6" member_type: VOTER last_known_addr { host: "127.0.222.2" port: 35933 } health_report { overall_health: UNKNOWN } } }
W20260809 12:13:50.832913 17958 tablet.cc:2369] T c56b5b890310479c9ea3b7ed3d43e63c P a83ae6a077a34a80985b4b65dfa7b6f6: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260809 12:13:50.861133 18115 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 12:13:50.855725 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54038 (local address 127.0.222.3:43523)
0809 12:13:50.856027 (+ 302us) server_negotiation.cc:206] Beginning negotiation
0809 12:13:50.856032 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0809 12:13:50.856049 (+ 17us) server_negotiation.cc:407] Connection header received
0809 12:13:50.856140 (+ 91us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 12:13:50.856144 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 12:13:50.856229 (+ 85us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 12:13:50.856366 (+ 137us) server_negotiation.cc:226] Negotiated authn=TOKEN
0809 12:13:50.856777 (+ 411us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 12:13:50.857492 (+ 715us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 12:13:50.859110 (+ 1618us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 12:13:50.859503 (+ 393us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 12:13:50.859598 (+ 95us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 12:13:50.860053 (+ 455us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0809 12:13:50.860555 (+ 502us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0809 12:13:50.860665 (+ 110us) server_negotiation.cc:1035] Waiting for connection context
0809 12:13:50.860859 (+ 194us) server_negotiation.cc:299] Negotiation successful
0809 12:13:50.860930 (+ 71us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":179}
W20260809 12:13:50.868191 18101 tablet.cc:2369] T c56b5b890310479c9ea3b7ed3d43e63c P b97494c2f09f448e85a9497d47d415f6: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260809 12:13:50.869447 17767 raft_consensus.cc:1261] T c56b5b890310479c9ea3b7ed3d43e63c P bd233f0a216e42e7858d2eb736a785bb [term 1 FOLLOWER]: Refusing update from remote peer b97494c2f09f448e85a9497d47d415f6: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260809 12:13:50.869520 17908 raft_consensus.cc:1261] T c56b5b890310479c9ea3b7ed3d43e63c P a83ae6a077a34a80985b4b65dfa7b6f6 [term 1 FOLLOWER]: Refusing update from remote peer b97494c2f09f448e85a9497d47d415f6: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260809 12:13:50.870471 18130 consensus_queue.cc:1035] T c56b5b890310479c9ea3b7ed3d43e63c P b97494c2f09f448e85a9497d47d415f6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a83ae6a077a34a80985b4b65dfa7b6f6" member_type: VOTER last_known_addr { host: "127.0.222.2" port: 35933 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260809 12:13:50.870829 18127 consensus_queue.cc:1035] T c56b5b890310479c9ea3b7ed3d43e63c P b97494c2f09f448e85a9497d47d415f6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "bd233f0a216e42e7858d2eb736a785bb" member_type: VOTER last_known_addr { host: "127.0.222.1" port: 41049 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260809 12:13:50.884020 18135 mvcc.cc:205] Tried to move back new op lower bound from 7316593176030625792 to 7316593175838187520. Current Snapshot: MvccSnapshot[applied={T|T < 7316593176030625792}]
I20260809 12:13:50.897133 17614 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:41812:
table { table_name: "test-table" } modify_external_catalogs: true
I20260809 12:13:50.897377 17614 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:41812:
table { table_name: "test-table" } modify_external_catalogs: true
I20260809 12:13:50.901000 17614 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 976efcee2bd94ed6b5c102cc07ac7ab1: Sending DeleteTablet for 3 replicas of tablet c56b5b890310479c9ea3b7ed3d43e63c
I20260809 12:13:50.902217 18028 tablet_service.cc:1514] Processing DeleteTablet for tablet c56b5b890310479c9ea3b7ed3d43e63c with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-09 12:13:50 UTC) from {username='kudu', principal='kudu/127.0.222.62@KRBTEST.COM'} at 127.0.0.1:54026
I20260809 12:13:50.902305 17888 tablet_service.cc:1514] Processing DeleteTablet for tablet c56b5b890310479c9ea3b7ed3d43e63c with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-09 12:13:50 UTC) from {username='kudu', principal='kudu/127.0.222.62@KRBTEST.COM'} at 127.0.0.1:49042
I20260809 12:13:50.902275 17747 tablet_service.cc:1514] Processing DeleteTablet for tablet c56b5b890310479c9ea3b7ed3d43e63c with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-09 12:13:50 UTC) from {username='kudu', principal='kudu/127.0.222.62@KRBTEST.COM'} at 127.0.0.1:60294
I20260809 12:13:50.906026 888 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskgH5Pjm/build/asan/bin/kudu pid 17681
I20260809 12:13:50.913319 18143 tablet_replica.cc:330] T c56b5b890310479c9ea3b7ed3d43e63c P b97494c2f09f448e85a9497d47d415f6: stopping tablet replica
I20260809 12:13:50.914055 18143 raft_consensus.cc:2229] T c56b5b890310479c9ea3b7ed3d43e63c P b97494c2f09f448e85a9497d47d415f6 [term 1 LEADER]: Raft consensus shutting down.
I20260809 12:13:50.915535 18142 tablet_replica.cc:330] T c56b5b890310479c9ea3b7ed3d43e63c P a83ae6a077a34a80985b4b65dfa7b6f6: stopping tablet replica
I20260809 12:13:50.915763 18144 tablet_replica.cc:330] T c56b5b890310479c9ea3b7ed3d43e63c P bd233f0a216e42e7858d2eb736a785bb: stopping tablet replica
I20260809 12:13:50.915738 18143 raft_consensus.cc:2258] T c56b5b890310479c9ea3b7ed3d43e63c P b97494c2f09f448e85a9497d47d415f6 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260809 12:13:50.916213 18144 raft_consensus.cc:2229] T c56b5b890310479c9ea3b7ed3d43e63c P bd233f0a216e42e7858d2eb736a785bb [term 1 FOLLOWER]: Raft consensus shutting down.
I20260809 12:13:50.916217 18142 raft_consensus.cc:2229] T c56b5b890310479c9ea3b7ed3d43e63c P a83ae6a077a34a80985b4b65dfa7b6f6 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260809 12:13:50.916812 18144 raft_consensus.cc:2258] T c56b5b890310479c9ea3b7ed3d43e63c P bd233f0a216e42e7858d2eb736a785bb [term 1 FOLLOWER]: Raft consensus is shut down!
I20260809 12:13:50.916962 18142 raft_consensus.cc:2258] T c56b5b890310479c9ea3b7ed3d43e63c P a83ae6a077a34a80985b4b65dfa7b6f6 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260809 12:13:50.919535 18143 ts_tablet_manager.cc:1905] T c56b5b890310479c9ea3b7ed3d43e63c P b97494c2f09f448e85a9497d47d415f6: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260809 12:13:50.920502 18144 ts_tablet_manager.cc:1905] T c56b5b890310479c9ea3b7ed3d43e63c P bd233f0a216e42e7858d2eb736a785bb: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260809 12:13:50.921212 18142 ts_tablet_manager.cc:1905] T c56b5b890310479c9ea3b7ed3d43e63c P a83ae6a077a34a80985b4b65dfa7b6f6: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260809 12:13:50.927248 18143 ts_tablet_manager.cc:1918] T c56b5b890310479c9ea3b7ed3d43e63c P b97494c2f09f448e85a9497d47d415f6: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260809 12:13:50.927500 18143 log.cc:1196] T c56b5b890310479c9ea3b7ed3d43e63c P b97494c2f09f448e85a9497d47d415f6: Deleting WAL directory at /tmp/dist-test-taskgH5Pjm/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786277382073938-888-0/minicluster-data/ts-2/wal/wals/c56b5b890310479c9ea3b7ed3d43e63c
I20260809 12:13:50.928171 18143 ts_tablet_manager.cc:1939] T c56b5b890310479c9ea3b7ed3d43e63c P b97494c2f09f448e85a9497d47d415f6: Deleting consensus metadata
Aug 09 12:13:50 dist-test-slave-2w3w krb5kdc[17567](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786277628, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.222.1@KRBTEST.COM
I20260809 12:13:50.930657 17598 catalog_manager.cc:4841] TS b97494c2f09f448e85a9497d47d415f6 (127.0.222.3:43523): tablet c56b5b890310479c9ea3b7ed3d43e63c (table test-table [id=0684479eb30f4e14bb2db8d41c3af017]) successfully deleted
I20260809 12:13:50.938215 18117 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 12:13:50.907152 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60310 (local address 127.0.222.1:41049)
0809 12:13:50.907321 (+ 169us) server_negotiation.cc:206] Beginning negotiation
0809 12:13:50.907329 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0809 12:13:50.917787 (+ 10458us) server_negotiation.cc:407] Connection header received
0809 12:13:50.918122 (+ 335us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 12:13:50.918130 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 12:13:50.918223 (+ 93us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 12:13:50.918301 (+ 78us) server_negotiation.cc:226] Negotiated authn=SASL
0809 12:13:50.920732 (+ 2431us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 12:13:50.921825 (+ 1093us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 12:13:50.924057 (+ 2232us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 12:13:50.924702 (+ 645us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 12:13:50.924789 (+ 87us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 12:13:50.930337 (+ 5548us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 12:13:50.930352 (+ 15us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 12:13:50.930370 (+ 18us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0809 12:13:50.930411 (+ 41us) server_negotiation.cc:922] calling sasl_server_start()
0809 12:13:50.934338 (+ 3927us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 12:13:50.935350 (+ 1012us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 12:13:50.935359 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 12:13:50.935363 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0809 12:13:50.935561 (+ 198us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 12:13:50.936029 (+ 468us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 12:13:50.936035 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 12:13:50.936043 (+ 8us) server_negotiation.cc:967] calling sasl_server_step()
0809 12:13:50.936539 (+ 496us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 12:13:50.936862 (+ 323us) server_negotiation.cc:1035] Waiting for connection context
0809 12:13:50.937559 (+ 697us) server_negotiation.cc:299] Negotiation successful
0809 12:13:50.937794 (+ 235us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":56}
I20260809 12:13:50.940096 17812 generic_service.cc:196] Checking for leaks (request via RPC)
I20260809 12:13:50.945969 18142 ts_tablet_manager.cc:1918] T c56b5b890310479c9ea3b7ed3d43e63c P a83ae6a077a34a80985b4b65dfa7b6f6: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260809 12:13:50.946189 18142 log.cc:1196] T c56b5b890310479c9ea3b7ed3d43e63c P a83ae6a077a34a80985b4b65dfa7b6f6: Deleting WAL directory at /tmp/dist-test-taskgH5Pjm/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786277382073938-888-0/minicluster-data/ts-1/wal/wals/c56b5b890310479c9ea3b7ed3d43e63c
I20260809 12:13:50.946801 18142 ts_tablet_manager.cc:1939] T c56b5b890310479c9ea3b7ed3d43e63c P a83ae6a077a34a80985b4b65dfa7b6f6: Deleting consensus metadata
I20260809 12:13:50.948971 17601 catalog_manager.cc:4841] TS a83ae6a077a34a80985b4b65dfa7b6f6 (127.0.222.2:35933): tablet c56b5b890310479c9ea3b7ed3d43e63c (table test-table [id=0684479eb30f4e14bb2db8d41c3af017]) successfully deleted
I20260809 12:13:51.040555 18144 ts_tablet_manager.cc:1918] T c56b5b890310479c9ea3b7ed3d43e63c P bd233f0a216e42e7858d2eb736a785bb: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260809 12:13:51.040850 18144 log.cc:1196] T c56b5b890310479c9ea3b7ed3d43e63c P bd233f0a216e42e7858d2eb736a785bb: Deleting WAL directory at /tmp/dist-test-taskgH5Pjm/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786277382073938-888-0/minicluster-data/ts-0/wal/wals/c56b5b890310479c9ea3b7ed3d43e63c
I20260809 12:13:51.041206 888 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskgH5Pjm/build/asan/bin/kudu with pid 17681
W20260809 12:13:51.064738 17600 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.0.222.1:41049 (error 108)
I20260809 12:13:51.064919 888 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskgH5Pjm/build/asan/bin/kudu pid 17823
I20260809 12:13:51.066916 18116 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 12:13:51.066447 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.222.1:41049 (local address 127.0.0.1:60320)
0809 12:13:51.066610 (+ 163us) negotiation.cc:107] Waiting for socket to connect
0809 12:13:51.066734 (+ 124us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.222.1:41049: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":37}
W20260809 12:13:51.067267 17600 catalog_manager.cc:4584] TS bd233f0a216e42e7858d2eb736a785bb (127.0.222.1:41049): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet c56b5b890310479c9ea3b7ed3d43e63c: Network error: Client connection negotiation failed: client connection to 127.0.222.1:41049: connect: Connection refused (error 111)
Aug 09 12:13:51 dist-test-slave-2w3w krb5kdc[17567](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786277628, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.222.2@KRBTEST.COM
I20260809 12:13:51.082623 18119 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 12:13:51.066450 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49058 (local address 127.0.222.2:35933)
0809 12:13:51.066684 (+ 234us) server_negotiation.cc:206] Beginning negotiation
0809 12:13:51.066689 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0809 12:13:51.066705 (+ 16us) server_negotiation.cc:407] Connection header received
0809 12:13:51.066762 (+ 57us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 12:13:51.066766 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 12:13:51.066829 (+ 63us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 12:13:51.066963 (+ 134us) server_negotiation.cc:226] Negotiated authn=SASL
0809 12:13:51.068599 (+ 1636us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 12:13:51.069327 (+ 728us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 12:13:51.070655 (+ 1328us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 12:13:51.071233 (+ 578us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 12:13:51.071390 (+ 157us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 12:13:51.076048 (+ 4658us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 12:13:51.076062 (+ 14us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 12:13:51.076077 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0809 12:13:51.076110 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0809 12:13:51.079266 (+ 3156us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 12:13:51.080186 (+ 920us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 12:13:51.080195 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 12:13:51.080199 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0809 12:13:51.080338 (+ 139us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 12:13:51.080814 (+ 476us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 12:13:51.080820 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 12:13:51.080822 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0809 12:13:51.081343 (+ 521us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 12:13:51.081671 (+ 328us) server_negotiation.cc:1035] Waiting for connection context
0809 12:13:51.082040 (+ 369us) server_negotiation.cc:299] Negotiation successful
0809 12:13:51.082260 (+ 220us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":113}
I20260809 12:13:51.091812 17952 generic_service.cc:196] Checking for leaks (request via RPC)
I20260809 12:13:51.121233 18116 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 12:13:51.120650 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.222.1:41049 (local address 127.0.0.1:60336)
0809 12:13:51.120905 (+ 255us) negotiation.cc:107] Waiting for socket to connect
0809 12:13:51.121017 (+ 112us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.222.1:41049: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":89}
I20260809 12:13:51.122434 18116 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 12:13:51.122017 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.222.1:41049 (local address 127.0.0.1:60350)
0809 12:13:51.122188 (+ 171us) negotiation.cc:107] Waiting for socket to connect
0809 12:13:51.122283 (+ 95us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.222.1:41049: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":57}
I20260809 12:13:51.184197 18116 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 12:13:51.183705 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.222.1:41049 (local address 127.0.0.1:60366)
0809 12:13:51.183888 (+ 183us) negotiation.cc:107] Waiting for socket to connect
0809 12:13:51.183982 (+ 94us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.222.1:41049: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":55}
I20260809 12:13:51.184557 888 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskgH5Pjm/build/asan/bin/kudu with pid 17823
I20260809 12:13:51.188895 18116 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 12:13:51.188437 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.222.1:41049 (local address 127.0.0.1:60368)
0809 12:13:51.188616 (+ 179us) negotiation.cc:107] Waiting for socket to connect
0809 12:13:51.188716 (+ 100us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.222.1:41049: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":70}
I20260809 12:13:51.213878 888 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskgH5Pjm/build/asan/bin/kudu pid 17963
Aug 09 12:13:51 dist-test-slave-2w3w krb5kdc[17567](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786277628, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.222.3@KRBTEST.COM
I20260809 12:13:51.234174 18115 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 12:13:51.217137 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54040 (local address 127.0.222.3:43523)
0809 12:13:51.217367 (+ 230us) server_negotiation.cc:206] Beginning negotiation
0809 12:13:51.217373 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0809 12:13:51.217395 (+ 22us) server_negotiation.cc:407] Connection header received
0809 12:13:51.217464 (+ 69us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 12:13:51.217470 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 12:13:51.217588 (+ 118us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 12:13:51.217718 (+ 130us) server_negotiation.cc:226] Negotiated authn=SASL
0809 12:13:51.219389 (+ 1671us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 12:13:51.220246 (+ 857us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 12:13:51.221646 (+ 1400us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 12:13:51.222140 (+ 494us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 12:13:51.222285 (+ 145us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 12:13:51.226879 (+ 4594us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 12:13:51.226898 (+ 19us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 12:13:51.226920 (+ 22us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0809 12:13:51.227009 (+ 89us) server_negotiation.cc:922] calling sasl_server_start()
0809 12:13:51.231181 (+ 4172us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 12:13:51.231867 (+ 686us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 12:13:51.231877 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 12:13:51.231880 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0809 12:13:51.232083 (+ 203us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 12:13:51.232489 (+ 406us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 12:13:51.232494 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 12:13:51.232498 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0809 12:13:51.232965 (+ 467us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 12:13:51.233340 (+ 375us) server_negotiation.cc:1035] Waiting for connection context
0809 12:13:51.233504 (+ 164us) server_negotiation.cc:299] Negotiation successful
0809 12:13:51.233730 (+ 226us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":42}
I20260809 12:13:51.235134 18094 generic_service.cc:196] Checking for leaks (request via RPC)
I20260809 12:13:51.269496 18116 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 12:13:51.269012 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.222.1:41049 (local address 127.0.0.1:60370)
0809 12:13:51.269249 (+ 237us) negotiation.cc:107] Waiting for socket to connect
0809 12:13:51.269343 (+ 94us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.222.1:41049: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":84}
I20260809 12:13:51.270641 18116 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 12:13:51.270233 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.222.1:41049 (local address 127.0.0.1:60384)
0809 12:13:51.270439 (+ 206us) negotiation.cc:107] Waiting for socket to connect
0809 12:13:51.270524 (+ 85us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.222.1:41049: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":116}
I20260809 12:13:51.360423 888 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskgH5Pjm/build/asan/bin/kudu with pid 17963
I20260809 12:13:51.384846 888 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskgH5Pjm/build/asan/bin/kudu pid 17582
I20260809 12:13:51.386060 17647 generic_service.cc:196] Checking for leaks (request via RPC)
I20260809 12:13:51.500437 888 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskgH5Pjm/build/asan/bin/kudu with pid 17582
I20260809 12:14:11.527141 888 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskgH5Pjm/build/asan/bin/kudu
/tmp/dist-test-taskgH5Pjm/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskgH5Pjm/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786277382073938-888-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskgH5Pjm/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786277382073938-888-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskgH5Pjm/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786277382073938-888-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskgH5Pjm/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786277382073938-888-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.222.62:45505
--webserver_interface=127.0.222.62
--webserver_port=35835
--keytab_file=/tmp/dist-test-taskgH5Pjm/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786277382073938-888-0/krb5kdc/kudu_127.0.222.62.keytab
--principal=kudu/127.0.222.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.222.20:41481
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.222.62:45505
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--authn_token_validity_seconds=20
--authz_token_validity_seconds=20
--tsk_rotation_seconds=20
--tsk_private_key_password_cmd=echo foo with env {KRB5CCNAME=/tmp/dist-test-taskgH5Pjm/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786277382073938-888-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskgH5Pjm/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786277382073938-888-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskgH5Pjm/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786277382073938-888-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260809 12:14:11.883900 18151 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260809 12:14:11.884193 18151 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260809 12:14:11.884287 18151 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260809 12:14:11.891506 18151 flags.cc:425] Enabled experimental flag: --authn_token_validity_seconds=20
W20260809 12:14:11.891601 18151 flags.cc:425] Enabled experimental flag: --authz_token_validity_seconds=20
W20260809 12:14:11.891642 18151 flags.cc:425] Enabled experimental flag: --tsk_rotation_seconds=20
W20260809 12:14:11.891698 18151 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260809 12:14:11.891752 18151 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260809 12:14:11.891815 18151 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260809 12:14:11.891862 18151 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260809 12:14:11.891911 18151 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260809 12:14:11.902190 18151 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.222.20:41481
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskgH5Pjm/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786277382073938-888-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskgH5Pjm/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786277382073938-888-0/minicluster-data/master-0/wal
--authn_token_validity_seconds=20
--authz_token_validity_seconds=20
--tsk_private_key_password_cmd=echo foo
--tsk_rotation_seconds=20
--ipki_ca_key_size=768
--master_addresses=127.0.222.62:45505
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskgH5Pjm/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786277382073938-888-0/krb5kdc/kudu_127.0.222.62.keytab
--principal=kudu/127.0.222.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.222.62:45505
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskgH5Pjm/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786277382073938-888-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.222.62
--webserver_port=35835
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskgH5Pjm/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786277382073938-888-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 09 Aug 2026 11:06:41 UTC on ccycloud.quasar-bwoogy.root.comops.site
build id 4575991
ASAN enabled
I20260809 12:14:11.903649 18151 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260809 12:14:11.905004 18151 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260809 12:14:11.932796 18161 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 12:14:11.932894 18159 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 12:14:11.932822 18158 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260809 12:14:11.933779 18151 server_base.cc:1034] running on GCE node
I20260809 12:14:11.934712 18151 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260809 12:14:11.936455 18151 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260809 12:14:11.937711 18151 hybrid_clock.cc:648] HybridClock initialized: now 1786277651937625 us; error 58 us; skew 500 ppm
I20260809 12:14:11.938131 18151 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 09 12:14:11 dist-test-slave-2w3w krb5kdc[17567](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786277651, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.222.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260809 12:14:11.945785 18151 init.cc:377] Logged in from keytab as kudu/127.0.222.62@KRBTEST.COM (short username kudu)
I20260809 12:14:11.947618 18151 webserver.cc:460] Webserver started at http://127.0.222.62:35835/ using document root <none> and password file <none>
I20260809 12:14:11.948431 18151 fs_manager.cc:359] Metadata directory not provided
I20260809 12:14:11.948582 18151 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260809 12:14:11.954908 18151 fs_manager.cc:709] Time spent opening directory manager: real 0.004s user 0.002s sys 0.001s
I20260809 12:14:11.957939 18169 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260809 12:14:11.959537 18151 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260809 12:14:11.959738 18151 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskgH5Pjm/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786277382073938-888-0/minicluster-data/master-0/data,/tmp/dist-test-taskgH5Pjm/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786277382073938-888-0/minicluster-data/master-0/wal
uuid: "976efcee2bd94ed6b5c102cc07ac7ab1"
format_stamp: "Formatted at 2026-08-09 12:13:48 on dist-test-slave-2w3w"
server_key: "5f75dd8c4f7b73a87d0db18621478cff"
server_key_iv: "fdfbac5a7eab9ac2b3d7ff0e02c44b99"
server_key_version: "encryptionkey@0"
I20260809 12:14:11.960843 18151 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskgH5Pjm/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786277382073938-888-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskgH5Pjm/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786277382073938-888-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskgH5Pjm/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786277382073938-888-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260809 12:14:11.992726 18151 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260809 12:14:11.997344 18151 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260809 12:14:11.997681 18151 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260809 12:14:12.018510 18151 rpc_server.cc:307] RPC server started. Bound to: 127.0.222.62:45505
I20260809 12:14:12.018551 18221 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.222.62:45505 every 8 connection(s)
I20260809 12:14:12.020198 18151 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskgH5Pjm/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786277382073938-888-0/minicluster-data/master-0/data/info.pb
I20260809 12:14:12.021505 888 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskgH5Pjm/build/asan/bin/kudu as pid 18151
I20260809 12:14:12.022622 888 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskgH5Pjm/build/asan/bin/kudu
/tmp/dist-test-taskgH5Pjm/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskgH5Pjm/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786277382073938-888-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskgH5Pjm/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786277382073938-888-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskgH5Pjm/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786277382073938-888-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskgH5Pjm/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786277382073938-888-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.222.1:41049
--local_ip_for_outbound_sockets=127.0.222.1
--tserver_master_addrs=127.0.222.62:45505
--webserver_port=38689
--webserver_interface=127.0.222.1
--keytab_file=/tmp/dist-test-taskgH5Pjm/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786277382073938-888-0/krb5kdc/kudu_127.0.222.1.keytab
--principal=kudu/127.0.222.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.222.20:41481
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskgH5Pjm/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786277382073938-888-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskgH5Pjm/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786277382073938-888-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskgH5Pjm/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786277382073938-888-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260809 12:14:12.030761 18222 sys_catalog.cc:263] Verifying existing consensus state
I20260809 12:14:12.036082 18222 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 976efcee2bd94ed6b5c102cc07ac7ab1: Bootstrap starting.
I20260809 12:14:12.057049 18222 log.cc:824] T 00000000000000000000000000000000 P 976efcee2bd94ed6b5c102cc07ac7ab1: Log is configured to *not* fsync() on all Append() calls
I20260809 12:14:12.069877 18222 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 976efcee2bd94ed6b5c102cc07ac7ab1: Bootstrap replayed 1/1 log segments. Stats: ops{read=8 overwritten=0 applied=8 ignored=0} inserts{seen=5 ignored=0} mutations{seen=4 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260809 12:14:12.070489 18222 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 976efcee2bd94ed6b5c102cc07ac7ab1: Bootstrap complete.
I20260809 12:14:12.080130 18222 raft_consensus.cc:348] T 00000000000000000000000000000000 P 976efcee2bd94ed6b5c102cc07ac7ab1 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "976efcee2bd94ed6b5c102cc07ac7ab1" member_type: VOTER last_known_addr { host: "127.0.222.62" port: 45505 } }
I20260809 12:14:12.081015 18222 raft_consensus.cc:729] T 00000000000000000000000000000000 P 976efcee2bd94ed6b5c102cc07ac7ab1 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 976efcee2bd94ed6b5c102cc07ac7ab1, State: Initialized, Role: FOLLOWER
I20260809 12:14:12.081688 18222 consensus_queue.cc:260] T 00000000000000000000000000000000 P 976efcee2bd94ed6b5c102cc07ac7ab1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 8, Last appended: 1.8, Last appended by leader: 8, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "976efcee2bd94ed6b5c102cc07ac7ab1" member_type: VOTER last_known_addr { host: "127.0.222.62" port: 45505 } }
I20260809 12:14:12.081943 18222 raft_consensus.cc:388] T 00000000000000000000000000000000 P 976efcee2bd94ed6b5c102cc07ac7ab1 [term 1 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260809 12:14:12.082072 18222 raft_consensus.cc:482] T 00000000000000000000000000000000 P 976efcee2bd94ed6b5c102cc07ac7ab1 [term 1 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260809 12:14:12.082264 18222 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 976efcee2bd94ed6b5c102cc07ac7ab1 [term 1 FOLLOWER]: Advancing to term 2
I20260809 12:14:12.086227 18222 raft_consensus.cc:504] T 00000000000000000000000000000000 P 976efcee2bd94ed6b5c102cc07ac7ab1 [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "976efcee2bd94ed6b5c102cc07ac7ab1" member_type: VOTER last_known_addr { host: "127.0.222.62" port: 45505 } }
I20260809 12:14:12.086776 18222 leader_election.cc:302] T 00000000000000000000000000000000 P 976efcee2bd94ed6b5c102cc07ac7ab1 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 976efcee2bd94ed6b5c102cc07ac7ab1; no voters:
I20260809 12:14:12.087787 18222 leader_election.cc:288] T 00000000000000000000000000000000 P 976efcee2bd94ed6b5c102cc07ac7ab1 [CANDIDATE]: Term 2 election: Requested vote from peers
I20260809 12:14:12.088198 18226 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 976efcee2bd94ed6b5c102cc07ac7ab1 [term 2 FOLLOWER]: Leader election won for term 2
I20260809 12:14:12.089588 18226 raft_consensus.cc:686] T 00000000000000000000000000000000 P 976efcee2bd94ed6b5c102cc07ac7ab1 [term 2 LEADER]: Becoming Leader. State: Replica: 976efcee2bd94ed6b5c102cc07ac7ab1, State: Running, Role: LEADER
I20260809 12:14:12.090155 18226 consensus_queue.cc:237] T 00000000000000000000000000000000 P 976efcee2bd94ed6b5c102cc07ac7ab1 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 1.8, Last appended by leader: 8, Current term: 2, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "976efcee2bd94ed6b5c102cc07ac7ab1" member_type: VOTER last_known_addr { host: "127.0.222.62" port: 45505 } }
I20260809 12:14:12.090540 18222 sys_catalog.cc:564] T 00000000000000000000000000000000 P 976efcee2bd94ed6b5c102cc07ac7ab1 [sys.catalog]: configured and running, proceeding with master startup.
I20260809 12:14:12.092769 18229 sys_catalog.cc:455] T 00000000000000000000000000000000 P 976efcee2bd94ed6b5c102cc07ac7ab1 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 976efcee2bd94ed6b5c102cc07ac7ab1. Latest consensus state: current_term: 2 leader_uuid: "976efcee2bd94ed6b5c102cc07ac7ab1" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "976efcee2bd94ed6b5c102cc07ac7ab1" member_type: VOTER last_known_addr { host: "127.0.222.62" port: 45505 } } }
I20260809 12:14:12.093168 18229 sys_catalog.cc:458] T 00000000000000000000000000000000 P 976efcee2bd94ed6b5c102cc07ac7ab1 [sys.catalog]: This master's current role is: LEADER
I20260809 12:14:12.094130 18227 sys_catalog.cc:455] T 00000000000000000000000000000000 P 976efcee2bd94ed6b5c102cc07ac7ab1 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 2 leader_uuid: "976efcee2bd94ed6b5c102cc07ac7ab1" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "976efcee2bd94ed6b5c102cc07ac7ab1" member_type: VOTER last_known_addr { host: "127.0.222.62" port: 45505 } } }
I20260809 12:14:12.094532 18227 sys_catalog.cc:458] T 00000000000000000000000000000000 P 976efcee2bd94ed6b5c102cc07ac7ab1 [sys.catalog]: This master's current role is: LEADER
I20260809 12:14:12.111351 18232 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260809 12:14:12.119700 18232 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260809 12:14:12.121206 18232 catalog_manager.cc:1259] Loaded cluster ID: d8085938efb84ddc8c174e7e9bb18f14
I20260809 12:14:12.121412 18232 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260809 12:14:12.126088 18232 catalog_manager.cc:1504] Loading token signing keys...
I20260809 12:14:12.127698 18232 catalog_manager.cc:5879] T 00000000000000000000000000000000 P 976efcee2bd94ed6b5c102cc07ac7ab1: Loaded TSK: 0
I20260809 12:14:12.169938 18232 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 976efcee2bd94ed6b5c102cc07ac7ab1: Generated new TSK 1
I20260809 12:14:12.170358 18232 catalog_manager.cc:1514] Initializing in-progress tserver states...
W20260809 12:14:12.401149 18224 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260809 12:14:12.401446 18224 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260809 12:14:12.401602 18224 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260809 12:14:12.409052 18224 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260809 12:14:12.409165 18224 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260809 12:14:12.409344 18224 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.222.1
I20260809 12:14:12.419348 18224 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.222.20:41481
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskgH5Pjm/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786277382073938-888-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskgH5Pjm/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786277382073938-888-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskgH5Pjm/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786277382073938-888-0/krb5kdc/kudu_127.0.222.1.keytab
--principal=kudu/127.0.222.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.222.1:41049
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskgH5Pjm/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786277382073938-888-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.222.1
--webserver_port=38689
--tserver_master_addrs=127.0.222.62:45505
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.222.1
--log_dir=/tmp/dist-test-taskgH5Pjm/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786277382073938-888-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 09 Aug 2026 11:06:41 UTC on ccycloud.quasar-bwoogy.root.comops.site
build id 4575991
ASAN enabled
I20260809 12:14:12.420691 18224 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260809 12:14:12.422042 18224 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260809 12:14:12.435567 18251 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 12:14:12.436018 18252 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 12:14:12.436301 18255 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260809 12:14:12.437161 18224 server_base.cc:1034] running on GCE node
I20260809 12:14:12.437875 18224 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260809 12:14:12.439021 18224 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260809 12:14:12.440274 18224 hybrid_clock.cc:648] HybridClock initialized: now 1786277652440228 us; error 31 us; skew 500 ppm
I20260809 12:14:12.440651 18224 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 09 12:14:12 dist-test-slave-2w3w krb5kdc[17567](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786277652, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.222.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260809 12:14:12.444460 18224 init.cc:377] Logged in from keytab as kudu/127.0.222.1@KRBTEST.COM (short username kudu)
I20260809 12:14:12.446134 18224 webserver.cc:460] Webserver started at http://127.0.222.1:38689/ using document root <none> and password file <none>
I20260809 12:14:12.446835 18224 fs_manager.cc:359] Metadata directory not provided
I20260809 12:14:12.447010 18224 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260809 12:14:12.452749 18224 fs_manager.cc:709] Time spent opening directory manager: real 0.004s user 0.004s sys 0.000s
I20260809 12:14:12.455786 18262 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260809 12:14:12.457371 18224 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260809 12:14:12.457590 18224 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskgH5Pjm/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786277382073938-888-0/minicluster-data/ts-0/data,/tmp/dist-test-taskgH5Pjm/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786277382073938-888-0/minicluster-data/ts-0/wal
uuid: "bd233f0a216e42e7858d2eb736a785bb"
format_stamp: "Formatted at 2026-08-09 12:13:49 on dist-test-slave-2w3w"
server_key: "9f4432a335e7b85d34e07fd2f3696829"
server_key_iv: "1d7e63d4b85cc01a747aca61fa912fc3"
server_key_version: "encryptionkey@0"
I20260809 12:14:12.458595 18224 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskgH5Pjm/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786277382073938-888-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskgH5Pjm/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786277382073938-888-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskgH5Pjm/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786277382073938-888-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260809 12:14:12.495100 18224 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260809 12:14:12.499496 18224 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260809 12:14:12.499791 18224 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260809 12:14:12.500825 18224 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260809 12:14:12.502668 18270 ts_tablet_manager.cc:542] Loading tablet metadata (0/1 complete)
W20260809 12:14:12.508396 18270 ts_tablet_manager.cc:1852] T c56b5b890310479c9ea3b7ed3d43e63c P bd233f0a216e42e7858d2eb736a785bb: Tablet Manager startup: Rolling forward tablet deletion of type TABLET_DATA_DELETED
I20260809 12:14:12.509349 18270 ts_tablet_manager.cc:1905] T c56b5b890310479c9ea3b7ed3d43e63c P bd233f0a216e42e7858d2eb736a785bb: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260809 12:14:12.516518 18270 ts_tablet_manager.cc:1918] T c56b5b890310479c9ea3b7ed3d43e63c P bd233f0a216e42e7858d2eb736a785bb: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260809 12:14:12.516752 18270 ts_tablet_manager.cc:1939] T c56b5b890310479c9ea3b7ed3d43e63c P bd233f0a216e42e7858d2eb736a785bb: Deleting consensus metadata
I20260809 12:14:12.517344 18224 ts_tablet_manager.cc:579] Loaded tablet metadata (1 total tablets, 0 live tablets)
I20260809 12:14:12.517498 18224 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.015s user 0.000s sys 0.000s
I20260809 12:14:12.517640 18224 ts_tablet_manager.cc:610] Registered 0 tablets
I20260809 12:14:12.517731 18224 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260809 12:14:12.554388 18224 rpc_server.cc:307] RPC server started. Bound to: 127.0.222.1:41049
I20260809 12:14:12.554450 18384 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.222.1:41049 every 8 connection(s)
I20260809 12:14:12.555794 18224 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskgH5Pjm/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786277382073938-888-0/minicluster-data/ts-0/data/info.pb
I20260809 12:14:12.566243 888 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskgH5Pjm/build/asan/bin/kudu as pid 18224
I20260809 12:14:12.567054 888 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskgH5Pjm/build/asan/bin/kudu
/tmp/dist-test-taskgH5Pjm/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskgH5Pjm/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786277382073938-888-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskgH5Pjm/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786277382073938-888-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskgH5Pjm/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786277382073938-888-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskgH5Pjm/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786277382073938-888-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.222.2:35933
--local_ip_for_outbound_sockets=127.0.222.2
--tserver_master_addrs=127.0.222.62:45505
--webserver_port=37553
--webserver_interface=127.0.222.2
--keytab_file=/tmp/dist-test-taskgH5Pjm/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786277382073938-888-0/krb5kdc/kudu_127.0.222.2.keytab
--principal=kudu/127.0.222.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.222.20:41481
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskgH5Pjm/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786277382073938-888-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskgH5Pjm/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786277382073938-888-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskgH5Pjm/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786277382073938-888-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Aug 09 12:14:12 dist-test-slave-2w3w krb5kdc[17567](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786277652, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.222.1@KRBTEST.COM for kudu/127.0.222.62@KRBTEST.COM
I20260809 12:14:12.581982 18388 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 12:14:12.558775 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.222.62:45505 (local address 127.0.222.1:52339)
0809 12:14:12.560376 (+ 1601us) negotiation.cc:107] Waiting for socket to connect
0809 12:14:12.560431 (+ 55us) client_negotiation.cc:174] Beginning negotiation
0809 12:14:12.562328 (+ 1897us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 12:14:12.563741 (+ 1413us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 12:14:12.563754 (+ 13us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 12:14:12.564414 (+ 660us) client_negotiation.cc:189] Negotiated authn=SASL
0809 12:14:12.565019 (+ 605us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 12:14:12.565036 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 12:14:12.567389 (+ 2353us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 12:14:12.567393 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 12:14:12.568636 (+ 1243us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 12:14:12.568649 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 12:14:12.570137 (+ 1488us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 12:14:12.570143 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 12:14:12.570316 (+ 173us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 12:14:12.571052 (+ 736us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0809 12:14:12.571080 (+ 28us) client_negotiation.cc:656] calling sasl_client_start()
0809 12:14:12.573739 (+ 2659us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0809 12:14:12.577561 (+ 3822us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0809 12:14:12.577572 (+ 11us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0809 12:14:12.577589 (+ 17us) client_negotiation.cc:760] calling sasl_client_step()
0809 12:14:12.578231 (+ 642us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0809 12:14:12.578677 (+ 446us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0809 12:14:12.578685 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0809 12:14:12.578691 (+ 6us) client_negotiation.cc:760] calling sasl_client_step()
0809 12:14:12.578930 (+ 239us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0809 12:14:12.579670 (+ 740us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0809 12:14:12.579678 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0809 12:14:12.580106 (+ 428us) client_negotiation.cc:769] Sending connection context
0809 12:14:12.580451 (+ 345us) client_negotiation.cc:240] Negotiation successful
0809 12:14:12.580996 (+ 545us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1297,"thread_start_us":1039,"threads_started":1}
I20260809 12:14:12.582232 18387 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 12:14:12.558737 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.222.1:52339 (local address 127.0.222.62:45505)
0809 12:14:12.559804 (+ 1067us) server_negotiation.cc:206] Beginning negotiation
0809 12:14:12.559817 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0809 12:14:12.560587 (+ 770us) server_negotiation.cc:407] Connection header received
0809 12:14:12.562855 (+ 2268us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 12:14:12.562865 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 12:14:12.563249 (+ 384us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 12:14:12.563602 (+ 353us) server_negotiation.cc:226] Negotiated authn=SASL
0809 12:14:12.565233 (+ 1631us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 12:14:12.567190 (+ 1957us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 12:14:12.568786 (+ 1596us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 12:14:12.569909 (+ 1123us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 12:14:12.570097 (+ 188us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 12:14:12.574060 (+ 3963us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 12:14:12.574074 (+ 14us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 12:14:12.574094 (+ 20us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0809 12:14:12.574152 (+ 58us) server_negotiation.cc:922] calling sasl_server_start()
0809 12:14:12.577398 (+ 3246us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 12:14:12.578371 (+ 973us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 12:14:12.578378 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 12:14:12.578383 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0809 12:14:12.578549 (+ 166us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 12:14:12.579050 (+ 501us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 12:14:12.579054 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 12:14:12.579056 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0809 12:14:12.579506 (+ 450us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 12:14:12.579659 (+ 153us) server_negotiation.cc:1035] Waiting for connection context
0809 12:14:12.580631 (+ 972us) server_negotiation.cc:299] Negotiation successful
0809 12:14:12.581008 (+ 377us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":798,"thread_start_us":305,"threads_started":1}
I20260809 12:14:12.585172 18385 heartbeater.cc:347] Connected to a master server at 127.0.222.62:45505
I20260809 12:14:12.585484 18385 heartbeater.cc:464] Registering TS with master...
I20260809 12:14:12.586200 18385 heartbeater.cc:511] Master 127.0.222.62:45505 requested a full tablet report, sending...
I20260809 12:14:12.588708 18186 ts_manager.cc:194] Registered new tserver with Master: bd233f0a216e42e7858d2eb736a785bb (127.0.222.1:41049)
I20260809 12:14:12.590936 18186 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.222.1@KRBTEST.COM'} at 127.0.222.1:52339
W20260809 12:14:12.914507 18389 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260809 12:14:12.914824 18389 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260809 12:14:12.914927 18389 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260809 12:14:12.922292 18389 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260809 12:14:12.922406 18389 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260809 12:14:12.922523 18389 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.222.2
I20260809 12:14:12.932474 18389 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.222.20:41481
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskgH5Pjm/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786277382073938-888-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskgH5Pjm/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786277382073938-888-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskgH5Pjm/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786277382073938-888-0/krb5kdc/kudu_127.0.222.2.keytab
--principal=kudu/127.0.222.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.222.2:35933
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskgH5Pjm/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786277382073938-888-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.222.2
--webserver_port=37553
--tserver_master_addrs=127.0.222.62:45505
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.222.2
--log_dir=/tmp/dist-test-taskgH5Pjm/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786277382073938-888-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 09 Aug 2026 11:06:41 UTC on ccycloud.quasar-bwoogy.root.comops.site
build id 4575991
ASAN enabled
I20260809 12:14:12.933816 18389 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260809 12:14:12.935194 18389 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260809 12:14:12.948649 18395 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 12:14:12.948720 18398 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 12:14:12.948707 18396 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260809 12:14:12.949503 18389 server_base.cc:1034] running on GCE node
I20260809 12:14:12.950109 18389 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260809 12:14:12.951215 18389 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260809 12:14:12.952414 18389 hybrid_clock.cc:648] HybridClock initialized: now 1786277652952354 us; error 60 us; skew 500 ppm
I20260809 12:14:12.952730 18389 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 09 12:14:12 dist-test-slave-2w3w krb5kdc[17567](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786277652, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.222.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260809 12:14:12.955924 18389 init.cc:377] Logged in from keytab as kudu/127.0.222.2@KRBTEST.COM (short username kudu)
I20260809 12:14:12.957401 18389 webserver.cc:460] Webserver started at http://127.0.222.2:37553/ using document root <none> and password file <none>
I20260809 12:14:12.958051 18389 fs_manager.cc:359] Metadata directory not provided
I20260809 12:14:12.958171 18389 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260809 12:14:12.964027 18389 fs_manager.cc:709] Time spent opening directory manager: real 0.004s user 0.003s sys 0.001s
I20260809 12:14:12.967092 18406 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260809 12:14:12.968423 18389 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.001s
I20260809 12:14:12.968609 18389 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskgH5Pjm/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786277382073938-888-0/minicluster-data/ts-1/data,/tmp/dist-test-taskgH5Pjm/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786277382073938-888-0/minicluster-data/ts-1/wal
uuid: "a83ae6a077a34a80985b4b65dfa7b6f6"
format_stamp: "Formatted at 2026-08-09 12:13:50 on dist-test-slave-2w3w"
server_key: "df2da0431ccce241d64ece7296ce9d25"
server_key_iv: "90e49b1f6082ddd27cd5c5b866344a61"
server_key_version: "encryptionkey@0"
I20260809 12:14:12.969668 18389 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskgH5Pjm/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786277382073938-888-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskgH5Pjm/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786277382073938-888-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskgH5Pjm/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786277382073938-888-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260809 12:14:12.989334 18389 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260809 12:14:12.994182 18389 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260809 12:14:12.994496 18389 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260809 12:14:12.995612 18389 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260809 12:14:12.997073 18389 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260809 12:14:12.997196 18389 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260809 12:14:12.997340 18389 ts_tablet_manager.cc:610] Registered 0 tablets
I20260809 12:14:12.997434 18389 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260809 12:14:13.033452 18389 rpc_server.cc:307] RPC server started. Bound to: 127.0.222.2:35933
I20260809 12:14:13.033502 18524 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.222.2:35933 every 8 connection(s)
I20260809 12:14:13.034906 18389 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskgH5Pjm/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786277382073938-888-0/minicluster-data/ts-1/data/info.pb
I20260809 12:14:13.042948 888 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskgH5Pjm/build/asan/bin/kudu as pid 18389
I20260809 12:14:13.043651 888 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskgH5Pjm/build/asan/bin/kudu
/tmp/dist-test-taskgH5Pjm/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskgH5Pjm/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786277382073938-888-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskgH5Pjm/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786277382073938-888-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskgH5Pjm/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786277382073938-888-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskgH5Pjm/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786277382073938-888-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.222.3:43523
--local_ip_for_outbound_sockets=127.0.222.3
--tserver_master_addrs=127.0.222.62:45505
--webserver_port=39089
--webserver_interface=127.0.222.3
--keytab_file=/tmp/dist-test-taskgH5Pjm/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786277382073938-888-0/krb5kdc/kudu_127.0.222.3.keytab
--principal=kudu/127.0.222.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.222.20:41481
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskgH5Pjm/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786277382073938-888-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskgH5Pjm/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786277382073938-888-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskgH5Pjm/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786277382073938-888-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Aug 09 12:14:13 dist-test-slave-2w3w krb5kdc[17567](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786277652, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.222.2@KRBTEST.COM for kudu/127.0.222.62@KRBTEST.COM
I20260809 12:14:13.059849 18387 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 12:14:13.038014 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.222.2:50909 (local address 127.0.222.62:45505)
0809 12:14:13.038192 (+ 178us) server_negotiation.cc:206] Beginning negotiation
0809 12:14:13.038196 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0809 12:14:13.039278 (+ 1082us) server_negotiation.cc:407] Connection header received
0809 12:14:13.041120 (+ 1842us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 12:14:13.041126 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 12:14:13.041250 (+ 124us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 12:14:13.041368 (+ 118us) server_negotiation.cc:226] Negotiated authn=SASL
0809 12:14:13.042886 (+ 1518us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 12:14:13.043730 (+ 844us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 12:14:13.046217 (+ 2487us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 12:14:13.046998 (+ 781us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 12:14:13.047129 (+ 131us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 12:14:13.053159 (+ 6030us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 12:14:13.053166 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 12:14:13.053171 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0809 12:14:13.053211 (+ 40us) server_negotiation.cc:922] calling sasl_server_start()
0809 12:14:13.056646 (+ 3435us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 12:14:13.057368 (+ 722us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 12:14:13.057374 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 12:14:13.057378 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0809 12:14:13.057564 (+ 186us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 12:14:13.058089 (+ 525us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 12:14:13.058096 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 12:14:13.058099 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0809 12:14:13.058374 (+ 275us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 12:14:13.058540 (+ 166us) server_negotiation.cc:1035] Waiting for connection context
0809 12:14:13.059255 (+ 715us) server_negotiation.cc:299] Negotiation successful
0809 12:14:13.059477 (+ 222us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":62}
I20260809 12:14:13.060814 18528 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 12:14:13.038367 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.222.62:45505 (local address 127.0.222.2:50909)
0809 12:14:13.039115 (+ 748us) negotiation.cc:107] Waiting for socket to connect
0809 12:14:13.039159 (+ 44us) client_negotiation.cc:174] Beginning negotiation
0809 12:14:13.040846 (+ 1687us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 12:14:13.041618 (+ 772us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 12:14:13.041626 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 12:14:13.042127 (+ 501us) client_negotiation.cc:189] Negotiated authn=SASL
0809 12:14:13.042704 (+ 577us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 12:14:13.042722 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 12:14:13.043961 (+ 1239us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 12:14:13.043966 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 12:14:13.046031 (+ 2065us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 12:14:13.046047 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 12:14:13.047334 (+ 1287us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 12:14:13.047343 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 12:14:13.047740 (+ 397us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 12:14:13.049589 (+ 1849us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0809 12:14:13.049633 (+ 44us) client_negotiation.cc:656] calling sasl_client_start()
0809 12:14:13.052852 (+ 3219us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0809 12:14:13.056789 (+ 3937us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0809 12:14:13.056798 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0809 12:14:13.056815 (+ 17us) client_negotiation.cc:760] calling sasl_client_step()
0809 12:14:13.057231 (+ 416us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0809 12:14:13.057722 (+ 491us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0809 12:14:13.057726 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0809 12:14:13.057729 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0809 12:14:13.057960 (+ 231us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0809 12:14:13.058517 (+ 557us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0809 12:14:13.058525 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0809 12:14:13.058855 (+ 330us) client_negotiation.cc:769] Sending connection context
0809 12:14:13.059184 (+ 329us) client_negotiation.cc:240] Negotiation successful
0809 12:14:13.059704 (+ 520us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":455,"thread_start_us":277,"threads_started":1}
I20260809 12:14:13.062486 18525 heartbeater.cc:347] Connected to a master server at 127.0.222.62:45505
I20260809 12:14:13.062846 18525 heartbeater.cc:464] Registering TS with master...
I20260809 12:14:13.063614 18525 heartbeater.cc:511] Master 127.0.222.62:45505 requested a full tablet report, sending...
I20260809 12:14:13.065475 18186 ts_manager.cc:194] Registered new tserver with Master: a83ae6a077a34a80985b4b65dfa7b6f6 (127.0.222.2:35933)
I20260809 12:14:13.066426 18186 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.222.2@KRBTEST.COM'} at 127.0.222.2:50909
W20260809 12:14:13.391391 18529 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260809 12:14:13.391801 18529 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260809 12:14:13.391987 18529 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260809 12:14:13.399278 18529 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260809 12:14:13.399446 18529 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260809 12:14:13.399647 18529 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.222.3
I20260809 12:14:13.410020 18529 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.222.20:41481
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskgH5Pjm/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786277382073938-888-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskgH5Pjm/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786277382073938-888-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskgH5Pjm/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786277382073938-888-0/krb5kdc/kudu_127.0.222.3.keytab
--principal=kudu/127.0.222.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.222.3:43523
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskgH5Pjm/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786277382073938-888-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.222.3
--webserver_port=39089
--tserver_master_addrs=127.0.222.62:45505
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.222.3
--log_dir=/tmp/dist-test-taskgH5Pjm/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786277382073938-888-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 09 Aug 2026 11:06:41 UTC on ccycloud.quasar-bwoogy.root.comops.site
build id 4575991
ASAN enabled
I20260809 12:14:13.411489 18529 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260809 12:14:13.412849 18529 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260809 12:14:13.426365 18535 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 12:14:13.426445 18537 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 12:14:13.426409 18539 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260809 12:14:13.427749 18529 server_base.cc:1034] running on GCE node
I20260809 12:14:13.428342 18529 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260809 12:14:13.429320 18529 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260809 12:14:13.430557 18529 hybrid_clock.cc:648] HybridClock initialized: now 1786277653430512 us; error 58 us; skew 500 ppm
I20260809 12:14:13.430900 18529 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 09 12:14:13 dist-test-slave-2w3w krb5kdc[17567](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786277653, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.222.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260809 12:14:13.434170 18529 init.cc:377] Logged in from keytab as kudu/127.0.222.3@KRBTEST.COM (short username kudu)
I20260809 12:14:13.435856 18529 webserver.cc:460] Webserver started at http://127.0.222.3:39089/ using document root <none> and password file <none>
I20260809 12:14:13.436568 18529 fs_manager.cc:359] Metadata directory not provided
I20260809 12:14:13.436731 18529 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260809 12:14:13.442201 18529 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.005s sys 0.000s
I20260809 12:14:13.444953 18547 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260809 12:14:13.446295 18529 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260809 12:14:13.446485 18529 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskgH5Pjm/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786277382073938-888-0/minicluster-data/ts-2/data,/tmp/dist-test-taskgH5Pjm/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786277382073938-888-0/minicluster-data/ts-2/wal
uuid: "b97494c2f09f448e85a9497d47d415f6"
format_stamp: "Formatted at 2026-08-09 12:13:50 on dist-test-slave-2w3w"
server_key: "2fb234b39084d1c2b523aaaa022a2ee0"
server_key_iv: "1ee45beb051160c66da10ad87b022316"
server_key_version: "encryptionkey@0"
I20260809 12:14:13.447669 18529 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskgH5Pjm/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786277382073938-888-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskgH5Pjm/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786277382073938-888-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskgH5Pjm/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786277382073938-888-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260809 12:14:13.461769 18529 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260809 12:14:13.466446 18529 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260809 12:14:13.466778 18529 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260809 12:14:13.467824 18529 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260809 12:14:13.469347 18529 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260809 12:14:13.469470 18529 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260809 12:14:13.469602 18529 ts_tablet_manager.cc:610] Registered 0 tablets
I20260809 12:14:13.469702 18529 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260809 12:14:13.503394 18529 rpc_server.cc:307] RPC server started. Bound to: 127.0.222.3:43523
I20260809 12:14:13.503443 18662 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.222.3:43523 every 8 connection(s)
I20260809 12:14:13.505255 18529 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskgH5Pjm/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786277382073938-888-0/minicluster-data/ts-2/data/info.pb
I20260809 12:14:13.507880 888 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskgH5Pjm/build/asan/bin/kudu as pid 18529
Aug 09 12:14:13 dist-test-slave-2w3w krb5kdc[17567](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786277653, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.222.3@KRBTEST.COM for kudu/127.0.222.62@KRBTEST.COM
I20260809 12:14:13.531977 18665 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 12:14:13.508872 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33832 (local address 127.0.222.62:45505)
0809 12:14:13.509332 (+ 460us) server_negotiation.cc:206] Beginning negotiation
0809 12:14:13.509337 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0809 12:14:13.515558 (+ 6221us) server_negotiation.cc:407] Connection header received
0809 12:14:13.515850 (+ 292us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 12:14:13.515858 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 12:14:13.515952 (+ 94us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 12:14:13.516082 (+ 130us) server_negotiation.cc:226] Negotiated authn=SASL
0809 12:14:13.518392 (+ 2310us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 12:14:13.519464 (+ 1072us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 12:14:13.521427 (+ 1963us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 12:14:13.522120 (+ 693us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 12:14:13.522256 (+ 136us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 12:14:13.525656 (+ 3400us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 12:14:13.525660 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 12:14:13.525663 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0809 12:14:13.525692 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0809 12:14:13.529044 (+ 3352us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 12:14:13.529712 (+ 668us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 12:14:13.529719 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 12:14:13.529723 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0809 12:14:13.529868 (+ 145us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 12:14:13.530271 (+ 403us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 12:14:13.530281 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 12:14:13.530284 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0809 12:14:13.530592 (+ 308us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 12:14:13.530897 (+ 305us) server_negotiation.cc:1035] Waiting for connection context
0809 12:14:13.531290 (+ 393us) server_negotiation.cc:299] Negotiation successful
0809 12:14:13.531593 (+ 303us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":27,"server-negotiator.queue_time_us":376,"thread_start_us":255,"threads_started":1}
I20260809 12:14:13.535640 18387 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 12:14:13.508100 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.222.3:54207 (local address 127.0.222.62:45505)
0809 12:14:13.508323 (+ 223us) server_negotiation.cc:206] Beginning negotiation
0809 12:14:13.508329 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0809 12:14:13.516373 (+ 8044us) server_negotiation.cc:407] Connection header received
0809 12:14:13.519028 (+ 2655us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 12:14:13.519035 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 12:14:13.519116 (+ 81us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 12:14:13.519220 (+ 104us) server_negotiation.cc:226] Negotiated authn=SASL
0809 12:14:13.521225 (+ 2005us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 12:14:13.522354 (+ 1129us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 12:14:13.524861 (+ 2507us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 12:14:13.525259 (+ 398us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 12:14:13.525359 (+ 100us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 12:14:13.529077 (+ 3718us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 12:14:13.529086 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 12:14:13.529093 (+ 7us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0809 12:14:13.529134 (+ 41us) server_negotiation.cc:922] calling sasl_server_start()
0809 12:14:13.532060 (+ 2926us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 12:14:13.533056 (+ 996us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 12:14:13.533060 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 12:14:13.533062 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0809 12:14:13.533185 (+ 123us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 12:14:13.533721 (+ 536us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 12:14:13.533727 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 12:14:13.533730 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0809 12:14:13.534114 (+ 384us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 12:14:13.534270 (+ 156us) server_negotiation.cc:1035] Waiting for connection context
0809 12:14:13.535238 (+ 968us) server_negotiation.cc:299] Negotiation successful
0809 12:14:13.535401 (+ 163us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":2,"server-negotiator.queue_time_us":85}
I20260809 12:14:13.536720 18666 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 12:14:13.508669 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.222.62:45505 (local address 127.0.222.3:54207)
0809 12:14:13.516184 (+ 7515us) negotiation.cc:107] Waiting for socket to connect
0809 12:14:13.516239 (+ 55us) client_negotiation.cc:174] Beginning negotiation
0809 12:14:13.518733 (+ 2494us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 12:14:13.519454 (+ 721us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 12:14:13.519463 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 12:14:13.520232 (+ 769us) client_negotiation.cc:189] Negotiated authn=SASL
0809 12:14:13.521036 (+ 804us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 12:14:13.521063 (+ 27us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 12:14:13.522537 (+ 1474us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 12:14:13.522543 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 12:14:13.524687 (+ 2144us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 12:14:13.524711 (+ 24us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 12:14:13.525506 (+ 795us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 12:14:13.525517 (+ 11us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 12:14:13.525777 (+ 260us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 12:14:13.526422 (+ 645us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0809 12:14:13.526444 (+ 22us) client_negotiation.cc:656] calling sasl_client_start()
0809 12:14:13.528870 (+ 2426us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0809 12:14:13.532250 (+ 3380us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0809 12:14:13.532258 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0809 12:14:13.532274 (+ 16us) client_negotiation.cc:760] calling sasl_client_step()
0809 12:14:13.532922 (+ 648us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0809 12:14:13.533307 (+ 385us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0809 12:14:13.533312 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0809 12:14:13.533315 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0809 12:14:13.533572 (+ 257us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0809 12:14:13.534245 (+ 673us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0809 12:14:13.534250 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0809 12:14:13.534736 (+ 486us) client_negotiation.cc:769] Sending connection context
0809 12:14:13.535139 (+ 403us) client_negotiation.cc:240] Negotiation successful
0809 12:14:13.535540 (+ 401us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":7194,"thread_start_us":6962,"threads_started":1}
I20260809 12:14:13.538107 18663 heartbeater.cc:347] Connected to a master server at 127.0.222.62:45505
I20260809 12:14:13.538427 18663 heartbeater.cc:464] Registering TS with master...
I20260809 12:14:13.539202 18663 heartbeater.cc:511] Master 127.0.222.62:45505 requested a full tablet report, sending...
I20260809 12:14:13.540585 18186 ts_manager.cc:194] Registered new tserver with Master: b97494c2f09f448e85a9497d47d415f6 (127.0.222.3:43523)
I20260809 12:14:13.541410 18186 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.222.3@KRBTEST.COM'} at 127.0.222.3:54207
I20260809 12:14:13.547219 888 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260809 12:14:13.553917 18387 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 12:14:13.548184 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33842 (local address 127.0.222.62:45505)
0809 12:14:13.548351 (+ 167us) server_negotiation.cc:206] Beginning negotiation
0809 12:14:13.548356 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0809 12:14:13.548827 (+ 471us) server_negotiation.cc:407] Connection header received
0809 12:14:13.549019 (+ 192us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 12:14:13.549022 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 12:14:13.549150 (+ 128us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 12:14:13.549273 (+ 123us) server_negotiation.cc:226] Negotiated authn=TOKEN
0809 12:14:13.549760 (+ 487us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 12:14:13.550494 (+ 734us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 12:14:13.552111 (+ 1617us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 12:14:13.552578 (+ 467us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 12:14:13.552674 (+ 96us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 12:14:13.553174 (+ 500us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0809 12:14:13.553430 (+ 256us) server_negotiation.cc:394] Sending RPC error: FATAL_INVALID_AUTHENTICATION_TOKEN: Not authorized: token expired
0809 12:14:13.553703 (+ 273us) negotiation.cc:327] Negotiation complete: Not authorized: Server connection negotiation failed: server connection from 127.0.0.1:33842: token expired
Metrics: {"server-negotiator.queue_time_us":63}
W20260809 12:14:13.554752 18107 master_proxy_rpc.cc:203] Re-attempting CreateTable request to leader Master (127.0.222.62:45505)
W20260809 12:14:13.555060 18387 negotiation.cc:344] Unauthorized connection attempt: Server connection negotiation failed: server connection from 127.0.0.1:33842: token expired
I20260809 12:14:13.568347 18387 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 12:14:13.555433 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33848 (local address 127.0.222.62:45505)
0809 12:14:13.555605 (+ 172us) server_negotiation.cc:206] Beginning negotiation
0809 12:14:13.555611 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0809 12:14:13.555630 (+ 19us) server_negotiation.cc:407] Connection header received
0809 12:14:13.555788 (+ 158us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 12:14:13.555793 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 12:14:13.555870 (+ 77us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 12:14:13.555973 (+ 103us) server_negotiation.cc:226] Negotiated authn=SASL
0809 12:14:13.557422 (+ 1449us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 12:14:13.558237 (+ 815us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 12:14:13.559738 (+ 1501us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 12:14:13.560137 (+ 399us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 12:14:13.560273 (+ 136us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 12:14:13.563004 (+ 2731us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 12:14:13.563012 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 12:14:13.563017 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0809 12:14:13.563086 (+ 69us) server_negotiation.cc:922] calling sasl_server_start()
0809 12:14:13.565513 (+ 2427us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 12:14:13.566242 (+ 729us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 12:14:13.566247 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 12:14:13.566254 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0809 12:14:13.566384 (+ 130us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 12:14:13.566808 (+ 424us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 12:14:13.566813 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 12:14:13.566816 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0809 12:14:13.567198 (+ 382us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 12:14:13.567458 (+ 260us) server_negotiation.cc:1035] Waiting for connection context
0809 12:14:13.567774 (+ 316us) server_negotiation.cc:299] Negotiation successful
0809 12:14:13.567997 (+ 223us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":60}
I20260809 12:14:13.571563 18186 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:33848:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260809 12:14:13.578866 18186 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260809 12:14:13.594672 18385 heartbeater.cc:503] Master 127.0.222.62:45505 was elected leader, sending a full tablet report...
I20260809 12:14:13.612919 18678 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 12:14:13.591543 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56242 (local address 127.0.222.3:43523)
0809 12:14:13.604304 (+ 12761us) server_negotiation.cc:206] Beginning negotiation
0809 12:14:13.604315 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0809 12:14:13.604356 (+ 41us) server_negotiation.cc:407] Connection header received
0809 12:14:13.604461 (+ 105us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 12:14:13.604467 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 12:14:13.604679 (+ 212us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 12:14:13.604911 (+ 232us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0809 12:14:13.605710 (+ 799us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 12:14:13.607089 (+ 1379us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 12:14:13.610529 (+ 3440us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 12:14:13.611521 (+ 992us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 12:14:13.611676 (+ 155us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 12:14:13.611924 (+ 248us) server_negotiation.cc:1035] Waiting for connection context
0809 12:14:13.612339 (+ 415us) server_negotiation.cc:299] Negotiation successful
0809 12:14:13.612547 (+ 208us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":12254,"thread_start_us":12083,"threads_started":1}
I20260809 12:14:13.612938 18677 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 12:14:13.591069 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.222.2:35933 (local address 127.0.0.1:57654)
0809 12:14:13.603174 (+ 12105us) negotiation.cc:107] Waiting for socket to connect
0809 12:14:13.603198 (+ 24us) client_negotiation.cc:174] Beginning negotiation
0809 12:14:13.603346 (+ 148us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 12:14:13.605328 (+ 1982us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 12:14:13.605335 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 12:14:13.605374 (+ 39us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0809 12:14:13.605801 (+ 427us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 12:14:13.605814 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 12:14:13.607620 (+ 1806us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 12:14:13.607628 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 12:14:13.609999 (+ 2371us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 12:14:13.610012 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 12:14:13.611881 (+ 1869us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 12:14:13.611888 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 12:14:13.612070 (+ 182us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 12:14:13.612104 (+ 34us) client_negotiation.cc:769] Sending connection context
0809 12:14:13.612174 (+ 70us) client_negotiation.cc:240] Negotiation successful
0809 12:14:13.612400 (+ 226us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":11941,"mutex_wait_us":84,"thread_start_us":12931,"threads_started":1}
I20260809 12:14:13.613039 18675 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 12:14:13.591560 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.222.1:41049 (local address 127.0.0.1:35056)
0809 12:14:13.603564 (+ 12004us) negotiation.cc:107] Waiting for socket to connect
0809 12:14:13.603585 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0809 12:14:13.603716 (+ 131us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 12:14:13.604761 (+ 1045us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 12:14:13.604769 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 12:14:13.604802 (+ 33us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0809 12:14:13.605404 (+ 602us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 12:14:13.605426 (+ 22us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 12:14:13.607342 (+ 1916us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 12:14:13.607350 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 12:14:13.610329 (+ 2979us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 12:14:13.610345 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 12:14:13.612178 (+ 1833us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 12:14:13.612184 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 12:14:13.612339 (+ 155us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 12:14:13.612372 (+ 33us) client_negotiation.cc:769] Sending connection context
0809 12:14:13.612473 (+ 101us) client_negotiation.cc:240] Negotiation successful
0809 12:14:13.612591 (+ 118us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":11829,"mutex_wait_us":223,"thread_start_us":11280,"threads_started":1}
I20260809 12:14:13.613554 18679 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 12:14:13.592391 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35056 (local address 127.0.222.1:41049)
0809 12:14:13.602526 (+ 10135us) server_negotiation.cc:206] Beginning negotiation
0809 12:14:13.602535 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0809 12:14:13.603862 (+ 1327us) server_negotiation.cc:407] Connection header received
0809 12:14:13.603986 (+ 124us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 12:14:13.603995 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 12:14:13.604290 (+ 295us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 12:14:13.604499 (+ 209us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0809 12:14:13.605639 (+ 1140us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 12:14:13.607228 (+ 1589us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 12:14:13.610505 (+ 3277us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 12:14:13.611958 (+ 1453us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 12:14:13.612085 (+ 127us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 12:14:13.612321 (+ 236us) server_negotiation.cc:1035] Waiting for connection context
0809 12:14:13.612527 (+ 206us) server_negotiation.cc:299] Negotiation successful
0809 12:14:13.612721 (+ 194us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":9955,"thread_start_us":9834,"threads_started":1}
I20260809 12:14:13.613608 18674 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 12:14:13.590593 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.222.3:43523 (local address 127.0.0.1:56242)
0809 12:14:13.600490 (+ 9897us) negotiation.cc:107] Waiting for socket to connect
0809 12:14:13.600554 (+ 64us) client_negotiation.cc:174] Beginning negotiation
0809 12:14:13.600756 (+ 202us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 12:14:13.604900 (+ 4144us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 12:14:13.604906 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 12:14:13.604937 (+ 31us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0809 12:14:13.605453 (+ 516us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 12:14:13.605470 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 12:14:13.607226 (+ 1756us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 12:14:13.607233 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 12:14:13.610329 (+ 3096us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 12:14:13.610345 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 12:14:13.611749 (+ 1404us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 12:14:13.611755 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 12:14:13.611967 (+ 212us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 12:14:13.612022 (+ 55us) client_negotiation.cc:769] Sending connection context
0809 12:14:13.612169 (+ 147us) client_negotiation.cc:240] Negotiation successful
0809 12:14:13.613264 (+ 1095us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":9564,"mutex_wait_us":1031,"thread_start_us":448,"threads_started":1}
I20260809 12:14:13.614180 18676 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 12:14:13.591490 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57654 (local address 127.0.222.2:35933)
0809 12:14:13.604724 (+ 13234us) server_negotiation.cc:206] Beginning negotiation
0809 12:14:13.604735 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0809 12:14:13.604762 (+ 27us) server_negotiation.cc:407] Connection header received
0809 12:14:13.604852 (+ 90us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 12:14:13.604862 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 12:14:13.605106 (+ 244us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 12:14:13.605313 (+ 207us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0809 12:14:13.605987 (+ 674us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 12:14:13.607487 (+ 1500us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 12:14:13.610161 (+ 2674us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 12:14:13.611620 (+ 1459us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 12:14:13.613400 (+ 1780us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 12:14:13.613590 (+ 190us) server_negotiation.cc:1035] Waiting for connection context
0809 12:14:13.613661 (+ 71us) server_negotiation.cc:299] Negotiation successful
0809 12:14:13.613826 (+ 165us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":13071,"thread_start_us":12965,"threads_started":1}
I20260809 12:14:13.616163 18314 tablet_service.cc:1467] Processing CreateTablet for tablet efb1fe44ea224877bd89ce3b81be955e (DEFAULT_TABLE table=test-table [id=7f6c197c48294e418fbd7b2737188c74]), partition=RANGE (key) PARTITION UNBOUNDED
I20260809 12:14:13.617677 18314 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet efb1fe44ea224877bd89ce3b81be955e. 1 dirs total, 0 dirs full, 0 dirs failed
I20260809 12:14:13.617370 18596 tablet_service.cc:1467] Processing CreateTablet for tablet efb1fe44ea224877bd89ce3b81be955e (DEFAULT_TABLE table=test-table [id=7f6c197c48294e418fbd7b2737188c74]), partition=RANGE (key) PARTITION UNBOUNDED
I20260809 12:14:13.617857 18457 tablet_service.cc:1467] Processing CreateTablet for tablet efb1fe44ea224877bd89ce3b81be955e (DEFAULT_TABLE table=test-table [id=7f6c197c48294e418fbd7b2737188c74]), partition=RANGE (key) PARTITION UNBOUNDED
I20260809 12:14:13.618546 18596 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet efb1fe44ea224877bd89ce3b81be955e. 1 dirs total, 0 dirs full, 0 dirs failed
I20260809 12:14:13.619174 18457 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet efb1fe44ea224877bd89ce3b81be955e. 1 dirs total, 0 dirs full, 0 dirs failed
I20260809 12:14:13.628412 18680 tablet_bootstrap.cc:492] T efb1fe44ea224877bd89ce3b81be955e P bd233f0a216e42e7858d2eb736a785bb: Bootstrap starting.
I20260809 12:14:13.632508 18682 tablet_bootstrap.cc:492] T efb1fe44ea224877bd89ce3b81be955e P b97494c2f09f448e85a9497d47d415f6: Bootstrap starting.
I20260809 12:14:13.632915 18680 tablet_bootstrap.cc:654] T efb1fe44ea224877bd89ce3b81be955e P bd233f0a216e42e7858d2eb736a785bb: Neither blocks nor log segments found. Creating new log.
I20260809 12:14:13.634176 18683 tablet_bootstrap.cc:492] T efb1fe44ea224877bd89ce3b81be955e P a83ae6a077a34a80985b4b65dfa7b6f6: Bootstrap starting.
I20260809 12:14:13.634200 18680 log.cc:824] T efb1fe44ea224877bd89ce3b81be955e P bd233f0a216e42e7858d2eb736a785bb: Log is configured to *not* fsync() on all Append() calls
I20260809 12:14:13.636377 18682 tablet_bootstrap.cc:654] T efb1fe44ea224877bd89ce3b81be955e P b97494c2f09f448e85a9497d47d415f6: Neither blocks nor log segments found. Creating new log.
I20260809 12:14:13.637362 18682 log.cc:824] T efb1fe44ea224877bd89ce3b81be955e P b97494c2f09f448e85a9497d47d415f6: Log is configured to *not* fsync() on all Append() calls
I20260809 12:14:13.637776 18680 tablet_bootstrap.cc:492] T efb1fe44ea224877bd89ce3b81be955e P bd233f0a216e42e7858d2eb736a785bb: No bootstrap required, opened a new log
I20260809 12:14:13.637768 18683 tablet_bootstrap.cc:654] T efb1fe44ea224877bd89ce3b81be955e P a83ae6a077a34a80985b4b65dfa7b6f6: Neither blocks nor log segments found. Creating new log.
I20260809 12:14:13.638036 18680 ts_tablet_manager.cc:1397] T efb1fe44ea224877bd89ce3b81be955e P bd233f0a216e42e7858d2eb736a785bb: Time spent bootstrapping tablet: real 0.010s user 0.007s sys 0.000s
I20260809 12:14:13.639063 18683 log.cc:824] T efb1fe44ea224877bd89ce3b81be955e P a83ae6a077a34a80985b4b65dfa7b6f6: Log is configured to *not* fsync() on all Append() calls
I20260809 12:14:13.641252 18682 tablet_bootstrap.cc:492] T efb1fe44ea224877bd89ce3b81be955e P b97494c2f09f448e85a9497d47d415f6: No bootstrap required, opened a new log
I20260809 12:14:13.641530 18682 ts_tablet_manager.cc:1397] T efb1fe44ea224877bd89ce3b81be955e P b97494c2f09f448e85a9497d47d415f6: Time spent bootstrapping tablet: real 0.009s user 0.002s sys 0.006s
I20260809 12:14:13.644701 18683 tablet_bootstrap.cc:492] T efb1fe44ea224877bd89ce3b81be955e P a83ae6a077a34a80985b4b65dfa7b6f6: No bootstrap required, opened a new log
I20260809 12:14:13.644963 18683 ts_tablet_manager.cc:1397] T efb1fe44ea224877bd89ce3b81be955e P a83ae6a077a34a80985b4b65dfa7b6f6: Time spent bootstrapping tablet: real 0.011s user 0.008s sys 0.000s
I20260809 12:14:13.648967 18682 raft_consensus.cc:348] T efb1fe44ea224877bd89ce3b81be955e P b97494c2f09f448e85a9497d47d415f6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b97494c2f09f448e85a9497d47d415f6" member_type: VOTER last_known_addr { host: "127.0.222.3" port: 43523 } } peers { permanent_uuid: "a83ae6a077a34a80985b4b65dfa7b6f6" member_type: VOTER last_known_addr { host: "127.0.222.2" port: 35933 } } peers { permanent_uuid: "bd233f0a216e42e7858d2eb736a785bb" member_type: VOTER last_known_addr { host: "127.0.222.1" port: 41049 } }
I20260809 12:14:13.649406 18682 raft_consensus.cc:374] T efb1fe44ea224877bd89ce3b81be955e P b97494c2f09f448e85a9497d47d415f6 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260809 12:14:13.649535 18682 raft_consensus.cc:729] T efb1fe44ea224877bd89ce3b81be955e P b97494c2f09f448e85a9497d47d415f6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b97494c2f09f448e85a9497d47d415f6, State: Initialized, Role: FOLLOWER
I20260809 12:14:13.649609 18680 raft_consensus.cc:348] T efb1fe44ea224877bd89ce3b81be955e P bd233f0a216e42e7858d2eb736a785bb [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b97494c2f09f448e85a9497d47d415f6" member_type: VOTER last_known_addr { host: "127.0.222.3" port: 43523 } } peers { permanent_uuid: "a83ae6a077a34a80985b4b65dfa7b6f6" member_type: VOTER last_known_addr { host: "127.0.222.2" port: 35933 } } peers { permanent_uuid: "bd233f0a216e42e7858d2eb736a785bb" member_type: VOTER last_known_addr { host: "127.0.222.1" port: 41049 } }
I20260809 12:14:13.650058 18680 raft_consensus.cc:374] T efb1fe44ea224877bd89ce3b81be955e P bd233f0a216e42e7858d2eb736a785bb [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260809 12:14:13.650163 18680 raft_consensus.cc:729] T efb1fe44ea224877bd89ce3b81be955e P bd233f0a216e42e7858d2eb736a785bb [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: bd233f0a216e42e7858d2eb736a785bb, State: Initialized, Role: FOLLOWER
I20260809 12:14:13.650214 18682 consensus_queue.cc:260] T efb1fe44ea224877bd89ce3b81be955e P b97494c2f09f448e85a9497d47d415f6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b97494c2f09f448e85a9497d47d415f6" member_type: VOTER last_known_addr { host: "127.0.222.3" port: 43523 } } peers { permanent_uuid: "a83ae6a077a34a80985b4b65dfa7b6f6" member_type: VOTER last_known_addr { host: "127.0.222.2" port: 35933 } } peers { permanent_uuid: "bd233f0a216e42e7858d2eb736a785bb" member_type: VOTER last_known_addr { host: "127.0.222.1" port: 41049 } }
I20260809 12:14:13.650791 18680 consensus_queue.cc:260] T efb1fe44ea224877bd89ce3b81be955e P bd233f0a216e42e7858d2eb736a785bb [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b97494c2f09f448e85a9497d47d415f6" member_type: VOTER last_known_addr { host: "127.0.222.3" port: 43523 } } peers { permanent_uuid: "a83ae6a077a34a80985b4b65dfa7b6f6" member_type: VOTER last_known_addr { host: "127.0.222.2" port: 35933 } } peers { permanent_uuid: "bd233f0a216e42e7858d2eb736a785bb" member_type: VOTER last_known_addr { host: "127.0.222.1" port: 41049 } }
I20260809 12:14:13.651522 18663 heartbeater.cc:503] Master 127.0.222.62:45505 was elected leader, sending a full tablet report...
I20260809 12:14:13.652179 18682 ts_tablet_manager.cc:1428] T efb1fe44ea224877bd89ce3b81be955e P b97494c2f09f448e85a9497d47d415f6: Time spent starting tablet: real 0.011s user 0.006s sys 0.003s
I20260809 12:14:13.657526 18683 raft_consensus.cc:348] T efb1fe44ea224877bd89ce3b81be955e P a83ae6a077a34a80985b4b65dfa7b6f6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b97494c2f09f448e85a9497d47d415f6" member_type: VOTER last_known_addr { host: "127.0.222.3" port: 43523 } } peers { permanent_uuid: "a83ae6a077a34a80985b4b65dfa7b6f6" member_type: VOTER last_known_addr { host: "127.0.222.2" port: 35933 } } peers { permanent_uuid: "bd233f0a216e42e7858d2eb736a785bb" member_type: VOTER last_known_addr { host: "127.0.222.1" port: 41049 } }
I20260809 12:14:13.658100 18683 raft_consensus.cc:374] T efb1fe44ea224877bd89ce3b81be955e P a83ae6a077a34a80985b4b65dfa7b6f6 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260809 12:14:13.658226 18683 raft_consensus.cc:729] T efb1fe44ea224877bd89ce3b81be955e P a83ae6a077a34a80985b4b65dfa7b6f6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a83ae6a077a34a80985b4b65dfa7b6f6, State: Initialized, Role: FOLLOWER
I20260809 12:14:13.658993 18683 consensus_queue.cc:260] T efb1fe44ea224877bd89ce3b81be955e P a83ae6a077a34a80985b4b65dfa7b6f6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b97494c2f09f448e85a9497d47d415f6" member_type: VOTER last_known_addr { host: "127.0.222.3" port: 43523 } } peers { permanent_uuid: "a83ae6a077a34a80985b4b65dfa7b6f6" member_type: VOTER last_known_addr { host: "127.0.222.2" port: 35933 } } peers { permanent_uuid: "bd233f0a216e42e7858d2eb736a785bb" member_type: VOTER last_known_addr { host: "127.0.222.1" port: 41049 } }
I20260809 12:14:13.660583 18680 ts_tablet_manager.cc:1428] T efb1fe44ea224877bd89ce3b81be955e P bd233f0a216e42e7858d2eb736a785bb: Time spent starting tablet: real 0.022s user 0.012s sys 0.011s
I20260809 12:14:13.661063 18525 heartbeater.cc:503] Master 127.0.222.62:45505 was elected leader, sending a full tablet report...
I20260809 12:14:13.661681 18683 ts_tablet_manager.cc:1428] T efb1fe44ea224877bd89ce3b81be955e P a83ae6a077a34a80985b4b65dfa7b6f6: Time spent starting tablet: real 0.017s user 0.013s sys 0.004s
W20260809 12:14:13.756978 18664 tablet.cc:2369] T efb1fe44ea224877bd89ce3b81be955e P b97494c2f09f448e85a9497d47d415f6: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260809 12:14:13.787074 18527 tablet.cc:2369] T efb1fe44ea224877bd89ce3b81be955e P a83ae6a077a34a80985b4b65dfa7b6f6: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260809 12:14:13.808324 18386 tablet.cc:2369] T efb1fe44ea224877bd89ce3b81be955e P bd233f0a216e42e7858d2eb736a785bb: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260809 12:14:13.936326 18687 raft_consensus.cc:482] T efb1fe44ea224877bd89ce3b81be955e P b97494c2f09f448e85a9497d47d415f6 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260809 12:14:13.936635 18687 raft_consensus.cc:504] T efb1fe44ea224877bd89ce3b81be955e P b97494c2f09f448e85a9497d47d415f6 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b97494c2f09f448e85a9497d47d415f6" member_type: VOTER last_known_addr { host: "127.0.222.3" port: 43523 } } peers { permanent_uuid: "a83ae6a077a34a80985b4b65dfa7b6f6" member_type: VOTER last_known_addr { host: "127.0.222.2" port: 35933 } } peers { permanent_uuid: "bd233f0a216e42e7858d2eb736a785bb" member_type: VOTER last_known_addr { host: "127.0.222.1" port: 41049 } }
I20260809 12:14:13.938416 18687 leader_election.cc:288] T efb1fe44ea224877bd89ce3b81be955e P b97494c2f09f448e85a9497d47d415f6 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers a83ae6a077a34a80985b4b65dfa7b6f6 (127.0.222.2:35933), bd233f0a216e42e7858d2eb736a785bb (127.0.222.1:41049)
I20260809 12:14:13.945680 18679 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 12:14:13.939100 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.222.3:47829 (local address 127.0.222.1:41049)
0809 12:14:13.939330 (+ 230us) server_negotiation.cc:206] Beginning negotiation
0809 12:14:13.939337 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0809 12:14:13.939363 (+ 26us) server_negotiation.cc:407] Connection header received
0809 12:14:13.939599 (+ 236us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 12:14:13.939605 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 12:14:13.939691 (+ 86us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 12:14:13.939803 (+ 112us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0809 12:14:13.940498 (+ 695us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 12:14:13.941501 (+ 1003us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 12:14:13.943891 (+ 2390us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 12:14:13.944843 (+ 952us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 12:14:13.944946 (+ 103us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 12:14:13.945027 (+ 81us) server_negotiation.cc:1035] Waiting for connection context
0809 12:14:13.945394 (+ 367us) server_negotiation.cc:299] Negotiation successful
0809 12:14:13.945498 (+ 104us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":79}
I20260809 12:14:13.945811 18666 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 12:14:13.938948 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.222.1:41049 (local address 127.0.222.3:47829)
0809 12:14:13.939231 (+ 283us) negotiation.cc:107] Waiting for socket to connect
0809 12:14:13.939255 (+ 24us) client_negotiation.cc:174] Beginning negotiation
0809 12:14:13.939411 (+ 156us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 12:14:13.940018 (+ 607us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 12:14:13.940024 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 12:14:13.940060 (+ 36us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0809 12:14:13.940283 (+ 223us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 12:14:13.940298 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 12:14:13.941649 (+ 1351us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 12:14:13.941653 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 12:14:13.943710 (+ 2057us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 12:14:13.943726 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 12:14:13.944998 (+ 1272us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 12:14:13.945003 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 12:14:13.945210 (+ 207us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 12:14:13.945249 (+ 39us) client_negotiation.cc:769] Sending connection context
0809 12:14:13.945364 (+ 115us) client_negotiation.cc:240] Negotiation successful
0809 12:14:13.945506 (+ 142us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":86,"mutex_wait_us":42}
I20260809 12:14:13.946756 18690 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 12:14:13.939461 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.222.2:35933 (local address 127.0.222.3:57587)
0809 12:14:13.940268 (+ 807us) negotiation.cc:107] Waiting for socket to connect
0809 12:14:13.940286 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0809 12:14:13.940420 (+ 134us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 12:14:13.940764 (+ 344us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 12:14:13.940767 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 12:14:13.940787 (+ 20us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0809 12:14:13.941107 (+ 320us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 12:14:13.941115 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 12:14:13.942239 (+ 1124us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 12:14:13.942244 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 12:14:13.944255 (+ 2011us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 12:14:13.944271 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 12:14:13.945972 (+ 1701us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 12:14:13.945976 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 12:14:13.946218 (+ 242us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 12:14:13.946255 (+ 37us) client_negotiation.cc:769] Sending connection context
0809 12:14:13.946326 (+ 71us) client_negotiation.cc:240] Negotiation successful
0809 12:14:13.946436 (+ 110us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":677,"mutex_wait_us":67,"thread_start_us":411,"threads_started":1}
I20260809 12:14:13.946786 18676 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 12:14:13.939660 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.222.3:57587 (local address 127.0.222.2:35933)
0809 12:14:13.939942 (+ 282us) server_negotiation.cc:206] Beginning negotiation
0809 12:14:13.939949 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0809 12:14:13.940397 (+ 448us) server_negotiation.cc:407] Connection header received
0809 12:14:13.940561 (+ 164us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 12:14:13.940569 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 12:14:13.940646 (+ 77us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 12:14:13.940748 (+ 102us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0809 12:14:13.941271 (+ 523us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 12:14:13.942098 (+ 827us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 12:14:13.944484 (+ 2386us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 12:14:13.945663 (+ 1179us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 12:14:13.946082 (+ 419us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 12:14:13.946211 (+ 129us) server_negotiation.cc:1035] Waiting for connection context
0809 12:14:13.946414 (+ 203us) server_negotiation.cc:299] Negotiation successful
0809 12:14:13.946553 (+ 139us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":82}
I20260809 12:14:13.947232 18336 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "efb1fe44ea224877bd89ce3b81be955e" candidate_uuid: "b97494c2f09f448e85a9497d47d415f6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "bd233f0a216e42e7858d2eb736a785bb" is_pre_election: true
I20260809 12:14:13.947880 18336 raft_consensus.cc:2454] T efb1fe44ea224877bd89ce3b81be955e P bd233f0a216e42e7858d2eb736a785bb [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b97494c2f09f448e85a9497d47d415f6 in term 0.
I20260809 12:14:13.947854 18478 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "efb1fe44ea224877bd89ce3b81be955e" candidate_uuid: "b97494c2f09f448e85a9497d47d415f6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a83ae6a077a34a80985b4b65dfa7b6f6" is_pre_election: true
I20260809 12:14:13.948442 18478 raft_consensus.cc:2454] T efb1fe44ea224877bd89ce3b81be955e P a83ae6a077a34a80985b4b65dfa7b6f6 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b97494c2f09f448e85a9497d47d415f6 in term 0.
I20260809 12:14:13.948719 18549 leader_election.cc:302] T efb1fe44ea224877bd89ce3b81be955e P b97494c2f09f448e85a9497d47d415f6 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: b97494c2f09f448e85a9497d47d415f6, bd233f0a216e42e7858d2eb736a785bb; no voters:
I20260809 12:14:13.949172 18687 raft_consensus.cc:2781] T efb1fe44ea224877bd89ce3b81be955e P b97494c2f09f448e85a9497d47d415f6 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260809 12:14:13.949368 18687 raft_consensus.cc:482] T efb1fe44ea224877bd89ce3b81be955e P b97494c2f09f448e85a9497d47d415f6 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260809 12:14:13.949501 18687 raft_consensus.cc:3037] T efb1fe44ea224877bd89ce3b81be955e P b97494c2f09f448e85a9497d47d415f6 [term 0 FOLLOWER]: Advancing to term 1
I20260809 12:14:13.952023 18687 raft_consensus.cc:504] T efb1fe44ea224877bd89ce3b81be955e P b97494c2f09f448e85a9497d47d415f6 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b97494c2f09f448e85a9497d47d415f6" member_type: VOTER last_known_addr { host: "127.0.222.3" port: 43523 } } peers { permanent_uuid: "a83ae6a077a34a80985b4b65dfa7b6f6" member_type: VOTER last_known_addr { host: "127.0.222.2" port: 35933 } } peers { permanent_uuid: "bd233f0a216e42e7858d2eb736a785bb" member_type: VOTER last_known_addr { host: "127.0.222.1" port: 41049 } }
I20260809 12:14:13.952683 18687 leader_election.cc:288] T efb1fe44ea224877bd89ce3b81be955e P b97494c2f09f448e85a9497d47d415f6 [CANDIDATE]: Term 1 election: Requested vote from peers a83ae6a077a34a80985b4b65dfa7b6f6 (127.0.222.2:35933), bd233f0a216e42e7858d2eb736a785bb (127.0.222.1:41049)
I20260809 12:14:13.953152 18336 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "efb1fe44ea224877bd89ce3b81be955e" candidate_uuid: "b97494c2f09f448e85a9497d47d415f6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "bd233f0a216e42e7858d2eb736a785bb"
I20260809 12:14:13.953186 18478 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "efb1fe44ea224877bd89ce3b81be955e" candidate_uuid: "b97494c2f09f448e85a9497d47d415f6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a83ae6a077a34a80985b4b65dfa7b6f6"
I20260809 12:14:13.953358 18336 raft_consensus.cc:3037] T efb1fe44ea224877bd89ce3b81be955e P bd233f0a216e42e7858d2eb736a785bb [term 0 FOLLOWER]: Advancing to term 1
I20260809 12:14:13.953399 18478 raft_consensus.cc:3037] T efb1fe44ea224877bd89ce3b81be955e P a83ae6a077a34a80985b4b65dfa7b6f6 [term 0 FOLLOWER]: Advancing to term 1
I20260809 12:14:13.956012 18336 raft_consensus.cc:2454] T efb1fe44ea224877bd89ce3b81be955e P bd233f0a216e42e7858d2eb736a785bb [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b97494c2f09f448e85a9497d47d415f6 in term 1.
I20260809 12:14:13.956164 18478 raft_consensus.cc:2454] T efb1fe44ea224877bd89ce3b81be955e P a83ae6a077a34a80985b4b65dfa7b6f6 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b97494c2f09f448e85a9497d47d415f6 in term 1.
I20260809 12:14:13.956555 18549 leader_election.cc:302] T efb1fe44ea224877bd89ce3b81be955e P b97494c2f09f448e85a9497d47d415f6 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: b97494c2f09f448e85a9497d47d415f6, bd233f0a216e42e7858d2eb736a785bb; no voters:
I20260809 12:14:13.956852 18687 raft_consensus.cc:2781] T efb1fe44ea224877bd89ce3b81be955e P b97494c2f09f448e85a9497d47d415f6 [term 1 FOLLOWER]: Leader election won for term 1
I20260809 12:14:13.957255 18687 raft_consensus.cc:686] T efb1fe44ea224877bd89ce3b81be955e P b97494c2f09f448e85a9497d47d415f6 [term 1 LEADER]: Becoming Leader. State: Replica: b97494c2f09f448e85a9497d47d415f6, State: Running, Role: LEADER
I20260809 12:14:13.957721 18687 consensus_queue.cc:237] T efb1fe44ea224877bd89ce3b81be955e P b97494c2f09f448e85a9497d47d415f6 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b97494c2f09f448e85a9497d47d415f6" member_type: VOTER last_known_addr { host: "127.0.222.3" port: 43523 } } peers { permanent_uuid: "a83ae6a077a34a80985b4b65dfa7b6f6" member_type: VOTER last_known_addr { host: "127.0.222.2" port: 35933 } } peers { permanent_uuid: "bd233f0a216e42e7858d2eb736a785bb" member_type: VOTER last_known_addr { host: "127.0.222.1" port: 41049 } }
I20260809 12:14:13.962232 18186 catalog_manager.cc:5495] T efb1fe44ea224877bd89ce3b81be955e P b97494c2f09f448e85a9497d47d415f6 reported cstate change: term changed from 0 to 1, leader changed from <none> to b97494c2f09f448e85a9497d47d415f6 (127.0.222.3). New cstate: current_term: 1 leader_uuid: "b97494c2f09f448e85a9497d47d415f6" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b97494c2f09f448e85a9497d47d415f6" member_type: VOTER last_known_addr { host: "127.0.222.3" port: 43523 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "a83ae6a077a34a80985b4b65dfa7b6f6" member_type: VOTER last_known_addr { host: "127.0.222.2" port: 35933 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "bd233f0a216e42e7858d2eb736a785bb" member_type: VOTER last_known_addr { host: "127.0.222.1" port: 41049 } health_report { overall_health: UNKNOWN } } }
I20260809 12:14:14.057308 18678 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 12:14:14.051085 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56258 (local address 127.0.222.3:43523)
0809 12:14:14.051345 (+ 260us) server_negotiation.cc:206] Beginning negotiation
0809 12:14:14.051354 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0809 12:14:14.051378 (+ 24us) server_negotiation.cc:407] Connection header received
0809 12:14:14.051655 (+ 277us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 12:14:14.051659 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 12:14:14.051749 (+ 90us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 12:14:14.051937 (+ 188us) server_negotiation.cc:226] Negotiated authn=TOKEN
0809 12:14:14.052383 (+ 446us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 12:14:14.053233 (+ 850us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 12:14:14.054899 (+ 1666us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 12:14:14.055395 (+ 496us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 12:14:14.055549 (+ 154us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 12:14:14.056123 (+ 574us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0809 12:14:14.056634 (+ 511us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0809 12:14:14.056827 (+ 193us) server_negotiation.cc:1035] Waiting for connection context
0809 12:14:14.056917 (+ 90us) server_negotiation.cc:299] Negotiation successful
0809 12:14:14.057006 (+ 89us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":65}
I20260809 12:14:14.066073 18336 raft_consensus.cc:1261] T efb1fe44ea224877bd89ce3b81be955e P bd233f0a216e42e7858d2eb736a785bb [term 1 FOLLOWER]: Refusing update from remote peer b97494c2f09f448e85a9497d47d415f6: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260809 12:14:14.066097 18478 raft_consensus.cc:1261] T efb1fe44ea224877bd89ce3b81be955e P a83ae6a077a34a80985b4b65dfa7b6f6 [term 1 FOLLOWER]: Refusing update from remote peer b97494c2f09f448e85a9497d47d415f6: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260809 12:14:14.067111 18691 consensus_queue.cc:1035] T efb1fe44ea224877bd89ce3b81be955e P b97494c2f09f448e85a9497d47d415f6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "bd233f0a216e42e7858d2eb736a785bb" member_type: VOTER last_known_addr { host: "127.0.222.1" port: 41049 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260809 12:14:14.067509 18687 consensus_queue.cc:1035] T efb1fe44ea224877bd89ce3b81be955e P b97494c2f09f448e85a9497d47d415f6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a83ae6a077a34a80985b4b65dfa7b6f6" member_type: VOTER last_known_addr { host: "127.0.222.2" port: 35933 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260809 12:14:14.082868 18696 mvcc.cc:205] Tried to move back new op lower bound from 7316593271043960832 to 7316593270614614016. Current Snapshot: MvccSnapshot[applied={T|T < 7316593271043960832}]
I20260809 12:14:14.094640 18185 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:33848:
table { table_name: "test-table" } modify_external_catalogs: true
I20260809 12:14:14.094915 18185 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:33848:
table { table_name: "test-table" } modify_external_catalogs: true
I20260809 12:14:14.098549 18185 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 976efcee2bd94ed6b5c102cc07ac7ab1: Sending DeleteTablet for 3 replicas of tablet efb1fe44ea224877bd89ce3b81be955e
I20260809 12:14:14.099865 18457 tablet_service.cc:1514] Processing DeleteTablet for tablet efb1fe44ea224877bd89ce3b81be955e with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-09 12:14:14 UTC) from {username='kudu', principal='kudu/127.0.222.62@KRBTEST.COM'} at 127.0.0.1:57654
I20260809 12:14:14.099834 18596 tablet_service.cc:1514] Processing DeleteTablet for tablet efb1fe44ea224877bd89ce3b81be955e with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-09 12:14:14 UTC) from {username='kudu', principal='kudu/127.0.222.62@KRBTEST.COM'} at 127.0.0.1:56242
I20260809 12:14:14.100858 18704 tablet_replica.cc:330] T efb1fe44ea224877bd89ce3b81be955e P a83ae6a077a34a80985b4b65dfa7b6f6: stopping tablet replica
I20260809 12:14:14.100780 18314 tablet_service.cc:1514] Processing DeleteTablet for tablet efb1fe44ea224877bd89ce3b81be955e with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-09 12:14:14 UTC) from {username='kudu', principal='kudu/127.0.222.62@KRBTEST.COM'} at 127.0.0.1:35056
I20260809 12:14:14.101303 18704 raft_consensus.cc:2229] T efb1fe44ea224877bd89ce3b81be955e P a83ae6a077a34a80985b4b65dfa7b6f6 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260809 12:14:14.101872 18704 raft_consensus.cc:2258] T efb1fe44ea224877bd89ce3b81be955e P a83ae6a077a34a80985b4b65dfa7b6f6 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260809 12:14:14.102528 888 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskgH5Pjm/build/asan/bin/kudu pid 18224
I20260809 12:14:14.102684 18703 tablet_replica.cc:330] T efb1fe44ea224877bd89ce3b81be955e P b97494c2f09f448e85a9497d47d415f6: stopping tablet replica
I20260809 12:14:14.103211 18703 raft_consensus.cc:2229] T efb1fe44ea224877bd89ce3b81be955e P b97494c2f09f448e85a9497d47d415f6 [term 1 LEADER]: Raft consensus shutting down.
I20260809 12:14:14.103873 18703 raft_consensus.cc:2258] T efb1fe44ea224877bd89ce3b81be955e P b97494c2f09f448e85a9497d47d415f6 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260809 12:14:14.105612 18704 ts_tablet_manager.cc:1905] T efb1fe44ea224877bd89ce3b81be955e P a83ae6a077a34a80985b4b65dfa7b6f6: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260809 12:14:14.111852 18705 tablet_replica.cc:330] T efb1fe44ea224877bd89ce3b81be955e P bd233f0a216e42e7858d2eb736a785bb: stopping tablet replica
I20260809 12:14:14.112469 18705 raft_consensus.cc:2229] T efb1fe44ea224877bd89ce3b81be955e P bd233f0a216e42e7858d2eb736a785bb [term 1 FOLLOWER]: Raft consensus shutting down.
I20260809 12:14:14.113111 18705 raft_consensus.cc:2258] T efb1fe44ea224877bd89ce3b81be955e P bd233f0a216e42e7858d2eb736a785bb [term 1 FOLLOWER]: Raft consensus is shut down!
I20260809 12:14:14.113878 18703 ts_tablet_manager.cc:1905] T efb1fe44ea224877bd89ce3b81be955e P b97494c2f09f448e85a9497d47d415f6: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260809 12:14:14.115638 18705 ts_tablet_manager.cc:1905] T efb1fe44ea224877bd89ce3b81be955e P bd233f0a216e42e7858d2eb736a785bb: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260809 12:14:14.120290 18704 ts_tablet_manager.cc:1918] T efb1fe44ea224877bd89ce3b81be955e P a83ae6a077a34a80985b4b65dfa7b6f6: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260809 12:14:14.120472 18704 log.cc:1196] T efb1fe44ea224877bd89ce3b81be955e P a83ae6a077a34a80985b4b65dfa7b6f6: Deleting WAL directory at /tmp/dist-test-taskgH5Pjm/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786277382073938-888-0/minicluster-data/ts-1/wal/wals/efb1fe44ea224877bd89ce3b81be955e
I20260809 12:14:14.121101 18704 ts_tablet_manager.cc:1939] T efb1fe44ea224877bd89ce3b81be955e P a83ae6a077a34a80985b4b65dfa7b6f6: Deleting consensus metadata
I20260809 12:14:14.122610 18703 ts_tablet_manager.cc:1918] T efb1fe44ea224877bd89ce3b81be955e P b97494c2f09f448e85a9497d47d415f6: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260809 12:14:14.122848 18703 log.cc:1196] T efb1fe44ea224877bd89ce3b81be955e P b97494c2f09f448e85a9497d47d415f6: Deleting WAL directory at /tmp/dist-test-taskgH5Pjm/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786277382073938-888-0/minicluster-data/ts-2/wal/wals/efb1fe44ea224877bd89ce3b81be955e
I20260809 12:14:14.123191 18172 catalog_manager.cc:4841] TS a83ae6a077a34a80985b4b65dfa7b6f6 (127.0.222.2:35933): tablet efb1fe44ea224877bd89ce3b81be955e (table test-table [id=7f6c197c48294e418fbd7b2737188c74]) successfully deleted
I20260809 12:14:14.123664 18703 ts_tablet_manager.cc:1939] T efb1fe44ea224877bd89ce3b81be955e P b97494c2f09f448e85a9497d47d415f6: Deleting consensus metadata
I20260809 12:14:14.123709 18705 ts_tablet_manager.cc:1918] T efb1fe44ea224877bd89ce3b81be955e P bd233f0a216e42e7858d2eb736a785bb: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260809 12:14:14.123970 18705 log.cc:1196] T efb1fe44ea224877bd89ce3b81be955e P bd233f0a216e42e7858d2eb736a785bb: Deleting WAL directory at /tmp/dist-test-taskgH5Pjm/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786277382073938-888-0/minicluster-data/ts-0/wal/wals/efb1fe44ea224877bd89ce3b81be955e
I20260809 12:14:14.124574 18705 ts_tablet_manager.cc:1939] T efb1fe44ea224877bd89ce3b81be955e P bd233f0a216e42e7858d2eb736a785bb: Deleting consensus metadata
I20260809 12:14:14.125829 18170 catalog_manager.cc:4841] TS b97494c2f09f448e85a9497d47d415f6 (127.0.222.3:43523): tablet efb1fe44ea224877bd89ce3b81be955e (table test-table [id=7f6c197c48294e418fbd7b2737188c74]) successfully deleted
I20260809 12:14:14.126484 18171 catalog_manager.cc:4841] TS bd233f0a216e42e7858d2eb736a785bb (127.0.222.1:41049): tablet efb1fe44ea224877bd89ce3b81be955e (table test-table [id=7f6c197c48294e418fbd7b2737188c74]) successfully deleted
I20260809 12:14:14.136318 18679 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 12:14:14.104221 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35068 (local address 127.0.222.1:41049)
0809 12:14:14.104492 (+ 271us) server_negotiation.cc:206] Beginning negotiation
0809 12:14:14.104503 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0809 12:14:14.111586 (+ 7083us) server_negotiation.cc:407] Connection header received
0809 12:14:14.111900 (+ 314us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 12:14:14.111908 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 12:14:14.112028 (+ 120us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 12:14:14.112110 (+ 82us) server_negotiation.cc:226] Negotiated authn=SASL
0809 12:14:14.116734 (+ 4624us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 12:14:14.117979 (+ 1245us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 12:14:14.124972 (+ 6993us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 12:14:14.125671 (+ 699us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 12:14:14.125775 (+ 104us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 12:14:14.129765 (+ 3990us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 12:14:14.129783 (+ 18us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 12:14:14.129803 (+ 20us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0809 12:14:14.129850 (+ 47us) server_negotiation.cc:922] calling sasl_server_start()
0809 12:14:14.133127 (+ 3277us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 12:14:14.134000 (+ 873us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 12:14:14.134007 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 12:14:14.134009 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0809 12:14:14.134142 (+ 133us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 12:14:14.134580 (+ 438us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 12:14:14.134584 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 12:14:14.134585 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0809 12:14:14.134948 (+ 363us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 12:14:14.135246 (+ 298us) server_negotiation.cc:1035] Waiting for connection context
0809 12:14:14.135653 (+ 407us) server_negotiation.cc:299] Negotiation successful
0809 12:14:14.135958 (+ 305us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":97}
I20260809 12:14:14.137061 18379 generic_service.cc:196] Checking for leaks (request via RPC)
I20260809 12:14:14.228555 888 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskgH5Pjm/build/asan/bin/kudu with pid 18224
I20260809 12:14:14.250718 888 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskgH5Pjm/build/asan/bin/kudu pid 18389
I20260809 12:14:14.266937 18676 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 12:14:14.251812 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57668 (local address 127.0.222.2:35933)
0809 12:14:14.252018 (+ 206us) server_negotiation.cc:206] Beginning negotiation
0809 12:14:14.252025 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0809 12:14:14.252158 (+ 133us) server_negotiation.cc:407] Connection header received
0809 12:14:14.252336 (+ 178us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 12:14:14.252343 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 12:14:14.252433 (+ 90us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 12:14:14.252568 (+ 135us) server_negotiation.cc:226] Negotiated authn=SASL
0809 12:14:14.253950 (+ 1382us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 12:14:14.255116 (+ 1166us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 12:14:14.256378 (+ 1262us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 12:14:14.257012 (+ 634us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 12:14:14.257142 (+ 130us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 12:14:14.260074 (+ 2932us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 12:14:14.260092 (+ 18us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 12:14:14.260112 (+ 20us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0809 12:14:14.260155 (+ 43us) server_negotiation.cc:922] calling sasl_server_start()
0809 12:14:14.263882 (+ 3727us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 12:14:14.264540 (+ 658us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 12:14:14.264550 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 12:14:14.264554 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0809 12:14:14.264725 (+ 171us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 12:14:14.265131 (+ 406us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 12:14:14.265136 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 12:14:14.265140 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0809 12:14:14.265661 (+ 521us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 12:14:14.265965 (+ 304us) server_negotiation.cc:1035] Waiting for connection context
0809 12:14:14.266299 (+ 334us) server_negotiation.cc:299] Negotiation successful
0809 12:14:14.266541 (+ 242us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":53}
I20260809 12:14:14.267735 18520 generic_service.cc:196] Checking for leaks (request via RPC)
I20260809 12:14:14.357470 888 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskgH5Pjm/build/asan/bin/kudu with pid 18389
I20260809 12:14:14.378612 888 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskgH5Pjm/build/asan/bin/kudu pid 18529
I20260809 12:14:14.395184 18678 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 12:14:14.379493 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56272 (local address 127.0.222.3:43523)
0809 12:14:14.379717 (+ 224us) server_negotiation.cc:206] Beginning negotiation
0809 12:14:14.379726 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0809 12:14:14.379751 (+ 25us) server_negotiation.cc:407] Connection header received
0809 12:14:14.379823 (+ 72us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 12:14:14.379829 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 12:14:14.379950 (+ 121us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 12:14:14.380079 (+ 129us) server_negotiation.cc:226] Negotiated authn=SASL
0809 12:14:14.381504 (+ 1425us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 12:14:14.382777 (+ 1273us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 12:14:14.384146 (+ 1369us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 12:14:14.384836 (+ 690us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 12:14:14.384973 (+ 137us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 12:14:14.387687 (+ 2714us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 12:14:14.387704 (+ 17us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 12:14:14.387725 (+ 21us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0809 12:14:14.387773 (+ 48us) server_negotiation.cc:922] calling sasl_server_start()
0809 12:14:14.392035 (+ 4262us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 12:14:14.392724 (+ 689us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 12:14:14.392736 (+ 12us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 12:14:14.392740 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0809 12:14:14.392960 (+ 220us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 12:14:14.393354 (+ 394us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 12:14:14.393360 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 12:14:14.393363 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0809 12:14:14.393853 (+ 490us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 12:14:14.394275 (+ 422us) server_negotiation.cc:1035] Waiting for connection context
0809 12:14:14.394445 (+ 170us) server_negotiation.cc:299] Negotiation successful
0809 12:14:14.394688 (+ 243us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":91}
I20260809 12:14:14.396023 18658 generic_service.cc:196] Checking for leaks (request via RPC)
I20260809 12:14:14.522203 888 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskgH5Pjm/build/asan/bin/kudu with pid 18529
I20260809 12:14:14.545785 888 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskgH5Pjm/build/asan/bin/kudu pid 18151
I20260809 12:14:14.547042 18217 generic_service.cc:196] Checking for leaks (request via RPC)
I20260809 12:14:14.662941 888 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskgH5Pjm/build/asan/bin/kudu with pid 18151
I20260809 12:14:34.690858 888 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskgH5Pjm/build/asan/bin/kudu
/tmp/dist-test-taskgH5Pjm/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskgH5Pjm/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786277382073938-888-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskgH5Pjm/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786277382073938-888-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskgH5Pjm/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786277382073938-888-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskgH5Pjm/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786277382073938-888-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.222.62:45505
--webserver_interface=127.0.222.62
--webserver_port=35835
--keytab_file=/tmp/dist-test-taskgH5Pjm/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786277382073938-888-0/krb5kdc/kudu_127.0.222.62.keytab
--principal=kudu/127.0.222.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.222.20:41481
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.222.62:45505
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--authn_token_validity_seconds=20
--authz_token_validity_seconds=20
--tsk_rotation_seconds=20 with env {KRB5CCNAME=/tmp/dist-test-taskgH5Pjm/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786277382073938-888-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskgH5Pjm/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786277382073938-888-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskgH5Pjm/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786277382073938-888-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260809 12:14:35.036947 18712 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260809 12:14:35.037277 18712 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260809 12:14:35.037437 18712 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260809 12:14:35.044785 18712 flags.cc:425] Enabled experimental flag: --authn_token_validity_seconds=20
W20260809 12:14:35.044909 18712 flags.cc:425] Enabled experimental flag: --authz_token_validity_seconds=20
W20260809 12:14:35.044981 18712 flags.cc:425] Enabled experimental flag: --tsk_rotation_seconds=20
W20260809 12:14:35.045063 18712 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260809 12:14:35.045145 18712 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260809 12:14:35.045250 18712 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260809 12:14:35.045321 18712 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260809 12:14:35.045405 18712 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260809 12:14:35.058324 18712 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.222.20:41481
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskgH5Pjm/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786277382073938-888-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskgH5Pjm/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786277382073938-888-0/minicluster-data/master-0/wal
--authn_token_validity_seconds=20
--authz_token_validity_seconds=20
--tsk_rotation_seconds=20
--ipki_ca_key_size=768
--master_addresses=127.0.222.62:45505
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskgH5Pjm/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786277382073938-888-0/krb5kdc/kudu_127.0.222.62.keytab
--principal=kudu/127.0.222.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.222.62:45505
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskgH5Pjm/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786277382073938-888-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.222.62
--webserver_port=35835
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskgH5Pjm/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786277382073938-888-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 09 Aug 2026 11:06:41 UTC on ccycloud.quasar-bwoogy.root.comops.site
build id 4575991
ASAN enabled
I20260809 12:14:35.059676 18712 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260809 12:14:35.060981 18712 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260809 12:14:35.073215 18718 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 12:14:35.073275 18717 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 12:14:35.073238 18720 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260809 12:14:35.073655 18712 server_base.cc:1034] running on GCE node
I20260809 12:14:35.074805 18712 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260809 12:14:35.076277 18712 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260809 12:14:35.077557 18712 hybrid_clock.cc:648] HybridClock initialized: now 1786277675077471 us; error 53 us; skew 500 ppm
I20260809 12:14:35.077963 18712 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 09 12:14:35 dist-test-slave-2w3w krb5kdc[17567](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786277675, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.222.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260809 12:14:35.085528 18712 init.cc:377] Logged in from keytab as kudu/127.0.222.62@KRBTEST.COM (short username kudu)
I20260809 12:14:35.087215 18712 webserver.cc:460] Webserver started at http://127.0.222.62:35835/ using document root <none> and password file <none>
I20260809 12:14:35.087967 18712 fs_manager.cc:359] Metadata directory not provided
I20260809 12:14:35.088102 18712 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260809 12:14:35.093830 18712 fs_manager.cc:709] Time spent opening directory manager: real 0.004s user 0.002s sys 0.003s
I20260809 12:14:35.096702 18727 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260809 12:14:35.097987 18712 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.000s
I20260809 12:14:35.098171 18712 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskgH5Pjm/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786277382073938-888-0/minicluster-data/master-0/data,/tmp/dist-test-taskgH5Pjm/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786277382073938-888-0/minicluster-data/master-0/wal
uuid: "976efcee2bd94ed6b5c102cc07ac7ab1"
format_stamp: "Formatted at 2026-08-09 12:13:48 on dist-test-slave-2w3w"
server_key: "5f75dd8c4f7b73a87d0db18621478cff"
server_key_iv: "fdfbac5a7eab9ac2b3d7ff0e02c44b99"
server_key_version: "encryptionkey@0"
I20260809 12:14:35.099195 18712 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskgH5Pjm/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786277382073938-888-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskgH5Pjm/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786277382073938-888-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskgH5Pjm/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786277382073938-888-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260809 12:14:35.122517 18712 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260809 12:14:35.126825 18712 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260809 12:14:35.127254 18712 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260809 12:14:35.146538 18712 rpc_server.cc:307] RPC server started. Bound to: 127.0.222.62:45505
I20260809 12:14:35.146584 18781 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.222.62:45505 every 8 connection(s)
I20260809 12:14:35.148134 18712 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskgH5Pjm/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786277382073938-888-0/minicluster-data/master-0/data/info.pb
I20260809 12:14:35.152886 888 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskgH5Pjm/build/asan/bin/kudu as pid 18712
I20260809 12:14:35.153970 888 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskgH5Pjm/build/asan/bin/kudu
/tmp/dist-test-taskgH5Pjm/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskgH5Pjm/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786277382073938-888-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskgH5Pjm/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786277382073938-888-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskgH5Pjm/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786277382073938-888-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskgH5Pjm/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786277382073938-888-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.222.1:41049
--local_ip_for_outbound_sockets=127.0.222.1
--tserver_master_addrs=127.0.222.62:45505
--webserver_port=38689
--webserver_interface=127.0.222.1
--keytab_file=/tmp/dist-test-taskgH5Pjm/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786277382073938-888-0/krb5kdc/kudu_127.0.222.1.keytab
--principal=kudu/127.0.222.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.222.20:41481
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskgH5Pjm/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786277382073938-888-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskgH5Pjm/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786277382073938-888-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskgH5Pjm/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786277382073938-888-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260809 12:14:35.156255 18782 sys_catalog.cc:263] Verifying existing consensus state
I20260809 12:14:35.161375 18782 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 976efcee2bd94ed6b5c102cc07ac7ab1: Bootstrap starting.
I20260809 12:14:35.184134 18782 log.cc:824] T 00000000000000000000000000000000 P 976efcee2bd94ed6b5c102cc07ac7ab1: Log is configured to *not* fsync() on all Append() calls
I20260809 12:14:35.208673 18782 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 976efcee2bd94ed6b5c102cc07ac7ab1: Bootstrap replayed 1/1 log segments. Stats: ops{read=14 overwritten=0 applied=14 ignored=0} inserts{seen=8 ignored=0} mutations{seen=8 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260809 12:14:35.209298 18782 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 976efcee2bd94ed6b5c102cc07ac7ab1: Bootstrap complete.
I20260809 12:14:35.219079 18782 raft_consensus.cc:348] T 00000000000000000000000000000000 P 976efcee2bd94ed6b5c102cc07ac7ab1 [term 2 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "976efcee2bd94ed6b5c102cc07ac7ab1" member_type: VOTER last_known_addr { host: "127.0.222.62" port: 45505 } }
I20260809 12:14:35.220077 18782 raft_consensus.cc:729] T 00000000000000000000000000000000 P 976efcee2bd94ed6b5c102cc07ac7ab1 [term 2 FOLLOWER]: Becoming Follower/Learner. State: Replica: 976efcee2bd94ed6b5c102cc07ac7ab1, State: Initialized, Role: FOLLOWER
I20260809 12:14:35.220712 18782 consensus_queue.cc:260] T 00000000000000000000000000000000 P 976efcee2bd94ed6b5c102cc07ac7ab1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 14, Last appended: 2.14, Last appended by leader: 14, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "976efcee2bd94ed6b5c102cc07ac7ab1" member_type: VOTER last_known_addr { host: "127.0.222.62" port: 45505 } }
I20260809 12:14:35.220974 18782 raft_consensus.cc:388] T 00000000000000000000000000000000 P 976efcee2bd94ed6b5c102cc07ac7ab1 [term 2 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260809 12:14:35.221107 18782 raft_consensus.cc:482] T 00000000000000000000000000000000 P 976efcee2bd94ed6b5c102cc07ac7ab1 [term 2 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260809 12:14:35.221289 18782 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 976efcee2bd94ed6b5c102cc07ac7ab1 [term 2 FOLLOWER]: Advancing to term 3
I20260809 12:14:35.225284 18782 raft_consensus.cc:504] T 00000000000000000000000000000000 P 976efcee2bd94ed6b5c102cc07ac7ab1 [term 3 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "976efcee2bd94ed6b5c102cc07ac7ab1" member_type: VOTER last_known_addr { host: "127.0.222.62" port: 45505 } }
I20260809 12:14:35.225795 18782 leader_election.cc:302] T 00000000000000000000000000000000 P 976efcee2bd94ed6b5c102cc07ac7ab1 [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 976efcee2bd94ed6b5c102cc07ac7ab1; no voters:
I20260809 12:14:35.226527 18782 leader_election.cc:288] T 00000000000000000000000000000000 P 976efcee2bd94ed6b5c102cc07ac7ab1 [CANDIDATE]: Term 3 election: Requested vote from peers
I20260809 12:14:35.226672 18786 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 976efcee2bd94ed6b5c102cc07ac7ab1 [term 3 FOLLOWER]: Leader election won for term 3
I20260809 12:14:35.228013 18786 raft_consensus.cc:686] T 00000000000000000000000000000000 P 976efcee2bd94ed6b5c102cc07ac7ab1 [term 3 LEADER]: Becoming Leader. State: Replica: 976efcee2bd94ed6b5c102cc07ac7ab1, State: Running, Role: LEADER
I20260809 12:14:35.228662 18786 consensus_queue.cc:237] T 00000000000000000000000000000000 P 976efcee2bd94ed6b5c102cc07ac7ab1 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 14, Committed index: 14, Last appended: 2.14, Last appended by leader: 14, Current term: 3, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "976efcee2bd94ed6b5c102cc07ac7ab1" member_type: VOTER last_known_addr { host: "127.0.222.62" port: 45505 } }
I20260809 12:14:35.229013 18782 sys_catalog.cc:564] T 00000000000000000000000000000000 P 976efcee2bd94ed6b5c102cc07ac7ab1 [sys.catalog]: configured and running, proceeding with master startup.
I20260809 12:14:35.231395 18788 sys_catalog.cc:455] T 00000000000000000000000000000000 P 976efcee2bd94ed6b5c102cc07ac7ab1 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 976efcee2bd94ed6b5c102cc07ac7ab1. Latest consensus state: current_term: 3 leader_uuid: "976efcee2bd94ed6b5c102cc07ac7ab1" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "976efcee2bd94ed6b5c102cc07ac7ab1" member_type: VOTER last_known_addr { host: "127.0.222.62" port: 45505 } } }
I20260809 12:14:35.231806 18788 sys_catalog.cc:458] T 00000000000000000000000000000000 P 976efcee2bd94ed6b5c102cc07ac7ab1 [sys.catalog]: This master's current role is: LEADER
I20260809 12:14:35.231564 18787 sys_catalog.cc:455] T 00000000000000000000000000000000 P 976efcee2bd94ed6b5c102cc07ac7ab1 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 3 leader_uuid: "976efcee2bd94ed6b5c102cc07ac7ab1" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "976efcee2bd94ed6b5c102cc07ac7ab1" member_type: VOTER last_known_addr { host: "127.0.222.62" port: 45505 } } }
I20260809 12:14:35.232021 18787 sys_catalog.cc:458] T 00000000000000000000000000000000 P 976efcee2bd94ed6b5c102cc07ac7ab1 [sys.catalog]: This master's current role is: LEADER
I20260809 12:14:35.243960 18793 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260809 12:14:35.253211 18793 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260809 12:14:35.255136 18793 catalog_manager.cc:1259] Loaded cluster ID: d8085938efb84ddc8c174e7e9bb18f14
I20260809 12:14:35.255347 18793 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260809 12:14:35.259637 18793 catalog_manager.cc:1504] Loading token signing keys...
I20260809 12:14:35.261242 18793 catalog_manager.cc:5879] T 00000000000000000000000000000000 P 976efcee2bd94ed6b5c102cc07ac7ab1: Loaded TSK: 0
I20260809 12:14:35.261510 18793 catalog_manager.cc:5879] T 00000000000000000000000000000000 P 976efcee2bd94ed6b5c102cc07ac7ab1: Loaded TSK: 1
F20260809 12:14:35.262723 18793 token_signing_key.cc:66] Check failed: _s.ok() Bad status: Runtime error: unable to load data from memory: error:0D0680A8:asn1 encoding routines:ASN1_CHECK_TLEN:wrong tag:tasn_dec.c:1220 error:0D06C03A:asn1 encoding routines:ASN1_D2I_EX_PRIMITIVE:nested asn1 error:tasn_dec.c:788 error:0D08303A:asn1 encoding routines:ASN1_TEMPLATE_NOEXP_D2I:nested asn1 error:tasn_dec.c:720:Field=version, Type=RSA error:04093004:rsa routines:OLD_RSA_PRIV_DECODE:RSA lib:rsa_ameth.c:121 error:0D0680A8:asn1 encoding routines:ASN1_CHECK_TLEN:wrong tag:tasn_dec.c:1220 error:0D06C03A:asn1 encoding routines:ASN1_D2I_EX_PRIMITIVE:nested asn1 error:tasn_dec.c:788 error:0D08303A:asn1 encoding routines:ASN1_TEMPLATE_NOEXP_D2I:nested asn1 error:tasn_dec.c:720:Field=version, Type=PKCS8_PRIV_KEY_INFO
*** Check failure stack trace: ***
@ 0x7ff44c8279ad google::LogMessage::Fail() at ??:0
@ 0x7ff44c82a88a google::LogMessage::SendToLog() at ??:0
@ 0x7ff44c8274de google::LogMessage::Flush() at ??:0
@ 0x7ff44c828a19 google::LogMessageFatal::~LogMessageFatal() at ??:0
@ 0x7ff45058b7d2 kudu::security::TokenSigningPrivateKey::TokenSigningPrivateKey() at ??:0
@ 0x7ff45057a73e kudu::security::TokenSigner::ImportKeys() at ??:0
@ 0x7ff4622561c0 kudu::master::CatalogManager::LoadTskEntries() at ??:0
@ 0x7ff462255783 kudu::master::CatalogManager::InitTokenSigner() at ??:0
@ 0x7ff4622e12a7 kudu::master::CatalogManager::PrepareForLeadershipTask()::$_10::operator()() at ??:0
@ 0x7ff4622e10c4 std::_Function_handler<>::_M_invoke() at ??:0
I20260809 12:14:35.310141 18808 catalog_manager.cc:795] Waiting for catalog manager background task thread to start: Service unavailable: Catalog manager is not initialized. State: Starting
@ 0x7ff46232b5e9 std::function<>::operator()() at ??:0
@ 0x7ff462259cdd kudu::master::CatalogManager::PrepareForLeadershipTask()::$_6::operator()() at ??:0
@ 0x7ff462257c36 kudu::master::CatalogManager::PrepareForLeadershipTask() at ??:0
@ 0x7ff4622df610 kudu::master::CatalogManager::ElectedAsLeaderCb()::$_2::operator()() at ??:0
@ 0x7ff4622df439 std::_Function_handler<>::_M_invoke() at ??:0
@ 0x9edb7d std::function<>::operator()() at ??:?
@ 0x7ff44f473345 kudu::ThreadPool::DispatchThread() at ??:0
@ 0x7ff44f47691c kudu::ThreadPool::CreateThread()::$_2::operator()() at ??:0
@ 0x7ff44f476799 std::_Function_handler<>::_M_invoke() at ??:0
@ 0x9edb7d std::function<>::operator()() at ??:?
@ 0x7ff44f451eaf kudu::Thread::SuperviseThread() at ??:0
@ 0x7ff45272b6db start_thread at ??:0
@ 0x7ff44b6c161f clone at ??:0
W20260809 12:14:35.574280 18784 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260809 12:14:35.574572 18784 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260809 12:14:35.574729 18784 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260809 12:14:35.582480 18784 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260809 12:14:35.582610 18784 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260809 12:14:35.582806 18784 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.222.1
I20260809 12:14:35.593098 18784 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.222.20:41481
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskgH5Pjm/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786277382073938-888-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskgH5Pjm/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786277382073938-888-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskgH5Pjm/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786277382073938-888-0/krb5kdc/kudu_127.0.222.1.keytab
--principal=kudu/127.0.222.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.222.1:41049
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskgH5Pjm/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786277382073938-888-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.222.1
--webserver_port=38689
--tserver_master_addrs=127.0.222.62:45505
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.222.1
--log_dir=/tmp/dist-test-taskgH5Pjm/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786277382073938-888-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 09 Aug 2026 11:06:41 UTC on ccycloud.quasar-bwoogy.root.comops.site
build id 4575991
ASAN enabled
I20260809 12:14:35.594470 18784 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260809 12:14:35.595847 18784 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260809 12:14:35.608932 18835 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 12:14:35.609043 18831 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 12:14:35.609071 18833 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260809 12:14:35.610055 18784 server_base.cc:1034] running on GCE node
I20260809 12:14:35.610729 18784 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260809 12:14:35.611699 18784 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260809 12:14:35.612931 18784 hybrid_clock.cc:648] HybridClock initialized: now 1786277675612883 us; error 55 us; skew 500 ppm
I20260809 12:14:35.613245 18784 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 09 12:14:35 dist-test-slave-2w3w krb5kdc[17567](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786277675, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.222.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260809 12:14:35.616878 18784 init.cc:377] Logged in from keytab as kudu/127.0.222.1@KRBTEST.COM (short username kudu)
I20260809 12:14:35.618373 18784 webserver.cc:460] Webserver started at http://127.0.222.1:38689/ using document root <none> and password file <none>
I20260809 12:14:35.619155 18784 fs_manager.cc:359] Metadata directory not provided
I20260809 12:14:35.619313 18784 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260809 12:14:35.624877 18784 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.005s sys 0.000s
I20260809 12:14:35.627858 18843 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260809 12:14:35.629238 18784 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.000s sys 0.003s
I20260809 12:14:35.629387 18784 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskgH5Pjm/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786277382073938-888-0/minicluster-data/ts-0/data,/tmp/dist-test-taskgH5Pjm/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786277382073938-888-0/minicluster-data/ts-0/wal
uuid: "bd233f0a216e42e7858d2eb736a785bb"
format_stamp: "Formatted at 2026-08-09 12:13:49 on dist-test-slave-2w3w"
server_key: "9f4432a335e7b85d34e07fd2f3696829"
server_key_iv: "1d7e63d4b85cc01a747aca61fa912fc3"
server_key_version: "encryptionkey@0"
I20260809 12:14:35.630367 18784 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskgH5Pjm/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786277382073938-888-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskgH5Pjm/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786277382073938-888-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskgH5Pjm/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786277382073938-888-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260809 12:14:35.652696 18784 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260809 12:14:35.657123 18784 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260809 12:14:35.657367 18784 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260809 12:14:35.658314 18784 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260809 12:14:35.659696 18784 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260809 12:14:35.659785 18784 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260809 12:14:35.659868 18784 ts_tablet_manager.cc:610] Registered 0 tablets
I20260809 12:14:35.659921 18784 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260809 12:14:35.695044 18784 rpc_server.cc:307] RPC server started. Bound to: 127.0.222.1:41049
I20260809 12:14:35.695091 18962 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.222.1:41049 every 8 connection(s)
I20260809 12:14:35.696512 18784 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskgH5Pjm/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786277382073938-888-0/minicluster-data/ts-0/data/info.pb
I20260809 12:14:35.696949 888 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskgH5Pjm/build/asan/bin/kudu as pid 18784
I20260809 12:14:35.697757 888 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskgH5Pjm/build/asan/bin/kudu
/tmp/dist-test-taskgH5Pjm/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskgH5Pjm/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786277382073938-888-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskgH5Pjm/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786277382073938-888-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskgH5Pjm/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786277382073938-888-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskgH5Pjm/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786277382073938-888-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.222.2:35933
--local_ip_for_outbound_sockets=127.0.222.2
--tserver_master_addrs=127.0.222.62:45505
--webserver_port=37553
--webserver_interface=127.0.222.2
--keytab_file=/tmp/dist-test-taskgH5Pjm/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786277382073938-888-0/krb5kdc/kudu_127.0.222.2.keytab
--principal=kudu/127.0.222.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.222.20:41481
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskgH5Pjm/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786277382073938-888-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskgH5Pjm/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786277382073938-888-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskgH5Pjm/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786277382073938-888-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260809 12:14:35.702041 18966 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 12:14:35.699684 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.222.62:45505 (local address 127.0.222.1:46831)
0809 12:14:35.700568 (+ 884us) negotiation.cc:107] Waiting for socket to connect
0809 12:14:35.700851 (+ 283us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.222.62:45505: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":552,"thread_start_us":331,"threads_started":1}
W20260809 12:14:35.702715 18964 heartbeater.cc:644] Failed to heartbeat to 127.0.222.62:45505 (0 consecutive failures): Network error: Failed to ping master at 127.0.222.62:45505: Client connection negotiation failed: client connection to 127.0.222.62:45505: connect: Connection refused (error 111)
I20260809 12:14:35.703696 18966 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 12:14:35.703268 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.222.62:45505 (local address 127.0.222.1:50789)
0809 12:14:35.703453 (+ 185us) negotiation.cc:107] Waiting for socket to connect
0809 12:14:35.703549 (+ 96us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.222.62:45505: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":50}
I20260809 12:14:35.704695 18966 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 12:14:35.704278 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.222.62:45505 (local address 127.0.222.1:33667)
0809 12:14:35.704436 (+ 158us) negotiation.cc:107] Waiting for socket to connect
0809 12:14:35.704536 (+ 100us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.222.62:45505: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":48}
W20260809 12:14:35.704963 18964 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
W20260809 12:14:36.039844 18968 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260809 12:14:36.040160 18968 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260809 12:14:36.040315 18968 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260809 12:14:36.047518 18968 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260809 12:14:36.047658 18968 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260809 12:14:36.047832 18968 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.222.2
I20260809 12:14:36.057189 18968 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.222.20:41481
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskgH5Pjm/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786277382073938-888-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskgH5Pjm/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786277382073938-888-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskgH5Pjm/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786277382073938-888-0/krb5kdc/kudu_127.0.222.2.keytab
--principal=kudu/127.0.222.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.222.2:35933
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskgH5Pjm/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786277382073938-888-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.222.2
--webserver_port=37553
--tserver_master_addrs=127.0.222.62:45505
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.222.2
--log_dir=/tmp/dist-test-taskgH5Pjm/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786277382073938-888-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 09 Aug 2026 11:06:41 UTC on ccycloud.quasar-bwoogy.root.comops.site
build id 4575991
ASAN enabled
I20260809 12:14:36.058447 18968 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260809 12:14:36.059721 18968 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260809 12:14:36.072896 18975 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 12:14:36.072867 18976 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 12:14:36.072917 18978 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260809 12:14:36.073753 18968 server_base.cc:1034] running on GCE node
I20260809 12:14:36.074306 18968 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260809 12:14:36.075279 18968 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260809 12:14:36.076507 18968 hybrid_clock.cc:648] HybridClock initialized: now 1786277676076460 us; error 55 us; skew 500 ppm
I20260809 12:14:36.076861 18968 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 09 12:14:36 dist-test-slave-2w3w krb5kdc[17567](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786277676, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.222.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260809 12:14:36.080019 18968 init.cc:377] Logged in from keytab as kudu/127.0.222.2@KRBTEST.COM (short username kudu)
I20260809 12:14:36.081437 18968 webserver.cc:460] Webserver started at http://127.0.222.2:37553/ using document root <none> and password file <none>
I20260809 12:14:36.082126 18968 fs_manager.cc:359] Metadata directory not provided
I20260809 12:14:36.082273 18968 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260809 12:14:36.087783 18968 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.003s sys 0.000s
I20260809 12:14:36.090456 18986 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260809 12:14:36.091840 18968 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.001s
I20260809 12:14:36.092018 18968 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskgH5Pjm/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786277382073938-888-0/minicluster-data/ts-1/data,/tmp/dist-test-taskgH5Pjm/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786277382073938-888-0/minicluster-data/ts-1/wal
uuid: "a83ae6a077a34a80985b4b65dfa7b6f6"
format_stamp: "Formatted at 2026-08-09 12:13:50 on dist-test-slave-2w3w"
server_key: "df2da0431ccce241d64ece7296ce9d25"
server_key_iv: "90e49b1f6082ddd27cd5c5b866344a61"
server_key_version: "encryptionkey@0"
I20260809 12:14:36.092998 18968 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskgH5Pjm/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786277382073938-888-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskgH5Pjm/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786277382073938-888-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskgH5Pjm/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786277382073938-888-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260809 12:14:36.120949 18968 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260809 12:14:36.125522 18968 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260809 12:14:36.125840 18968 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260809 12:14:36.126883 18968 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260809 12:14:36.128423 18968 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260809 12:14:36.128553 18968 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260809 12:14:36.128684 18968 ts_tablet_manager.cc:610] Registered 0 tablets
I20260809 12:14:36.128772 18968 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260809 12:14:36.163213 18968 rpc_server.cc:307] RPC server started. Bound to: 127.0.222.2:35933
I20260809 12:14:36.163303 19108 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.222.2:35933 every 8 connection(s)
I20260809 12:14:36.164762 18968 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskgH5Pjm/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786277382073938-888-0/minicluster-data/ts-1/data/info.pb
I20260809 12:14:36.168934 888 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskgH5Pjm/build/asan/bin/kudu as pid 18968
I20260809 12:14:36.169734 888 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskgH5Pjm/build/asan/bin/kudu
/tmp/dist-test-taskgH5Pjm/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskgH5Pjm/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786277382073938-888-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskgH5Pjm/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786277382073938-888-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskgH5Pjm/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786277382073938-888-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskgH5Pjm/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786277382073938-888-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.222.3:43523
--local_ip_for_outbound_sockets=127.0.222.3
--tserver_master_addrs=127.0.222.62:45505
--webserver_port=39089
--webserver_interface=127.0.222.3
--keytab_file=/tmp/dist-test-taskgH5Pjm/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786277382073938-888-0/krb5kdc/kudu_127.0.222.3.keytab
--principal=kudu/127.0.222.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.222.20:41481
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskgH5Pjm/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786277382073938-888-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskgH5Pjm/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786277382073938-888-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskgH5Pjm/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786277382073938-888-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260809 12:14:36.170018 19111 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 12:14:36.167728 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.222.62:45505 (local address 127.0.222.2:56401)
0809 12:14:36.168535 (+ 807us) negotiation.cc:107] Waiting for socket to connect
0809 12:14:36.168837 (+ 302us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.222.62:45505: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":478,"thread_start_us":286,"threads_started":1}
W20260809 12:14:36.170753 19109 heartbeater.cc:644] Failed to heartbeat to 127.0.222.62:45505 (0 consecutive failures): Network error: Failed to ping master at 127.0.222.62:45505: Client connection negotiation failed: client connection to 127.0.222.62:45505: connect: Connection refused (error 111)
I20260809 12:14:36.171741 19111 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 12:14:36.171299 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.222.62:45505 (local address 127.0.222.2:60111)
0809 12:14:36.171491 (+ 192us) negotiation.cc:107] Waiting for socket to connect
0809 12:14:36.171591 (+ 100us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.222.62:45505: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":51}
I20260809 12:14:36.172732 19111 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 12:14:36.172298 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.222.62:45505 (local address 127.0.222.2:38421)
0809 12:14:36.172466 (+ 168us) negotiation.cc:107] Waiting for socket to connect
0809 12:14:36.172571 (+ 105us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.222.62:45505: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":48}
W20260809 12:14:36.172994 19109 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
W20260809 12:14:36.510282 19112 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260809 12:14:36.510576 19112 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260809 12:14:36.510749 19112 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260809 12:14:36.518251 19112 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260809 12:14:36.518366 19112 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260809 12:14:36.518572 19112 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.222.3
I20260809 12:14:36.528676 19112 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.222.20:41481
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskgH5Pjm/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786277382073938-888-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskgH5Pjm/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786277382073938-888-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskgH5Pjm/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786277382073938-888-0/krb5kdc/kudu_127.0.222.3.keytab
--principal=kudu/127.0.222.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.222.3:43523
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskgH5Pjm/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786277382073938-888-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.222.3
--webserver_port=39089
--tserver_master_addrs=127.0.222.62:45505
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.222.3
--log_dir=/tmp/dist-test-taskgH5Pjm/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786277382073938-888-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 09 Aug 2026 11:06:41 UTC on ccycloud.quasar-bwoogy.root.comops.site
build id 4575991
ASAN enabled
I20260809 12:14:36.530018 19112 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260809 12:14:36.531379 19112 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260809 12:14:36.544570 19118 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 12:14:36.544528 19119 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 12:14:36.544617 19121 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260809 12:14:36.545596 19112 server_base.cc:1034] running on GCE node
I20260809 12:14:36.546233 19112 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260809 12:14:36.547283 19112 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260809 12:14:36.548542 19112 hybrid_clock.cc:648] HybridClock initialized: now 1786277676548488 us; error 46 us; skew 500 ppm
I20260809 12:14:36.548940 19112 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 09 12:14:36 dist-test-slave-2w3w krb5kdc[17567](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786277676, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.222.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260809 12:14:36.552219 19112 init.cc:377] Logged in from keytab as kudu/127.0.222.3@KRBTEST.COM (short username kudu)
I20260809 12:14:36.553632 19112 webserver.cc:460] Webserver started at http://127.0.222.3:39089/ using document root <none> and password file <none>
I20260809 12:14:36.554291 19112 fs_manager.cc:359] Metadata directory not provided
I20260809 12:14:36.554471 19112 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260809 12:14:36.559891 19112 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.004s sys 0.000s
I20260809 12:14:36.562495 19128 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260809 12:14:36.563823 19112 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260809 12:14:36.563995 19112 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskgH5Pjm/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786277382073938-888-0/minicluster-data/ts-2/data,/tmp/dist-test-taskgH5Pjm/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786277382073938-888-0/minicluster-data/ts-2/wal
uuid: "b97494c2f09f448e85a9497d47d415f6"
format_stamp: "Formatted at 2026-08-09 12:13:50 on dist-test-slave-2w3w"
server_key: "2fb234b39084d1c2b523aaaa022a2ee0"
server_key_iv: "1ee45beb051160c66da10ad87b022316"
server_key_version: "encryptionkey@0"
I20260809 12:14:36.564970 19112 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskgH5Pjm/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786277382073938-888-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskgH5Pjm/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786277382073938-888-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskgH5Pjm/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786277382073938-888-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260809 12:14:36.598515 19112 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260809 12:14:36.602836 19112 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260809 12:14:36.603147 19112 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260809 12:14:36.604202 19112 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260809 12:14:36.605626 19112 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260809 12:14:36.605765 19112 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260809 12:14:36.605890 19112 ts_tablet_manager.cc:610] Registered 0 tablets
I20260809 12:14:36.605978 19112 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260809 12:14:36.640743 19112 rpc_server.cc:307] RPC server started. Bound to: 127.0.222.3:43523
I20260809 12:14:36.640825 19250 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.222.3:43523 every 8 connection(s)
I20260809 12:14:36.642198 19112 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskgH5Pjm/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786277382073938-888-0/minicluster-data/ts-2/data/info.pb
I20260809 12:14:36.647053 19253 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 12:14:36.645126 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.222.62:45505 (local address 127.0.222.3:35497)
0809 12:14:36.645843 (+ 717us) negotiation.cc:107] Waiting for socket to connect
0809 12:14:36.646116 (+ 273us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.222.62:45505: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":477,"thread_start_us":284,"threads_started":1}
W20260809 12:14:36.647711 19251 heartbeater.cc:644] Failed to heartbeat to 127.0.222.62:45505 (0 consecutive failures): Network error: Failed to ping master at 127.0.222.62:45505: Client connection negotiation failed: client connection to 127.0.222.62:45505: connect: Connection refused (error 111)
I20260809 12:14:36.648499 19253 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 12:14:36.648168 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.222.62:45505 (local address 127.0.222.3:34965)
0809 12:14:36.648313 (+ 145us) negotiation.cc:107] Waiting for socket to connect
0809 12:14:36.648394 (+ 81us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.222.62:45505: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":50}
I20260809 12:14:36.649505 19253 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 12:14:36.649163 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.222.62:45505 (local address 127.0.222.3:53543)
0809 12:14:36.649295 (+ 132us) negotiation.cc:107] Waiting for socket to connect
0809 12:14:36.649399 (+ 104us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.222.62:45505: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":42}
W20260809 12:14:36.649744 19251 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
I20260809 12:14:36.651245 888 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskgH5Pjm/build/asan/bin/kudu as pid 19112
I20260809 12:14:36.706532 19255 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 12:14:36.705697 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.222.62:45505 (local address 127.0.222.1:60891)
0809 12:14:36.706195 (+ 498us) negotiation.cc:107] Waiting for socket to connect
0809 12:14:36.706290 (+ 95us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.222.62:45505: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":387,"thread_start_us":251,"threads_started":1}
I20260809 12:14:37.174366 19256 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 12:14:37.173543 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.222.62:45505 (local address 127.0.222.2:51505)
0809 12:14:37.174025 (+ 482us) negotiation.cc:107] Waiting for socket to connect
0809 12:14:37.174125 (+ 100us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.222.62:45505: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":374,"thread_start_us":218,"threads_started":1}
I20260809 12:14:37.651057 19257 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 12:14:37.650224 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.222.62:45505 (local address 127.0.222.3:38665)
0809 12:14:37.650673 (+ 449us) negotiation.cc:107] Waiting for socket to connect
0809 12:14:37.650762 (+ 89us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.222.62:45505: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":350,"thread_start_us":224,"threads_started":1}
I20260809 12:14:37.708302 19259 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 12:14:37.707536 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.222.62:45505 (local address 127.0.222.1:35367)
0809 12:14:37.708009 (+ 473us) negotiation.cc:107] Waiting for socket to connect
0809 12:14:37.708094 (+ 85us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.222.62:45505: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":368,"thread_start_us":218,"threads_started":1}
I20260809 12:14:38.175963 19260 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 12:14:38.175252 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.222.62:45505 (local address 127.0.222.2:34979)
0809 12:14:38.175720 (+ 468us) negotiation.cc:107] Waiting for socket to connect
0809 12:14:38.175800 (+ 80us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.222.62:45505: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":370,"thread_start_us":236,"threads_started":1}
I20260809 12:14:38.652624 19261 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 12:14:38.651902 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.222.62:45505 (local address 127.0.222.3:40281)
0809 12:14:38.652373 (+ 471us) negotiation.cc:107] Waiting for socket to connect
0809 12:14:38.652458 (+ 85us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.222.62:45505: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":372,"thread_start_us":225,"threads_started":1}
I20260809 12:14:38.710004 19262 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 12:14:38.709299 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.222.62:45505 (local address 127.0.222.1:57603)
0809 12:14:38.709735 (+ 436us) negotiation.cc:107] Waiting for socket to connect
0809 12:14:38.709820 (+ 85us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.222.62:45505: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":340,"thread_start_us":233,"threads_started":1}
I20260809 12:14:39.177887 19263 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 12:14:39.177000 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.222.62:45505 (local address 127.0.222.2:34017)
0809 12:14:39.177567 (+ 567us) negotiation.cc:107] Waiting for socket to connect
0809 12:14:39.177683 (+ 116us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.222.62:45505: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":429,"thread_start_us":309,"threads_started":1}
I20260809 12:14:39.654320 19264 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 12:14:39.653593 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.222.62:45505 (local address 127.0.222.3:43161)
0809 12:14:39.654055 (+ 462us) negotiation.cc:107] Waiting for socket to connect
0809 12:14:39.654136 (+ 81us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.222.62:45505: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":364,"thread_start_us":249,"threads_started":1}
I20260809 12:14:39.711866 19265 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 12:14:39.711092 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.222.62:45505 (local address 127.0.222.1:43169)
0809 12:14:39.711612 (+ 520us) negotiation.cc:107] Waiting for socket to connect
0809 12:14:39.711691 (+ 79us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.222.62:45505: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":418,"thread_start_us":300,"threads_started":1}
I20260809 12:14:40.179842 19266 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 12:14:40.178896 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.222.62:45505 (local address 127.0.222.2:60561)
0809 12:14:40.179495 (+ 599us) negotiation.cc:107] Waiting for socket to connect
0809 12:14:40.179576 (+ 81us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.222.62:45505: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":482,"thread_start_us":341,"threads_started":1}
I20260809 12:14:40.656018 19267 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 12:14:40.655277 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.222.62:45505 (local address 127.0.222.3:55063)
0809 12:14:40.655747 (+ 470us) negotiation.cc:107] Waiting for socket to connect
0809 12:14:40.655832 (+ 85us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.222.62:45505: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":367,"thread_start_us":241,"threads_started":1}
I20260809 12:14:40.713923 19268 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 12:14:40.712982 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.222.62:45505 (local address 127.0.222.1:57149)
0809 12:14:40.713605 (+ 623us) negotiation.cc:107] Waiting for socket to connect
0809 12:14:40.713711 (+ 106us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.222.62:45505: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":498,"thread_start_us":319,"threads_started":1}
I20260809 12:14:41.181494 19269 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 12:14:41.180748 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.222.62:45505 (local address 127.0.222.2:41595)
0809 12:14:41.181245 (+ 497us) negotiation.cc:107] Waiting for socket to connect
0809 12:14:41.181323 (+ 78us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.222.62:45505: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":387,"thread_start_us":262,"threads_started":1}
I20260809 12:14:41.657804 19270 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 12:14:41.657083 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.222.62:45505 (local address 127.0.222.3:33009)
0809 12:14:41.657556 (+ 473us) negotiation.cc:107] Waiting for socket to connect
0809 12:14:41.657636 (+ 80us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.222.62:45505: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":371,"thread_start_us":260,"threads_started":1}
I20260809 12:14:41.715976 19271 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 12:14:41.714963 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.222.62:45505 (local address 127.0.222.1:33181)
0809 12:14:41.715661 (+ 698us) negotiation.cc:107] Waiting for socket to connect
0809 12:14:41.715788 (+ 127us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.222.62:45505: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":553,"thread_start_us":388,"threads_started":1}
I20260809 12:14:42.183491 19272 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 12:14:42.182579 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.222.62:45505 (local address 127.0.222.2:56493)
0809 12:14:42.183219 (+ 640us) negotiation.cc:107] Waiting for socket to connect
0809 12:14:42.183314 (+ 95us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.222.62:45505: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":522,"thread_start_us":337,"threads_started":1}
I20260809 12:14:42.659452 19273 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 12:14:42.658654 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.222.62:45505 (local address 127.0.222.3:39331)
0809 12:14:42.659203 (+ 549us) negotiation.cc:107] Waiting for socket to connect
0809 12:14:42.659292 (+ 89us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.222.62:45505: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":457,"thread_start_us":337,"threads_started":1}
I20260809 12:14:42.717792 19274 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 12:14:42.716925 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.222.62:45505 (local address 127.0.222.1:40585)
0809 12:14:42.717495 (+ 570us) negotiation.cc:107] Waiting for socket to connect
0809 12:14:42.717576 (+ 81us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.222.62:45505: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":462,"thread_start_us":251,"threads_started":1}
I20260809 12:14:43.185182 19275 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 12:14:43.184384 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.222.62:45505 (local address 127.0.222.2:49809)
0809 12:14:43.184883 (+ 499us) negotiation.cc:107] Waiting for socket to connect
0809 12:14:43.184963 (+ 80us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.222.62:45505: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":394,"thread_start_us":230,"threads_started":1}
I20260809 12:14:43.661291 19276 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 12:14:43.660477 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.222.62:45505 (local address 127.0.222.3:51699)
0809 12:14:43.660992 (+ 515us) negotiation.cc:107] Waiting for socket to connect
0809 12:14:43.661080 (+ 88us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.222.62:45505: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":409,"thread_start_us":244,"threads_started":1}
I20260809 12:14:43.719533 19277 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 12:14:43.718743 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.222.62:45505 (local address 127.0.222.1:38813)
0809 12:14:43.719285 (+ 542us) negotiation.cc:107] Waiting for socket to connect
0809 12:14:43.719367 (+ 82us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.222.62:45505: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":444,"thread_start_us":311,"threads_started":1}
I20260809 12:14:44.186892 19278 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 12:14:44.186169 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.222.62:45505 (local address 127.0.222.2:43143)
0809 12:14:44.186617 (+ 448us) negotiation.cc:107] Waiting for socket to connect
0809 12:14:44.186696 (+ 79us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.222.62:45505: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":354,"thread_start_us":230,"threads_started":1}
I20260809 12:14:44.663081 19279 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 12:14:44.662197 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.222.62:45505 (local address 127.0.222.3:43945)
0809 12:14:44.662718 (+ 521us) negotiation.cc:107] Waiting for socket to connect
0809 12:14:44.662840 (+ 122us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.222.62:45505: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":394,"thread_start_us":262,"threads_started":1}
I20260809 12:14:44.721184 19280 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 12:14:44.720430 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.222.62:45505 (local address 127.0.222.1:50425)
0809 12:14:44.720923 (+ 493us) negotiation.cc:107] Waiting for socket to connect
0809 12:14:44.721009 (+ 86us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.222.62:45505: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":389,"thread_start_us":248,"threads_started":1}
I20260809 12:14:45.188607 19281 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 12:14:45.187793 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.222.62:45505 (local address 127.0.222.2:51305)
0809 12:14:45.188297 (+ 504us) negotiation.cc:107] Waiting for socket to connect
0809 12:14:45.188415 (+ 118us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.222.62:45505: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":399,"thread_start_us":257,"threads_started":1}
I20260809 12:14:45.665174 19282 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 12:14:45.664051 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.222.62:45505 (local address 127.0.222.3:35037)
0809 12:14:45.664863 (+ 812us) negotiation.cc:107] Waiting for socket to connect
0809 12:14:45.664988 (+ 125us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.222.62:45505: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":668,"thread_start_us":517,"threads_started":1}
I20260809 12:14:45.722873 19283 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 12:14:45.722173 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.222.62:45505 (local address 127.0.222.1:48167)
0809 12:14:45.722625 (+ 452us) negotiation.cc:107] Waiting for socket to connect
0809 12:14:45.722707 (+ 82us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.222.62:45505: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":357,"thread_start_us":249,"threads_started":1}
I20260809 12:14:46.190207 19284 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 12:14:46.189508 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.222.62:45505 (local address 127.0.222.2:40173)
0809 12:14:46.189960 (+ 452us) negotiation.cc:107] Waiting for socket to connect
0809 12:14:46.190045 (+ 85us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.222.62:45505: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":360,"thread_start_us":257,"threads_started":1}
I20260809 12:14:46.666922 19285 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 12:14:46.666203 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.222.62:45505 (local address 127.0.222.3:35061)
0809 12:14:46.666686 (+ 483us) negotiation.cc:107] Waiting for socket to connect
0809 12:14:46.666762 (+ 76us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.222.62:45505: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":389,"thread_start_us":273,"threads_started":1}
I20260809 12:14:46.724642 19286 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 12:14:46.723897 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.222.62:45505 (local address 127.0.222.1:49811)
0809 12:14:46.724359 (+ 462us) negotiation.cc:107] Waiting for socket to connect
0809 12:14:46.724459 (+ 100us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.222.62:45505: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":360,"thread_start_us":239,"threads_started":1}
I20260809 12:14:47.191953 19287 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 12:14:47.191221 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.222.62:45505 (local address 127.0.222.2:50813)
0809 12:14:47.191661 (+ 440us) negotiation.cc:107] Waiting for socket to connect
0809 12:14:47.191754 (+ 93us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.222.62:45505: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":348,"thread_start_us":222,"threads_started":1}
I20260809 12:14:47.668789 19288 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 12:14:47.667927 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.222.62:45505 (local address 127.0.222.3:59683)
0809 12:14:47.668509 (+ 582us) negotiation.cc:107] Waiting for socket to connect
0809 12:14:47.668606 (+ 97us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.222.62:45505: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":463,"thread_start_us":332,"threads_started":1}
I20260809 12:14:47.726276 19289 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 12:14:47.725569 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.222.62:45505 (local address 127.0.222.1:56235)
0809 12:14:47.726040 (+ 471us) negotiation.cc:107] Waiting for socket to connect
0809 12:14:47.726118 (+ 78us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.222.62:45505: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":354,"thread_start_us":244,"threads_started":1}
I20260809 12:14:48.193625 19290 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 12:14:48.192890 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.222.62:45505 (local address 127.0.222.2:45347)
0809 12:14:48.193369 (+ 479us) negotiation.cc:107] Waiting for socket to connect
0809 12:14:48.193453 (+ 84us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.222.62:45505: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":370,"thread_start_us":255,"threads_started":1}
I20260809 12:14:48.670446 19291 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 12:14:48.669761 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.222.62:45505 (local address 127.0.222.3:47647)
0809 12:14:48.670197 (+ 436us) negotiation.cc:107] Waiting for socket to connect
0809 12:14:48.670282 (+ 85us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.222.62:45505: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":332,"thread_start_us":222,"threads_started":1}
I20260809 12:14:48.728053 19292 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 12:14:48.727283 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.222.62:45505 (local address 127.0.222.1:36003)
0809 12:14:48.727736 (+ 453us) negotiation.cc:107] Waiting for socket to connect
0809 12:14:48.727861 (+ 125us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.222.62:45505: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":358,"thread_start_us":216,"threads_started":1}
I20260809 12:14:49.195528 19293 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 12:14:49.194678 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.222.62:45505 (local address 127.0.222.2:53033)
0809 12:14:49.195227 (+ 549us) negotiation.cc:107] Waiting for socket to connect
0809 12:14:49.195324 (+ 97us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.222.62:45505: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":451,"thread_start_us":294,"threads_started":1}
I20260809 12:14:49.672262 19294 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 12:14:49.671388 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.222.62:45505 (local address 127.0.222.3:35651)
0809 12:14:49.671941 (+ 553us) negotiation.cc:107] Waiting for socket to connect
0809 12:14:49.672044 (+ 103us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.222.62:45505: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":446,"thread_start_us":263,"threads_started":1}
I20260809 12:14:49.730057 19295 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 12:14:49.729216 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.222.62:45505 (local address 127.0.222.1:56555)
0809 12:14:49.729758 (+ 542us) negotiation.cc:107] Waiting for socket to connect
0809 12:14:49.729840 (+ 82us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.222.62:45505: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":421,"thread_start_us":313,"threads_started":1}
I20260809 12:14:50.197258 19296 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 12:14:50.196456 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.222.62:45505 (local address 127.0.222.2:49763)
0809 12:14:50.196956 (+ 500us) negotiation.cc:107] Waiting for socket to connect
0809 12:14:50.197057 (+ 101us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.222.62:45505: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":391,"thread_start_us":249,"threads_started":1}
I20260809 12:14:50.673827 19297 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 12:14:50.673116 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.222.62:45505 (local address 127.0.222.3:52681)
0809 12:14:50.673568 (+ 452us) negotiation.cc:107] Waiting for socket to connect
0809 12:14:50.673649 (+ 81us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.222.62:45505: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":340,"thread_start_us":225,"threads_started":1}
I20260809 12:14:50.732197 19298 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 12:14:50.731411 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.222.62:45505 (local address 127.0.222.1:33583)
0809 12:14:50.731885 (+ 474us) negotiation.cc:107] Waiting for socket to connect
0809 12:14:50.732002 (+ 117us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.222.62:45505: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":360,"thread_start_us":247,"threads_started":1}
I20260809 12:14:51.199112 19299 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 12:14:51.198310 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.222.62:45505 (local address 127.0.222.2:35769)
0809 12:14:51.198736 (+ 426us) negotiation.cc:107] Waiting for socket to connect
0809 12:14:51.198853 (+ 117us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.222.62:45505: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":328,"thread_start_us":220,"threads_started":1}
I20260809 12:14:51.662802 888 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskgH5Pjm/build/asan/bin/kudu pid 18784
I20260809 12:14:51.675819 19301 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 12:14:51.675035 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.222.62:45505 (local address 127.0.222.3:57247)
0809 12:14:51.675525 (+ 490us) negotiation.cc:107] Waiting for socket to connect
0809 12:14:51.675624 (+ 99us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.222.62:45505: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":383,"thread_start_us":256,"threads_started":1}
I20260809 12:14:51.681962 19300 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 12:14:51.664105 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37376 (local address 127.0.222.1:41049)
0809 12:14:51.664775 (+ 670us) server_negotiation.cc:206] Beginning negotiation
0809 12:14:51.664789 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0809 12:14:51.664831 (+ 42us) server_negotiation.cc:407] Connection header received
0809 12:14:51.666430 (+ 1599us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 12:14:51.666440 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 12:14:51.666874 (+ 434us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 12:14:51.667256 (+ 382us) server_negotiation.cc:226] Negotiated authn=SASL
0809 12:14:51.668865 (+ 1609us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 12:14:51.670263 (+ 1398us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 12:14:51.671680 (+ 1417us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 12:14:51.672378 (+ 698us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 12:14:51.672532 (+ 154us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 12:14:51.675531 (+ 2999us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 12:14:51.675541 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 12:14:51.675556 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0809 12:14:51.675593 (+ 37us) server_negotiation.cc:922] calling sasl_server_start()
0809 12:14:51.678602 (+ 3009us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 12:14:51.679371 (+ 769us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 12:14:51.679378 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 12:14:51.679384 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0809 12:14:51.679559 (+ 175us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 12:14:51.679948 (+ 389us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 12:14:51.679951 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 12:14:51.679953 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0809 12:14:51.680507 (+ 554us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 12:14:51.680739 (+ 232us) server_negotiation.cc:1035] Waiting for connection context
0809 12:14:51.681113 (+ 374us) server_negotiation.cc:299] Negotiation successful
0809 12:14:51.681645 (+ 532us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":504,"thread_start_us":352,"threads_started":1}
I20260809 12:14:51.683534 18958 generic_service.cc:196] Checking for leaks (request via RPC)
I20260809 12:14:51.768661 19303 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 12:14:51.767378 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.222.62:45505 (local address 127.0.222.1:50159)
0809 12:14:51.768051 (+ 673us) negotiation.cc:107] Waiting for socket to connect
0809 12:14:51.768309 (+ 258us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.222.62:45505: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":531,"thread_start_us":281,"threads_started":1}
I20260809 12:14:51.771147 888 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskgH5Pjm/build/asan/bin/kudu with pid 18784
I20260809 12:14:51.792673 888 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskgH5Pjm/build/asan/bin/kudu pid 18968
I20260809 12:14:51.810467 19304 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 12:14:51.793764 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52312 (local address 127.0.222.2:35933)
0809 12:14:51.794347 (+ 583us) server_negotiation.cc:206] Beginning negotiation
0809 12:14:51.794361 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0809 12:14:51.794398 (+ 37us) server_negotiation.cc:407] Connection header received
0809 12:14:51.796138 (+ 1740us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 12:14:51.796148 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 12:14:51.796642 (+ 494us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 12:14:51.796947 (+ 305us) server_negotiation.cc:226] Negotiated authn=SASL
0809 12:14:51.798454 (+ 1507us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 12:14:51.799814 (+ 1360us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 12:14:51.800937 (+ 1123us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 12:14:51.801580 (+ 643us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 12:14:51.801709 (+ 129us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 12:14:51.804349 (+ 2640us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 12:14:51.804356 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 12:14:51.804370 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0809 12:14:51.804410 (+ 40us) server_negotiation.cc:922] calling sasl_server_start()
0809 12:14:51.807561 (+ 3151us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 12:14:51.808209 (+ 648us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 12:14:51.808216 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 12:14:51.808223 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0809 12:14:51.808401 (+ 178us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 12:14:51.808750 (+ 349us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 12:14:51.808753 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 12:14:51.808755 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0809 12:14:51.809254 (+ 499us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 12:14:51.809486 (+ 232us) server_negotiation.cc:1035] Waiting for connection context
0809 12:14:51.809839 (+ 353us) server_negotiation.cc:299] Negotiation successful
0809 12:14:51.810149 (+ 310us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":434,"thread_start_us":299,"threads_started":1}
I20260809 12:14:51.811954 19104 generic_service.cc:196] Checking for leaks (request via RPC)
I20260809 12:14:51.896298 888 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskgH5Pjm/build/asan/bin/kudu with pid 18968
I20260809 12:14:51.917539 888 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskgH5Pjm/build/asan/bin/kudu pid 19112
I20260809 12:14:51.936964 19306 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 12:14:51.918647 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58394 (local address 127.0.222.3:43523)
0809 12:14:51.919345 (+ 698us) server_negotiation.cc:206] Beginning negotiation
0809 12:14:51.919359 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0809 12:14:51.919409 (+ 50us) server_negotiation.cc:407] Connection header received
0809 12:14:51.921039 (+ 1630us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 12:14:51.921049 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 12:14:51.921478 (+ 429us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 12:14:51.921832 (+ 354us) server_negotiation.cc:226] Negotiated authn=SASL
0809 12:14:51.923542 (+ 1710us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 12:14:51.924743 (+ 1201us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 12:14:51.926087 (+ 1344us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 12:14:51.926944 (+ 857us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 12:14:51.927189 (+ 245us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 12:14:51.930105 (+ 2916us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 12:14:51.930113 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 12:14:51.930129 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0809 12:14:51.930168 (+ 39us) server_negotiation.cc:922] calling sasl_server_start()
0809 12:14:51.933338 (+ 3170us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 12:14:51.934127 (+ 789us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 12:14:51.934134 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 12:14:51.934141 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0809 12:14:51.934321 (+ 180us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 12:14:51.934752 (+ 431us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 12:14:51.934758 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 12:14:51.934761 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0809 12:14:51.935436 (+ 675us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 12:14:51.935748 (+ 312us) server_negotiation.cc:1035] Waiting for connection context
0809 12:14:51.936203 (+ 455us) server_negotiation.cc:299] Negotiation successful
0809 12:14:51.936651 (+ 448us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":537,"thread_start_us":375,"threads_started":1}
I20260809 12:14:51.938372 19246 generic_service.cc:196] Checking for leaks (request via RPC)
I20260809 12:14:52.023500 888 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskgH5Pjm/build/asan/bin/kudu with pid 19112
2026-08-09T12:14:52Z chronyd exiting
[ OK ] SecurityITest.TestEnableAndDisableEncryptedTSK (65049 ms)
[----------] 30 tests from SecurityITest (309882 ms total)
[----------] 6 tests from EncryptionPolicyTest
[ RUN ] EncryptionPolicyTest.TestEncryptionPolicy/0
Loading random data
Initializing database '/tmp/dist-test-taskgH5Pjm/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786277382073938-888-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 09 12:14:52 dist-test-slave-2w3w krb5kdc[19310](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 09 12:14:52 dist-test-slave-2w3w krb5kdc[19310](info): set up 2 sockets
Aug 09 12:14:52 dist-test-slave-2w3w krb5kdc[19310](info): commencing operation
krb5kdc: starting...
W20260809 12:14:53.209781 888 mini_kdc.cc:121] Time spent starting KDC: real 1.142s user 0.004s sys 0.041s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskgH5Pjm/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786277382073938-888-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskgH5Pjm/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786277382073938-888-0/krb5kdc/test-user.keytab.
Aug 09 12:14:53 dist-test-slave-2w3w krb5kdc[19310](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786277693, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-09T12:14:53Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-09T12:14:53Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.222.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.222.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.222.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskgH5Pjm/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786277382073938-888-0/krb5kdc/kudu_127.0.222.62.keytab.
Entry for principal kudu/127.0.222.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskgH5Pjm/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786277382073938-888-0/krb5kdc/kudu_127.0.222.62.keytab.
I20260809 12:14:53.530680 888 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskgH5Pjm/build/asan/bin/kudu
/tmp/dist-test-taskgH5Pjm/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskgH5Pjm/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786277382073938-888-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskgH5Pjm/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786277382073938-888-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskgH5Pjm/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786277382073938-888-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskgH5Pjm/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786277382073938-888-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.222.62:45597
--webserver_interface=127.0.222.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskgH5Pjm/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786277382073938-888-0/krb5kdc/kudu_127.0.222.62.keytab
--principal=kudu/127.0.222.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.222.20:40777
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.222.62:45597
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskgH5Pjm/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786277382073938-888-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskgH5Pjm/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786277382073938-888-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskgH5Pjm/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786277382073938-888-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260809 12:14:53.877430 19324 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260809 12:14:53.877709 19324 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260809 12:14:53.877811 19324 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260809 12:14:53.885136 19324 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260809 12:14:53.885246 19324 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260809 12:14:53.885305 19324 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260809 12:14:53.885372 19324 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260809 12:14:53.885418 19324 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260809 12:14:53.895324 19324 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.222.20:40777
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskgH5Pjm/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786277382073938-888-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskgH5Pjm/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786277382073938-888-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.222.62:45597
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskgH5Pjm/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786277382073938-888-0/krb5kdc/kudu_127.0.222.62.keytab
--principal=kudu/127.0.222.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.222.62:45597
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskgH5Pjm/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786277382073938-888-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.222.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskgH5Pjm/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786277382073938-888-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 09 Aug 2026 11:06:41 UTC on ccycloud.quasar-bwoogy.root.comops.site
build id 4575991
ASAN enabled
I20260809 12:14:53.896601 19324 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260809 12:14:53.897922 19324 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260809 12:14:53.910512 19330 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 12:14:53.910804 19331 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 12:14:53.911017 19334 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260809 12:14:53.911458 19324 server_base.cc:1034] running on GCE node
I20260809 12:14:53.912266 19324 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260809 12:14:53.913728 19324 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260809 12:14:53.915014 19324 hybrid_clock.cc:648] HybridClock initialized: now 1786277693914927 us; error 62 us; skew 500 ppm
I20260809 12:14:53.915391 19324 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 09 12:14:53 dist-test-slave-2w3w krb5kdc[19310](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786277693, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.222.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260809 12:14:53.918624 19324 init.cc:377] Logged in from keytab as kudu/127.0.222.62@KRBTEST.COM (short username kudu)
I20260809 12:14:53.920281 19324 webserver.cc:460] Webserver started at http://127.0.222.62:37731/ using document root <none> and password file <none>
I20260809 12:14:53.920959 19324 fs_manager.cc:359] Metadata directory not provided
I20260809 12:14:53.921064 19324 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260809 12:14:53.921340 19324 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260809 12:14:53.924460 19324 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskgH5Pjm/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786277382073938-888-0/minicluster-data/master-0/data/instance:
uuid: "c7606c7b6c614665a58919c9f580fc7e"
format_stamp: "Formatted at 2026-08-09 12:14:53 on dist-test-slave-2w3w"
server_key: "274331882c76985e4c736a74ad7ac566"
server_key_iv: "2a41ee1c7e1743cf833cdefcbfdfc85c"
server_key_version: "encryptionkey@0"
I20260809 12:14:53.925233 19324 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskgH5Pjm/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786277382073938-888-0/minicluster-data/master-0/wal/instance:
uuid: "c7606c7b6c614665a58919c9f580fc7e"
format_stamp: "Formatted at 2026-08-09 12:14:53 on dist-test-slave-2w3w"
server_key: "274331882c76985e4c736a74ad7ac566"
server_key_iv: "2a41ee1c7e1743cf833cdefcbfdfc85c"
server_key_version: "encryptionkey@0"
I20260809 12:14:53.930673 19324 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.003s sys 0.003s
I20260809 12:14:53.934417 19343 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260809 12:14:53.935935 19324 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260809 12:14:53.936089 19324 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskgH5Pjm/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786277382073938-888-0/minicluster-data/master-0/data,/tmp/dist-test-taskgH5Pjm/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786277382073938-888-0/minicluster-data/master-0/wal
uuid: "c7606c7b6c614665a58919c9f580fc7e"
format_stamp: "Formatted at 2026-08-09 12:14:53 on dist-test-slave-2w3w"
server_key: "274331882c76985e4c736a74ad7ac566"
server_key_iv: "2a41ee1c7e1743cf833cdefcbfdfc85c"
server_key_version: "encryptionkey@0"
I20260809 12:14:53.936266 19324 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskgH5Pjm/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786277382073938-888-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskgH5Pjm/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786277382073938-888-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskgH5Pjm/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786277382073938-888-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260809 12:14:53.951155 19324 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260809 12:14:53.955415 19324 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260809 12:14:53.955672 19324 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260809 12:14:53.975812 19324 rpc_server.cc:307] RPC server started. Bound to: 127.0.222.62:45597
I20260809 12:14:53.975840 19396 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.222.62:45597 every 8 connection(s)
I20260809 12:14:53.977298 19324 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskgH5Pjm/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786277382073938-888-0/minicluster-data/master-0/data/info.pb
I20260809 12:14:53.981151 19397 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260809 12:14:53.982112 888 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskgH5Pjm/build/asan/bin/kudu as pid 19324
I20260809 12:14:53.982368 888 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskgH5Pjm/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786277382073938-888-0/minicluster-data/master-0/wal/instance
I20260809 12:14:53.983345 888 external_mini_cluster.cc:1428] Setting key 0d691ba2065cb2746659405e8750ef4c
I20260809 12:14:53.996485 19397 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P c7606c7b6c614665a58919c9f580fc7e: Bootstrap starting.
Aug 09 12:14:53 dist-test-slave-2w3w krb5kdc[19310](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786277693, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.222.62@KRBTEST.COM
I20260809 12:14:54.000365 19397 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P c7606c7b6c614665a58919c9f580fc7e: Neither blocks nor log segments found. Creating new log.
I20260809 12:14:54.001587 19397 log.cc:824] T 00000000000000000000000000000000 P c7606c7b6c614665a58919c9f580fc7e: Log is configured to *not* fsync() on all Append() calls
I20260809 12:14:54.004683 19397 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P c7606c7b6c614665a58919c9f580fc7e: No bootstrap required, opened a new log
I20260809 12:14:54.008311 19400 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 12:14:53.985601 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39158 (local address 127.0.222.62:45597)
0809 12:14:53.986571 (+ 970us) server_negotiation.cc:206] Beginning negotiation
0809 12:14:53.986583 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0809 12:14:53.986626 (+ 43us) server_negotiation.cc:407] Connection header received
0809 12:14:53.988365 (+ 1739us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 12:14:53.988391 (+ 26us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 12:14:53.988796 (+ 405us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 12:14:53.989128 (+ 332us) server_negotiation.cc:226] Negotiated authn=SASL
0809 12:14:53.991076 (+ 1948us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 12:14:53.992373 (+ 1297us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 12:14:53.994382 (+ 2009us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 12:14:53.995137 (+ 755us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 12:14:53.995266 (+ 129us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 12:14:54.000377 (+ 5111us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 12:14:54.000385 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 12:14:54.000398 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0809 12:14:54.000441 (+ 43us) server_negotiation.cc:922] calling sasl_server_start()
0809 12:14:54.003941 (+ 3500us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 12:14:54.004717 (+ 776us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 12:14:54.004726 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 12:14:54.004736 (+ 10us) server_negotiation.cc:967] calling sasl_server_step()
0809 12:14:54.004995 (+ 259us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 12:14:54.005403 (+ 408us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 12:14:54.005409 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 12:14:54.005412 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0809 12:14:54.005959 (+ 547us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 12:14:54.006305 (+ 346us) server_negotiation.cc:1035] Waiting for connection context
0809 12:14:54.006810 (+ 505us) server_negotiation.cc:299] Negotiation successful
0809 12:14:54.007299 (+ 489us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":740,"thread_start_us":515,"threads_started":1}
I20260809 12:14:54.012930 19397 raft_consensus.cc:348] T 00000000000000000000000000000000 P c7606c7b6c614665a58919c9f580fc7e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c7606c7b6c614665a58919c9f580fc7e" member_type: VOTER last_known_addr { host: "127.0.222.62" port: 45597 } }
I20260809 12:14:54.013330 19397 raft_consensus.cc:374] T 00000000000000000000000000000000 P c7606c7b6c614665a58919c9f580fc7e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260809 12:14:54.013448 19397 raft_consensus.cc:729] T 00000000000000000000000000000000 P c7606c7b6c614665a58919c9f580fc7e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c7606c7b6c614665a58919c9f580fc7e, State: Initialized, Role: FOLLOWER
I20260809 12:14:54.014068 19397 consensus_queue.cc:260] T 00000000000000000000000000000000 P c7606c7b6c614665a58919c9f580fc7e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c7606c7b6c614665a58919c9f580fc7e" member_type: VOTER last_known_addr { host: "127.0.222.62" port: 45597 } }
I20260809 12:14:54.014322 19397 raft_consensus.cc:388] T 00000000000000000000000000000000 P c7606c7b6c614665a58919c9f580fc7e [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260809 12:14:54.014484 19397 raft_consensus.cc:482] T 00000000000000000000000000000000 P c7606c7b6c614665a58919c9f580fc7e [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260809 12:14:54.014667 19397 raft_consensus.cc:3037] T 00000000000000000000000000000000 P c7606c7b6c614665a58919c9f580fc7e [term 0 FOLLOWER]: Advancing to term 1
I20260809 12:14:54.017326 19397 raft_consensus.cc:504] T 00000000000000000000000000000000 P c7606c7b6c614665a58919c9f580fc7e [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c7606c7b6c614665a58919c9f580fc7e" member_type: VOTER last_known_addr { host: "127.0.222.62" port: 45597 } }
I20260809 12:14:54.017868 19397 leader_election.cc:302] T 00000000000000000000000000000000 P c7606c7b6c614665a58919c9f580fc7e [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: c7606c7b6c614665a58919c9f580fc7e; no voters:
I20260809 12:14:54.018502 19397 leader_election.cc:288] T 00000000000000000000000000000000 P c7606c7b6c614665a58919c9f580fc7e [CANDIDATE]: Term 1 election: Requested vote from peers
I20260809 12:14:54.018646 19402 raft_consensus.cc:2781] T 00000000000000000000000000000000 P c7606c7b6c614665a58919c9f580fc7e [term 1 FOLLOWER]: Leader election won for term 1
I20260809 12:14:54.019443 19402 raft_consensus.cc:686] T 00000000000000000000000000000000 P c7606c7b6c614665a58919c9f580fc7e [term 1 LEADER]: Becoming Leader. State: Replica: c7606c7b6c614665a58919c9f580fc7e, State: Running, Role: LEADER
I20260809 12:14:54.020010 19402 consensus_queue.cc:237] T 00000000000000000000000000000000 P c7606c7b6c614665a58919c9f580fc7e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c7606c7b6c614665a58919c9f580fc7e" member_type: VOTER last_known_addr { host: "127.0.222.62" port: 45597 } }
I20260809 12:14:54.020728 19397 sys_catalog.cc:564] T 00000000000000000000000000000000 P c7606c7b6c614665a58919c9f580fc7e [sys.catalog]: configured and running, proceeding with master startup.
I20260809 12:14:54.022866 19404 sys_catalog.cc:455] T 00000000000000000000000000000000 P c7606c7b6c614665a58919c9f580fc7e [sys.catalog]: SysCatalogTable state changed. Reason: New leader c7606c7b6c614665a58919c9f580fc7e. Latest consensus state: current_term: 1 leader_uuid: "c7606c7b6c614665a58919c9f580fc7e" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c7606c7b6c614665a58919c9f580fc7e" member_type: VOTER last_known_addr { host: "127.0.222.62" port: 45597 } } }
I20260809 12:14:54.023102 19403 sys_catalog.cc:455] T 00000000000000000000000000000000 P c7606c7b6c614665a58919c9f580fc7e [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "c7606c7b6c614665a58919c9f580fc7e" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c7606c7b6c614665a58919c9f580fc7e" member_type: VOTER last_known_addr { host: "127.0.222.62" port: 45597 } } }
I20260809 12:14:54.023350 19404 sys_catalog.cc:458] T 00000000000000000000000000000000 P c7606c7b6c614665a58919c9f580fc7e [sys.catalog]: This master's current role is: LEADER
I20260809 12:14:54.023363 19403 sys_catalog.cc:458] T 00000000000000000000000000000000 P c7606c7b6c614665a58919c9f580fc7e [sys.catalog]: This master's current role is: LEADER
I20260809 12:14:54.025991 19409 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260809 12:14:54.031486 19409 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260809 12:14:54.040719 19409 catalog_manager.cc:1347] Generated new cluster ID: b4d46c69782240ba96543b823652d86d
I20260809 12:14:54.040838 19409 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260809 12:14:54.066210 19409 catalog_manager.cc:1370] Generated new certificate authority record
I20260809 12:14:54.068270 19409 catalog_manager.cc:1504] Loading token signing keys...
I20260809 12:14:54.076473 19409 catalog_manager.cc:5868] T 00000000000000000000000000000000 P c7606c7b6c614665a58919c9f580fc7e: Generated new TSK 0
I20260809 12:14:54.077343 19409 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.0.222.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.222.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.222.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskgH5Pjm/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786277382073938-888-0/krb5kdc/kudu_127.0.222.1.keytab.
Entry for principal kudu/127.0.222.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskgH5Pjm/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786277382073938-888-0/krb5kdc/kudu_127.0.222.1.keytab.
I20260809 12:14:54.169049 888 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskgH5Pjm/build/asan/bin/kudu
/tmp/dist-test-taskgH5Pjm/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskgH5Pjm/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786277382073938-888-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskgH5Pjm/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786277382073938-888-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskgH5Pjm/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786277382073938-888-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskgH5Pjm/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786277382073938-888-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.222.1:0
--local_ip_for_outbound_sockets=127.0.222.1
--webserver_interface=127.0.222.1
--webserver_port=0
--tserver_master_addrs=127.0.222.62:45597
--keytab_file=/tmp/dist-test-taskgH5Pjm/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786277382073938-888-0/krb5kdc/kudu_127.0.222.1.keytab
--principal=kudu/127.0.222.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.222.20:40777
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskgH5Pjm/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786277382073938-888-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskgH5Pjm/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786277382073938-888-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskgH5Pjm/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786277382073938-888-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260809 12:14:54.520306 19423 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260809 12:14:54.520593 19423 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260809 12:14:54.520787 19423 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260809 12:14:54.528410 19423 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260809 12:14:54.528539 19423 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260809 12:14:54.528721 19423 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.222.1
I20260809 12:14:54.538487 19423 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.222.20:40777
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskgH5Pjm/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786277382073938-888-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskgH5Pjm/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786277382073938-888-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskgH5Pjm/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786277382073938-888-0/krb5kdc/kudu_127.0.222.1.keytab
--principal=kudu/127.0.222.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.222.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskgH5Pjm/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786277382073938-888-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.222.1
--webserver_port=0
--tserver_master_addrs=127.0.222.62:45597
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.222.1
--log_dir=/tmp/dist-test-taskgH5Pjm/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786277382073938-888-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 09 Aug 2026 11:06:41 UTC on ccycloud.quasar-bwoogy.root.comops.site
build id 4575991
ASAN enabled
I20260809 12:14:54.539906 19423 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260809 12:14:54.541326 19423 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260809 12:14:54.554656 19428 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 12:14:54.554765 19429 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 12:14:54.554708 19431 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260809 12:14:54.555686 19423 server_base.cc:1034] running on GCE node
I20260809 12:14:54.556375 19423 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260809 12:14:54.557404 19423 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260809 12:14:54.558606 19423 hybrid_clock.cc:648] HybridClock initialized: now 1786277694558563 us; error 39 us; skew 500 ppm
I20260809 12:14:54.558934 19423 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 09 12:14:54 dist-test-slave-2w3w krb5kdc[19310](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786277694, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.222.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260809 12:14:54.562209 19423 init.cc:377] Logged in from keytab as kudu/127.0.222.1@KRBTEST.COM (short username kudu)
I20260809 12:14:54.563851 19423 webserver.cc:460] Webserver started at http://127.0.222.1:32983/ using document root <none> and password file <none>
I20260809 12:14:54.564563 19423 fs_manager.cc:359] Metadata directory not provided
I20260809 12:14:54.564713 19423 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260809 12:14:54.565042 19423 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260809 12:14:54.568009 19423 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskgH5Pjm/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786277382073938-888-0/minicluster-data/ts-0/data/instance:
uuid: "b88447d41a7f4ca4b3f742239780061b"
format_stamp: "Formatted at 2026-08-09 12:14:54 on dist-test-slave-2w3w"
server_key: "cf301a45f3d4ec677248e2b284ac9bae"
server_key_iv: "3a251ea08b2e0ab50cf1f1a15dcd4002"
server_key_version: "encryptionkey@0"
I20260809 12:14:54.568800 19423 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskgH5Pjm/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786277382073938-888-0/minicluster-data/ts-0/wal/instance:
uuid: "b88447d41a7f4ca4b3f742239780061b"
format_stamp: "Formatted at 2026-08-09 12:14:54 on dist-test-slave-2w3w"
server_key: "cf301a45f3d4ec677248e2b284ac9bae"
server_key_iv: "3a251ea08b2e0ab50cf1f1a15dcd4002"
server_key_version: "encryptionkey@0"
I20260809 12:14:54.573880 19423 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.000s
I20260809 12:14:54.577446 19439 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260809 12:14:54.578763 19423 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.001s
I20260809 12:14:54.578940 19423 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskgH5Pjm/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786277382073938-888-0/minicluster-data/ts-0/data,/tmp/dist-test-taskgH5Pjm/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786277382073938-888-0/minicluster-data/ts-0/wal
uuid: "b88447d41a7f4ca4b3f742239780061b"
format_stamp: "Formatted at 2026-08-09 12:14:54 on dist-test-slave-2w3w"
server_key: "cf301a45f3d4ec677248e2b284ac9bae"
server_key_iv: "3a251ea08b2e0ab50cf1f1a15dcd4002"
server_key_version: "encryptionkey@0"
I20260809 12:14:54.579200 19423 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskgH5Pjm/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786277382073938-888-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskgH5Pjm/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786277382073938-888-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskgH5Pjm/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786277382073938-888-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260809 12:14:54.606702 19423 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260809 12:14:54.610910 19423 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260809 12:14:54.611241 19423 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260809 12:14:54.612247 19423 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260809 12:14:54.613615 19423 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260809 12:14:54.613690 19423 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260809 12:14:54.613855 19423 ts_tablet_manager.cc:610] Registered 0 tablets
I20260809 12:14:54.613940 19423 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260809 12:14:54.648689 19423 rpc_server.cc:307] RPC server started. Bound to: 127.0.222.1:37951
I20260809 12:14:54.648738 19560 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.222.1:37951 every 8 connection(s)
I20260809 12:14:54.650043 19423 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskgH5Pjm/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786277382073938-888-0/minicluster-data/ts-0/data/info.pb
I20260809 12:14:54.655830 888 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskgH5Pjm/build/asan/bin/kudu as pid 19423
I20260809 12:14:54.656075 888 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskgH5Pjm/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786277382073938-888-0/minicluster-data/ts-0/wal/instance
I20260809 12:14:54.656970 888 external_mini_cluster.cc:1428] Setting key e51a306fd9fec64d5862c898ae86b184
Aug 09 12:14:54 dist-test-slave-2w3w krb5kdc[19310](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786277694, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.222.1@KRBTEST.COM for kudu/127.0.222.62@KRBTEST.COM
I20260809 12:14:54.674695 19563 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 12:14:54.652677 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.222.1:53511 (local address 127.0.222.62:45597)
0809 12:14:54.653127 (+ 450us) server_negotiation.cc:206] Beginning negotiation
0809 12:14:54.653132 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0809 12:14:54.654143 (+ 1011us) server_negotiation.cc:407] Connection header received
0809 12:14:54.656430 (+ 2287us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 12:14:54.656435 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 12:14:54.656511 (+ 76us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 12:14:54.656607 (+ 96us) server_negotiation.cc:226] Negotiated authn=SASL
0809 12:14:54.658558 (+ 1951us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 12:14:54.659566 (+ 1008us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 12:14:54.662209 (+ 2643us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 12:14:54.662611 (+ 402us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 12:14:54.662720 (+ 109us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 12:14:54.667180 (+ 4460us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 12:14:54.667189 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 12:14:54.667197 (+ 8us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0809 12:14:54.667245 (+ 48us) server_negotiation.cc:922] calling sasl_server_start()
0809 12:14:54.670929 (+ 3684us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 12:14:54.671880 (+ 951us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 12:14:54.671888 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 12:14:54.671894 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0809 12:14:54.672050 (+ 156us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 12:14:54.672548 (+ 498us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 12:14:54.672555 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 12:14:54.672561 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0809 12:14:54.672941 (+ 380us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 12:14:54.673100 (+ 159us) server_negotiation.cc:1035] Waiting for connection context
0809 12:14:54.673986 (+ 886us) server_negotiation.cc:299] Negotiation successful
0809 12:14:54.674252 (+ 266us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":342,"thread_start_us":229,"threads_started":1}
I20260809 12:14:54.675403 19564 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 12:14:54.653233 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.222.62:45597 (local address 127.0.222.1:53511)
0809 12:14:54.653975 (+ 742us) negotiation.cc:107] Waiting for socket to connect
0809 12:14:54.654018 (+ 43us) client_negotiation.cc:174] Beginning negotiation
0809 12:14:54.656121 (+ 2103us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 12:14:54.656889 (+ 768us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 12:14:54.656900 (+ 11us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 12:14:54.657677 (+ 777us) client_negotiation.cc:189] Negotiated authn=SASL
0809 12:14:54.658388 (+ 711us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 12:14:54.658407 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 12:14:54.659737 (+ 1330us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 12:14:54.659744 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 12:14:54.661977 (+ 2233us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 12:14:54.661995 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 12:14:54.662845 (+ 850us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 12:14:54.662854 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 12:14:54.663216 (+ 362us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 12:14:54.663941 (+ 725us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0809 12:14:54.663973 (+ 32us) client_negotiation.cc:656] calling sasl_client_start()
0809 12:14:54.666905 (+ 2932us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0809 12:14:54.671132 (+ 4227us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0809 12:14:54.671143 (+ 11us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0809 12:14:54.671163 (+ 20us) client_negotiation.cc:760] calling sasl_client_step()
0809 12:14:54.671751 (+ 588us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0809 12:14:54.672177 (+ 426us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0809 12:14:54.672184 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0809 12:14:54.672190 (+ 6us) client_negotiation.cc:760] calling sasl_client_step()
0809 12:14:54.672426 (+ 236us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0809 12:14:54.673072 (+ 646us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0809 12:14:54.673082 (+ 10us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0809 12:14:54.673516 (+ 434us) client_negotiation.cc:769] Sending connection context
0809 12:14:54.673815 (+ 299us) client_negotiation.cc:240] Negotiation successful
0809 12:14:54.674213 (+ 398us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":504,"thread_start_us":319,"threads_started":1}
I20260809 12:14:54.677034 19561 heartbeater.cc:347] Connected to a master server at 127.0.222.62:45597
I20260809 12:14:54.677361 19561 heartbeater.cc:464] Registering TS with master...
I20260809 12:14:54.678100 19561 heartbeater.cc:511] Master 127.0.222.62:45597 requested a full tablet report, sending...
I20260809 12:14:54.680295 19360 ts_manager.cc:194] Registered new tserver with Master: b88447d41a7f4ca4b3f742239780061b (127.0.222.1:37951)
I20260809 12:14:54.681754 19360 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.222.1@KRBTEST.COM'} at 127.0.222.1:53511
WARNING: no policy specified for kudu/127.0.222.2@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.222.2@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.222.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskgH5Pjm/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786277382073938-888-0/krb5kdc/kudu_127.0.222.2.keytab.
Entry for principal kudu/127.0.222.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskgH5Pjm/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786277382073938-888-0/krb5kdc/kudu_127.0.222.2.keytab.
I20260809 12:14:54.730926 888 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskgH5Pjm/build/asan/bin/kudu
/tmp/dist-test-taskgH5Pjm/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskgH5Pjm/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786277382073938-888-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskgH5Pjm/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786277382073938-888-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskgH5Pjm/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786277382073938-888-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskgH5Pjm/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786277382073938-888-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.222.2:0
--local_ip_for_outbound_sockets=127.0.222.2
--webserver_interface=127.0.222.2
--webserver_port=0
--tserver_master_addrs=127.0.222.62:45597
--keytab_file=/tmp/dist-test-taskgH5Pjm/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786277382073938-888-0/krb5kdc/kudu_127.0.222.2.keytab
--principal=kudu/127.0.222.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.222.20:40777
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskgH5Pjm/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786277382073938-888-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskgH5Pjm/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786277382073938-888-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskgH5Pjm/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786277382073938-888-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260809 12:14:55.081797 19567 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260809 12:14:55.082062 19567 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260809 12:14:55.082156 19567 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260809 12:14:55.089324 19567 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260809 12:14:55.089438 19567 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260809 12:14:55.089552 19567 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.222.2
I20260809 12:14:55.099268 19567 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.222.20:40777
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskgH5Pjm/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786277382073938-888-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskgH5Pjm/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786277382073938-888-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskgH5Pjm/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786277382073938-888-0/krb5kdc/kudu_127.0.222.2.keytab
--principal=kudu/127.0.222.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.222.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskgH5Pjm/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786277382073938-888-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.222.2
--webserver_port=0
--tserver_master_addrs=127.0.222.62:45597
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.222.2
--log_dir=/tmp/dist-test-taskgH5Pjm/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786277382073938-888-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 09 Aug 2026 11:06:41 UTC on ccycloud.quasar-bwoogy.root.comops.site
build id 4575991
ASAN enabled
I20260809 12:14:55.100498 19567 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260809 12:14:55.101796 19567 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260809 12:14:55.115227 19572 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 12:14:55.115309 19573 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 12:14:55.115265 19575 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260809 12:14:55.116348 19567 server_base.cc:1034] running on GCE node
I20260809 12:14:55.116981 19567 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260809 12:14:55.117919 19567 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260809 12:14:55.119132 19567 hybrid_clock.cc:648] HybridClock initialized: now 1786277695119085 us; error 41 us; skew 500 ppm
I20260809 12:14:55.119455 19567 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 09 12:14:55 dist-test-slave-2w3w krb5kdc[19310](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786277695, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.222.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260809 12:14:55.122829 19567 init.cc:377] Logged in from keytab as kudu/127.0.222.2@KRBTEST.COM (short username kudu)
I20260809 12:14:55.124389 19567 webserver.cc:460] Webserver started at http://127.0.222.2:37471/ using document root <none> and password file <none>
I20260809 12:14:55.125115 19567 fs_manager.cc:359] Metadata directory not provided
I20260809 12:14:55.125265 19567 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260809 12:14:55.125603 19567 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260809 12:14:55.128697 19567 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskgH5Pjm/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786277382073938-888-0/minicluster-data/ts-1/data/instance:
uuid: "0053eca7b1ac45d8b5672e15b7724512"
format_stamp: "Formatted at 2026-08-09 12:14:55 on dist-test-slave-2w3w"
server_key: "0fe50a47ddb210d5a2b1b1b0f79d217a"
server_key_iv: "0280811c9c6c2336d8ba8e936ce38e5f"
server_key_version: "encryptionkey@0"
I20260809 12:14:55.129518 19567 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskgH5Pjm/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786277382073938-888-0/minicluster-data/ts-1/wal/instance:
uuid: "0053eca7b1ac45d8b5672e15b7724512"
format_stamp: "Formatted at 2026-08-09 12:14:55 on dist-test-slave-2w3w"
server_key: "0fe50a47ddb210d5a2b1b1b0f79d217a"
server_key_iv: "0280811c9c6c2336d8ba8e936ce38e5f"
server_key_version: "encryptionkey@0"
I20260809 12:14:55.134802 19567 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.007s sys 0.000s
I20260809 12:14:55.138279 19583 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260809 12:14:55.139734 19567 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260809 12:14:55.139930 19567 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskgH5Pjm/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786277382073938-888-0/minicluster-data/ts-1/data,/tmp/dist-test-taskgH5Pjm/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786277382073938-888-0/minicluster-data/ts-1/wal
uuid: "0053eca7b1ac45d8b5672e15b7724512"
format_stamp: "Formatted at 2026-08-09 12:14:55 on dist-test-slave-2w3w"
server_key: "0fe50a47ddb210d5a2b1b1b0f79d217a"
server_key_iv: "0280811c9c6c2336d8ba8e936ce38e5f"
server_key_version: "encryptionkey@0"
I20260809 12:14:55.140134 19567 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskgH5Pjm/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786277382073938-888-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskgH5Pjm/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786277382073938-888-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskgH5Pjm/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786277382073938-888-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260809 12:14:55.151353 19567 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260809 12:14:55.155722 19567 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260809 12:14:55.156028 19567 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260809 12:14:55.157024 19567 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260809 12:14:55.158380 19567 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260809 12:14:55.158485 19567 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260809 12:14:55.158648 19567 ts_tablet_manager.cc:610] Registered 0 tablets
I20260809 12:14:55.158721 19567 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260809 12:14:55.192890 19567 rpc_server.cc:307] RPC server started. Bound to: 127.0.222.2:40181
I20260809 12:14:55.192935 19708 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.222.2:40181 every 8 connection(s)
I20260809 12:14:55.194258 19567 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskgH5Pjm/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786277382073938-888-0/minicluster-data/ts-1/data/info.pb
I20260809 12:14:55.202797 888 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskgH5Pjm/build/asan/bin/kudu as pid 19567
I20260809 12:14:55.203052 888 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskgH5Pjm/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786277382073938-888-0/minicluster-data/ts-1/wal/instance
I20260809 12:14:55.203517 888 external_mini_cluster.cc:1428] Setting key 25cf206df7983aff889b9b9addb70b50
Aug 09 12:14:55 dist-test-slave-2w3w krb5kdc[19310](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786277695, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.222.2@KRBTEST.COM for kudu/127.0.222.62@KRBTEST.COM
I20260809 12:14:55.216812 19711 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 12:14:55.197031 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.222.2:56231 (local address 127.0.222.62:45597)
0809 12:14:55.197538 (+ 507us) server_negotiation.cc:206] Beginning negotiation
0809 12:14:55.197542 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0809 12:14:55.198360 (+ 818us) server_negotiation.cc:407] Connection header received
0809 12:14:55.200345 (+ 1985us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 12:14:55.200350 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 12:14:55.200454 (+ 104us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 12:14:55.200587 (+ 133us) server_negotiation.cc:226] Negotiated authn=SASL
0809 12:14:55.202148 (+ 1561us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 12:14:55.202966 (+ 818us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 12:14:55.204843 (+ 1877us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 12:14:55.205400 (+ 557us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 12:14:55.205510 (+ 110us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 12:14:55.209406 (+ 3896us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 12:14:55.209416 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 12:14:55.209423 (+ 7us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0809 12:14:55.209471 (+ 48us) server_negotiation.cc:922] calling sasl_server_start()
0809 12:14:55.213247 (+ 3776us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 12:14:55.214036 (+ 789us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 12:14:55.214045 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 12:14:55.214051 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0809 12:14:55.214227 (+ 176us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 12:14:55.214675 (+ 448us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 12:14:55.214682 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 12:14:55.214688 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0809 12:14:55.215148 (+ 460us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 12:14:55.215320 (+ 172us) server_negotiation.cc:1035] Waiting for connection context
0809 12:14:55.216036 (+ 716us) server_negotiation.cc:299] Negotiation successful
0809 12:14:55.216324 (+ 288us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":397,"thread_start_us":261,"threads_started":1}
I20260809 12:14:55.217535 19713 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 12:14:55.197456 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.222.62:45597 (local address 127.0.222.2:56231)
0809 12:14:55.198179 (+ 723us) negotiation.cc:107] Waiting for socket to connect
0809 12:14:55.198228 (+ 49us) client_negotiation.cc:174] Beginning negotiation
0809 12:14:55.200060 (+ 1832us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 12:14:55.200840 (+ 780us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 12:14:55.200848 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 12:14:55.201378 (+ 530us) client_negotiation.cc:189] Negotiated authn=SASL
0809 12:14:55.201956 (+ 578us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 12:14:55.201972 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 12:14:55.203169 (+ 1197us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 12:14:55.203173 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 12:14:55.204695 (+ 1522us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 12:14:55.204716 (+ 21us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 12:14:55.205646 (+ 930us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 12:14:55.205655 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 12:14:55.205875 (+ 220us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 12:14:55.206602 (+ 727us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0809 12:14:55.206625 (+ 23us) client_negotiation.cc:656] calling sasl_client_start()
0809 12:14:55.209138 (+ 2513us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0809 12:14:55.213422 (+ 4284us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0809 12:14:55.213431 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0809 12:14:55.213446 (+ 15us) client_negotiation.cc:760] calling sasl_client_step()
0809 12:14:55.213906 (+ 460us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0809 12:14:55.214350 (+ 444us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0809 12:14:55.214353 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0809 12:14:55.214356 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0809 12:14:55.214559 (+ 203us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0809 12:14:55.215287 (+ 728us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0809 12:14:55.215293 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0809 12:14:55.215648 (+ 355us) client_negotiation.cc:769] Sending connection context
0809 12:14:55.215899 (+ 251us) client_negotiation.cc:240] Negotiation successful
0809 12:14:55.216334 (+ 435us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":496,"thread_start_us":297,"threads_started":1}
I20260809 12:14:55.219223 19709 heartbeater.cc:347] Connected to a master server at 127.0.222.62:45597
I20260809 12:14:55.219559 19709 heartbeater.cc:464] Registering TS with master...
I20260809 12:14:55.220335 19709 heartbeater.cc:511] Master 127.0.222.62:45597 requested a full tablet report, sending...
I20260809 12:14:55.222033 19360 ts_manager.cc:194] Registered new tserver with Master: 0053eca7b1ac45d8b5672e15b7724512 (127.0.222.2:40181)
I20260809 12:14:55.223027 19360 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.222.2@KRBTEST.COM'} at 127.0.222.2:56231
WARNING: no policy specified for kudu/127.0.222.3@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.222.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.222.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskgH5Pjm/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786277382073938-888-0/krb5kdc/kudu_127.0.222.3.keytab.
Entry for principal kudu/127.0.222.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskgH5Pjm/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786277382073938-888-0/krb5kdc/kudu_127.0.222.3.keytab.
I20260809 12:14:55.277922 888 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskgH5Pjm/build/asan/bin/kudu
/tmp/dist-test-taskgH5Pjm/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskgH5Pjm/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786277382073938-888-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskgH5Pjm/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786277382073938-888-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskgH5Pjm/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786277382073938-888-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskgH5Pjm/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786277382073938-888-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.222.3:0
--local_ip_for_outbound_sockets=127.0.222.3
--webserver_interface=127.0.222.3
--webserver_port=0
--tserver_master_addrs=127.0.222.62:45597
--keytab_file=/tmp/dist-test-taskgH5Pjm/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786277382073938-888-0/krb5kdc/kudu_127.0.222.3.keytab
--principal=kudu/127.0.222.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.222.20:40777
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskgH5Pjm/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786277382073938-888-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskgH5Pjm/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786277382073938-888-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskgH5Pjm/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786277382073938-888-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260809 12:14:55.620090 19716 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260809 12:14:55.620388 19716 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260809 12:14:55.620548 19716 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260809 12:14:55.628029 19716 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260809 12:14:55.628141 19716 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260809 12:14:55.628325 19716 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.222.3
I20260809 12:14:55.638422 19716 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.222.20:40777
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskgH5Pjm/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786277382073938-888-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskgH5Pjm/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786277382073938-888-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskgH5Pjm/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786277382073938-888-0/krb5kdc/kudu_127.0.222.3.keytab
--principal=kudu/127.0.222.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.222.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskgH5Pjm/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786277382073938-888-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.222.3
--webserver_port=0
--tserver_master_addrs=127.0.222.62:45597
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.222.3
--log_dir=/tmp/dist-test-taskgH5Pjm/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786277382073938-888-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 09 Aug 2026 11:06:41 UTC on ccycloud.quasar-bwoogy.root.comops.site
build id 4575991
ASAN enabled
I20260809 12:14:55.639837 19716 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260809 12:14:55.641201 19716 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260809 12:14:55.657840 19725 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 12:14:55.657977 19721 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 12:14:55.657954 19722 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260809 12:14:55.659180 19716 server_base.cc:1034] running on GCE node
I20260809 12:14:55.659786 19716 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260809 12:14:55.660739 19716 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260809 12:14:55.662004 19716 hybrid_clock.cc:648] HybridClock initialized: now 1786277695661959 us; error 68 us; skew 500 ppm
I20260809 12:14:55.662353 19716 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 09 12:14:55 dist-test-slave-2w3w krb5kdc[19310](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786277695, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.222.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260809 12:14:55.665555 19716 init.cc:377] Logged in from keytab as kudu/127.0.222.3@KRBTEST.COM (short username kudu)
I20260809 12:14:55.667076 19716 webserver.cc:460] Webserver started at http://127.0.222.3:37775/ using document root <none> and password file <none>
I20260809 12:14:55.667762 19716 fs_manager.cc:359] Metadata directory not provided
I20260809 12:14:55.667920 19716 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260809 12:14:55.668243 19716 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260809 12:14:55.671198 19716 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskgH5Pjm/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786277382073938-888-0/minicluster-data/ts-2/data/instance:
uuid: "6a7d552ab0be491a8f544bb5c7f003d1"
format_stamp: "Formatted at 2026-08-09 12:14:55 on dist-test-slave-2w3w"
server_key: "f33d1133fef221cb47a08e7ff744dd31"
server_key_iv: "39ef64040817061d5f0496059677c425"
server_key_version: "encryptionkey@0"
I20260809 12:14:55.672010 19716 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskgH5Pjm/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786277382073938-888-0/minicluster-data/ts-2/wal/instance:
uuid: "6a7d552ab0be491a8f544bb5c7f003d1"
format_stamp: "Formatted at 2026-08-09 12:14:55 on dist-test-slave-2w3w"
server_key: "f33d1133fef221cb47a08e7ff744dd31"
server_key_iv: "39ef64040817061d5f0496059677c425"
server_key_version: "encryptionkey@0"
I20260809 12:14:55.677129 19716 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.000s
I20260809 12:14:55.680591 19733 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260809 12:14:55.681883 19716 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.000s
I20260809 12:14:55.682062 19716 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskgH5Pjm/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786277382073938-888-0/minicluster-data/ts-2/data,/tmp/dist-test-taskgH5Pjm/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786277382073938-888-0/minicluster-data/ts-2/wal
uuid: "6a7d552ab0be491a8f544bb5c7f003d1"
format_stamp: "Formatted at 2026-08-09 12:14:55 on dist-test-slave-2w3w"
server_key: "f33d1133fef221cb47a08e7ff744dd31"
server_key_iv: "39ef64040817061d5f0496059677c425"
server_key_version: "encryptionkey@0"
I20260809 12:14:55.682263 19716 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskgH5Pjm/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786277382073938-888-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskgH5Pjm/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786277382073938-888-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskgH5Pjm/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786277382073938-888-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260809 12:14:55.685521 19561 heartbeater.cc:503] Master 127.0.222.62:45597 was elected leader, sending a full tablet report...
I20260809 12:14:55.700675 19716 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260809 12:14:55.704702 19716 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260809 12:14:55.705005 19716 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260809 12:14:55.705982 19716 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260809 12:14:55.707381 19716 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260809 12:14:55.707484 19716 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260809 12:14:55.707657 19716 ts_tablet_manager.cc:610] Registered 0 tablets
I20260809 12:14:55.707729 19716 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260809 12:14:55.742216 19716 rpc_server.cc:307] RPC server started. Bound to: 127.0.222.3:41815
I20260809 12:14:55.742259 19857 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.222.3:41815 every 8 connection(s)
I20260809 12:14:55.743602 19716 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskgH5Pjm/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786277382073938-888-0/minicluster-data/ts-2/data/info.pb
I20260809 12:14:55.749962 888 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskgH5Pjm/build/asan/bin/kudu as pid 19716
I20260809 12:14:55.750160 888 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskgH5Pjm/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786277382073938-888-0/minicluster-data/ts-2/wal/instance
I20260809 12:14:55.750733 888 external_mini_cluster.cc:1428] Setting key d9173b19d4d80be16d8aa455dd6ef71b
Aug 09 12:14:55 dist-test-slave-2w3w krb5kdc[19310](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786277695, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.222.3@KRBTEST.COM for kudu/127.0.222.62@KRBTEST.COM
I20260809 12:14:55.763756 19860 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 12:14:55.746187 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.222.3:54609 (local address 127.0.222.62:45597)
0809 12:14:55.746680 (+ 493us) server_negotiation.cc:206] Beginning negotiation
0809 12:14:55.746685 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0809 12:14:55.747580 (+ 895us) server_negotiation.cc:407] Connection header received
0809 12:14:55.749454 (+ 1874us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 12:14:55.749460 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 12:14:55.749563 (+ 103us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 12:14:55.749687 (+ 124us) server_negotiation.cc:226] Negotiated authn=SASL
0809 12:14:55.751397 (+ 1710us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 12:14:55.752332 (+ 935us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 12:14:55.754203 (+ 1871us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 12:14:55.754641 (+ 438us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 12:14:55.754800 (+ 159us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 12:14:55.758119 (+ 3319us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 12:14:55.758124 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 12:14:55.758127 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0809 12:14:55.758160 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0809 12:14:55.760668 (+ 2508us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 12:14:55.761383 (+ 715us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 12:14:55.761390 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 12:14:55.761398 (+ 8us) server_negotiation.cc:967] calling sasl_server_step()
0809 12:14:55.761517 (+ 119us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 12:14:55.761961 (+ 444us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 12:14:55.761966 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 12:14:55.761969 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0809 12:14:55.762239 (+ 270us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 12:14:55.762412 (+ 173us) server_negotiation.cc:1035] Waiting for connection context
0809 12:14:55.763143 (+ 731us) server_negotiation.cc:299] Negotiation successful
0809 12:14:55.763379 (+ 236us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":385,"thread_start_us":263,"threads_started":1}
I20260809 12:14:55.764818 19861 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 12:14:55.746657 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.222.62:45597 (local address 127.0.222.3:54609)
0809 12:14:55.747414 (+ 757us) negotiation.cc:107] Waiting for socket to connect
0809 12:14:55.747453 (+ 39us) client_negotiation.cc:174] Beginning negotiation
0809 12:14:55.749182 (+ 1729us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 12:14:55.749928 (+ 746us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 12:14:55.749936 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 12:14:55.750567 (+ 631us) client_negotiation.cc:189] Negotiated authn=SASL
0809 12:14:55.751220 (+ 653us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 12:14:55.751239 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 12:14:55.752501 (+ 1262us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 12:14:55.752507 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 12:14:55.754063 (+ 1556us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 12:14:55.754075 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 12:14:55.754913 (+ 838us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 12:14:55.754921 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 12:14:55.755167 (+ 246us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 12:14:55.755760 (+ 593us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0809 12:14:55.755787 (+ 27us) client_negotiation.cc:656] calling sasl_client_start()
0809 12:14:55.757902 (+ 2115us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0809 12:14:55.760814 (+ 2912us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0809 12:14:55.760823 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0809 12:14:55.760839 (+ 16us) client_negotiation.cc:760] calling sasl_client_step()
0809 12:14:55.761245 (+ 406us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0809 12:14:55.761630 (+ 385us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0809 12:14:55.761636 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0809 12:14:55.761639 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0809 12:14:55.761833 (+ 194us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0809 12:14:55.762383 (+ 550us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0809 12:14:55.762392 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0809 12:14:55.762717 (+ 325us) client_negotiation.cc:769] Sending connection context
0809 12:14:55.763013 (+ 296us) client_negotiation.cc:240] Negotiation successful
0809 12:14:55.763526 (+ 513us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":529,"thread_start_us":340,"threads_started":1}
I20260809 12:14:55.766273 19858 heartbeater.cc:347] Connected to a master server at 127.0.222.62:45597
I20260809 12:14:55.766588 19858 heartbeater.cc:464] Registering TS with master...
I20260809 12:14:55.767349 19858 heartbeater.cc:511] Master 127.0.222.62:45597 requested a full tablet report, sending...
I20260809 12:14:55.768801 19359 ts_manager.cc:194] Registered new tserver with Master: 6a7d552ab0be491a8f544bb5c7f003d1 (127.0.222.3:41815)
I20260809 12:14:55.769652 19359 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.222.3@KRBTEST.COM'} at 127.0.222.3:54609
I20260809 12:14:55.778421 888 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260809 12:14:55.795639 19860 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 12:14:55.783081 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39168 (local address 127.0.222.62:45597)
0809 12:14:55.783241 (+ 160us) server_negotiation.cc:206] Beginning negotiation
0809 12:14:55.783245 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0809 12:14:55.783726 (+ 481us) server_negotiation.cc:407] Connection header received
0809 12:14:55.783981 (+ 255us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 12:14:55.783986 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 12:14:55.784061 (+ 75us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 12:14:55.784164 (+ 103us) server_negotiation.cc:226] Negotiated authn=SASL
0809 12:14:55.785682 (+ 1518us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 12:14:55.786427 (+ 745us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 12:14:55.787892 (+ 1465us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 12:14:55.788280 (+ 388us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 12:14:55.788384 (+ 104us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 12:14:55.790729 (+ 2345us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 12:14:55.790734 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 12:14:55.790737 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0809 12:14:55.790767 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0809 12:14:55.793212 (+ 2445us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 12:14:55.793820 (+ 608us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 12:14:55.793823 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 12:14:55.793826 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0809 12:14:55.793940 (+ 114us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 12:14:55.794253 (+ 313us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 12:14:55.794256 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 12:14:55.794258 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0809 12:14:55.794574 (+ 316us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 12:14:55.794765 (+ 191us) server_negotiation.cc:1035] Waiting for connection context
0809 12:14:55.795213 (+ 448us) server_negotiation.cc:299] Negotiation successful
0809 12:14:55.795409 (+ 196us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":54}
I20260809 12:14:55.800004 19359 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:39168:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260809 12:14:55.806795 19359 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260809 12:14:55.832778 19872 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 12:14:55.819218 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33466 (local address 127.0.222.3:41815)
0809 12:14:55.824849 (+ 5631us) server_negotiation.cc:206] Beginning negotiation
0809 12:14:55.824857 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0809 12:14:55.824890 (+ 33us) server_negotiation.cc:407] Connection header received
0809 12:14:55.824999 (+ 109us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 12:14:55.825006 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 12:14:55.825211 (+ 205us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 12:14:55.825447 (+ 236us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0809 12:14:55.826656 (+ 1209us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 12:14:55.828239 (+ 1583us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 12:14:55.830468 (+ 2229us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 12:14:55.831629 (+ 1161us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 12:14:55.831783 (+ 154us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 12:14:55.832031 (+ 248us) server_negotiation.cc:1035] Waiting for connection context
0809 12:14:55.832289 (+ 258us) server_negotiation.cc:299] Negotiation successful
0809 12:14:55.832452 (+ 163us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":5471,"thread_start_us":5328,"threads_started":1}
I20260809 12:14:55.832871 19874 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 12:14:55.819176 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.222.3:41815 (local address 127.0.0.1:33466)
0809 12:14:55.821749 (+ 2573us) negotiation.cc:107] Waiting for socket to connect
0809 12:14:55.821821 (+ 72us) client_negotiation.cc:174] Beginning negotiation
0809 12:14:55.823442 (+ 1621us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 12:14:55.825423 (+ 1981us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 12:14:55.825428 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 12:14:55.825459 (+ 31us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0809 12:14:55.826395 (+ 936us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 12:14:55.826412 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 12:14:55.828374 (+ 1962us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 12:14:55.828381 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 12:14:55.830290 (+ 1909us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 12:14:55.830303 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 12:14:55.831832 (+ 1529us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 12:14:55.831839 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 12:14:55.832031 (+ 192us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 12:14:55.832083 (+ 52us) client_negotiation.cc:769] Sending connection context
0809 12:14:55.832229 (+ 146us) client_negotiation.cc:240] Negotiation successful
0809 12:14:55.832436 (+ 207us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":2252,"mutex_wait_us":424,"thread_start_us":4583,"threads_started":1}
I20260809 12:14:55.833997 19877 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 12:14:55.821162 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39424 (local address 127.0.222.1:37951)
0809 12:14:55.824542 (+ 3380us) server_negotiation.cc:206] Beginning negotiation
0809 12:14:55.824551 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0809 12:14:55.825093 (+ 542us) server_negotiation.cc:407] Connection header received
0809 12:14:55.825324 (+ 231us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 12:14:55.825333 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 12:14:55.825567 (+ 234us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 12:14:55.825747 (+ 180us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0809 12:14:55.826719 (+ 972us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 12:14:55.827897 (+ 1178us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 12:14:55.831106 (+ 3209us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 12:14:55.832537 (+ 1431us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 12:14:55.832661 (+ 124us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 12:14:55.832899 (+ 238us) server_negotiation.cc:1035] Waiting for connection context
0809 12:14:55.833466 (+ 567us) server_negotiation.cc:299] Negotiation successful
0809 12:14:55.833634 (+ 168us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":3170,"thread_start_us":2986,"threads_started":1}
I20260809 12:14:55.834228 19873 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 12:14:55.819184 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.222.2:40181 (local address 127.0.0.1:58190)
0809 12:14:55.824152 (+ 4968us) negotiation.cc:107] Waiting for socket to connect
0809 12:14:55.824174 (+ 22us) client_negotiation.cc:174] Beginning negotiation
0809 12:14:55.824371 (+ 197us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 12:14:55.825258 (+ 887us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 12:14:55.825266 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 12:14:55.825294 (+ 28us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0809 12:14:55.825961 (+ 667us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 12:14:55.825979 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 12:14:55.828098 (+ 2119us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 12:14:55.828101 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 12:14:55.831528 (+ 3427us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 12:14:55.831546 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 12:14:55.833368 (+ 1822us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 12:14:55.833377 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 12:14:55.833582 (+ 205us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 12:14:55.833612 (+ 30us) client_negotiation.cc:769] Sending connection context
0809 12:14:55.833701 (+ 89us) client_negotiation.cc:240] Negotiation successful
0809 12:14:55.833831 (+ 130us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":4813,"mutex_wait_us":449,"thread_start_us":1881,"threads_started":1}
I20260809 12:14:55.834725 19876 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 12:14:55.820808 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.222.1:37951 (local address 127.0.0.1:39424)
0809 12:14:55.824965 (+ 4157us) negotiation.cc:107] Waiting for socket to connect
0809 12:14:55.824990 (+ 25us) client_negotiation.cc:174] Beginning negotiation
0809 12:14:55.825138 (+ 148us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 12:14:55.825916 (+ 778us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 12:14:55.825922 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 12:14:55.825948 (+ 26us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0809 12:14:55.826520 (+ 572us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 12:14:55.826532 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 12:14:55.828088 (+ 1556us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 12:14:55.828093 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 12:14:55.830884 (+ 2791us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 12:14:55.830898 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 12:14:55.833049 (+ 2151us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 12:14:55.833055 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 12:14:55.833192 (+ 137us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 12:14:55.833250 (+ 58us) client_negotiation.cc:769] Sending connection context
0809 12:14:55.834260 (+ 1010us) client_negotiation.cc:240] Negotiation successful
0809 12:14:55.834390 (+ 130us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":4024,"mutex_wait_us":189,"thread_start_us":3931,"threads_started":1}
I20260809 12:14:55.834951 19875 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 12:14:55.819569 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58190 (local address 127.0.222.2:40181)
0809 12:14:55.821095 (+ 1526us) server_negotiation.cc:206] Beginning negotiation
0809 12:14:55.821106 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0809 12:14:55.824271 (+ 3165us) server_negotiation.cc:407] Connection header received
0809 12:14:55.824582 (+ 311us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 12:14:55.824591 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 12:14:55.824861 (+ 270us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 12:14:55.825031 (+ 170us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0809 12:14:55.826184 (+ 1153us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 12:14:55.827969 (+ 1785us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 12:14:55.831700 (+ 3731us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 12:14:55.833143 (+ 1443us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 12:14:55.833281 (+ 138us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 12:14:55.833493 (+ 212us) server_negotiation.cc:1035] Waiting for connection context
0809 12:14:55.834441 (+ 948us) server_negotiation.cc:299] Negotiation successful
0809 12:14:55.834614 (+ 173us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1297,"thread_start_us":1795,"threads_started":1}
I20260809 12:14:55.837895 19492 tablet_service.cc:1467] Processing CreateTablet for tablet 3f43c98c79c3480295160d01baf8d5dc (DEFAULT_TABLE table=test-table [id=214415e2ad0a4acfacf747eb44be5143]), partition=RANGE (key) PARTITION UNBOUNDED
I20260809 12:14:55.837937 19783 tablet_service.cc:1467] Processing CreateTablet for tablet 3f43c98c79c3480295160d01baf8d5dc (DEFAULT_TABLE table=test-table [id=214415e2ad0a4acfacf747eb44be5143]), partition=RANGE (key) PARTITION UNBOUNDED
I20260809 12:14:55.838369 19640 tablet_service.cc:1467] Processing CreateTablet for tablet 3f43c98c79c3480295160d01baf8d5dc (DEFAULT_TABLE table=test-table [id=214415e2ad0a4acfacf747eb44be5143]), partition=RANGE (key) PARTITION UNBOUNDED
I20260809 12:14:55.839519 19492 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 3f43c98c79c3480295160d01baf8d5dc. 1 dirs total, 0 dirs full, 0 dirs failed
I20260809 12:14:55.839635 19783 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 3f43c98c79c3480295160d01baf8d5dc. 1 dirs total, 0 dirs full, 0 dirs failed
I20260809 12:14:55.839860 19640 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 3f43c98c79c3480295160d01baf8d5dc. 1 dirs total, 0 dirs full, 0 dirs failed
I20260809 12:14:55.853108 19878 tablet_bootstrap.cc:492] T 3f43c98c79c3480295160d01baf8d5dc P b88447d41a7f4ca4b3f742239780061b: Bootstrap starting.
I20260809 12:14:55.854216 19879 tablet_bootstrap.cc:492] T 3f43c98c79c3480295160d01baf8d5dc P 6a7d552ab0be491a8f544bb5c7f003d1: Bootstrap starting.
I20260809 12:14:55.855929 19880 tablet_bootstrap.cc:492] T 3f43c98c79c3480295160d01baf8d5dc P 0053eca7b1ac45d8b5672e15b7724512: Bootstrap starting.
I20260809 12:14:55.856930 19878 tablet_bootstrap.cc:654] T 3f43c98c79c3480295160d01baf8d5dc P b88447d41a7f4ca4b3f742239780061b: Neither blocks nor log segments found. Creating new log.
I20260809 12:14:55.857872 19879 tablet_bootstrap.cc:654] T 3f43c98c79c3480295160d01baf8d5dc P 6a7d552ab0be491a8f544bb5c7f003d1: Neither blocks nor log segments found. Creating new log.
I20260809 12:14:55.858296 19878 log.cc:824] T 3f43c98c79c3480295160d01baf8d5dc P b88447d41a7f4ca4b3f742239780061b: Log is configured to *not* fsync() on all Append() calls
I20260809 12:14:55.858736 19880 tablet_bootstrap.cc:654] T 3f43c98c79c3480295160d01baf8d5dc P 0053eca7b1ac45d8b5672e15b7724512: Neither blocks nor log segments found. Creating new log.
I20260809 12:14:55.859153 19879 log.cc:824] T 3f43c98c79c3480295160d01baf8d5dc P 6a7d552ab0be491a8f544bb5c7f003d1: Log is configured to *not* fsync() on all Append() calls
I20260809 12:14:55.859982 19880 log.cc:824] T 3f43c98c79c3480295160d01baf8d5dc P 0053eca7b1ac45d8b5672e15b7724512: Log is configured to *not* fsync() on all Append() calls
I20260809 12:14:55.861786 19878 tablet_bootstrap.cc:492] T 3f43c98c79c3480295160d01baf8d5dc P b88447d41a7f4ca4b3f742239780061b: No bootstrap required, opened a new log
I20260809 12:14:55.862062 19878 ts_tablet_manager.cc:1397] T 3f43c98c79c3480295160d01baf8d5dc P b88447d41a7f4ca4b3f742239780061b: Time spent bootstrapping tablet: real 0.010s user 0.004s sys 0.004s
I20260809 12:14:55.862072 19879 tablet_bootstrap.cc:492] T 3f43c98c79c3480295160d01baf8d5dc P 6a7d552ab0be491a8f544bb5c7f003d1: No bootstrap required, opened a new log
I20260809 12:14:55.862344 19879 ts_tablet_manager.cc:1397] T 3f43c98c79c3480295160d01baf8d5dc P 6a7d552ab0be491a8f544bb5c7f003d1: Time spent bootstrapping tablet: real 0.008s user 0.003s sys 0.003s
I20260809 12:14:55.864210 19880 tablet_bootstrap.cc:492] T 3f43c98c79c3480295160d01baf8d5dc P 0053eca7b1ac45d8b5672e15b7724512: No bootstrap required, opened a new log
I20260809 12:14:55.864459 19880 ts_tablet_manager.cc:1397] T 3f43c98c79c3480295160d01baf8d5dc P 0053eca7b1ac45d8b5672e15b7724512: Time spent bootstrapping tablet: real 0.009s user 0.007s sys 0.000s
I20260809 12:14:55.869535 19879 raft_consensus.cc:348] T 3f43c98c79c3480295160d01baf8d5dc P 6a7d552ab0be491a8f544bb5c7f003d1 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6a7d552ab0be491a8f544bb5c7f003d1" member_type: VOTER last_known_addr { host: "127.0.222.3" port: 41815 } } peers { permanent_uuid: "0053eca7b1ac45d8b5672e15b7724512" member_type: VOTER last_known_addr { host: "127.0.222.2" port: 40181 } } peers { permanent_uuid: "b88447d41a7f4ca4b3f742239780061b" member_type: VOTER last_known_addr { host: "127.0.222.1" port: 37951 } }
I20260809 12:14:55.869925 19879 raft_consensus.cc:374] T 3f43c98c79c3480295160d01baf8d5dc P 6a7d552ab0be491a8f544bb5c7f003d1 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260809 12:14:55.870054 19879 raft_consensus.cc:729] T 3f43c98c79c3480295160d01baf8d5dc P 6a7d552ab0be491a8f544bb5c7f003d1 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6a7d552ab0be491a8f544bb5c7f003d1, State: Initialized, Role: FOLLOWER
I20260809 12:14:55.870663 19879 consensus_queue.cc:260] T 3f43c98c79c3480295160d01baf8d5dc P 6a7d552ab0be491a8f544bb5c7f003d1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6a7d552ab0be491a8f544bb5c7f003d1" member_type: VOTER last_known_addr { host: "127.0.222.3" port: 41815 } } peers { permanent_uuid: "0053eca7b1ac45d8b5672e15b7724512" member_type: VOTER last_known_addr { host: "127.0.222.2" port: 40181 } } peers { permanent_uuid: "b88447d41a7f4ca4b3f742239780061b" member_type: VOTER last_known_addr { host: "127.0.222.1" port: 37951 } }
I20260809 12:14:55.872808 19878 raft_consensus.cc:348] T 3f43c98c79c3480295160d01baf8d5dc P b88447d41a7f4ca4b3f742239780061b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6a7d552ab0be491a8f544bb5c7f003d1" member_type: VOTER last_known_addr { host: "127.0.222.3" port: 41815 } } peers { permanent_uuid: "0053eca7b1ac45d8b5672e15b7724512" member_type: VOTER last_known_addr { host: "127.0.222.2" port: 40181 } } peers { permanent_uuid: "b88447d41a7f4ca4b3f742239780061b" member_type: VOTER last_known_addr { host: "127.0.222.1" port: 37951 } }
I20260809 12:14:55.873283 19878 raft_consensus.cc:374] T 3f43c98c79c3480295160d01baf8d5dc P b88447d41a7f4ca4b3f742239780061b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260809 12:14:55.873377 19858 heartbeater.cc:503] Master 127.0.222.62:45597 was elected leader, sending a full tablet report...
I20260809 12:14:55.873417 19878 raft_consensus.cc:729] T 3f43c98c79c3480295160d01baf8d5dc P b88447d41a7f4ca4b3f742239780061b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b88447d41a7f4ca4b3f742239780061b, State: Initialized, Role: FOLLOWER
I20260809 12:14:55.874089 19879 ts_tablet_manager.cc:1428] T 3f43c98c79c3480295160d01baf8d5dc P 6a7d552ab0be491a8f544bb5c7f003d1: Time spent starting tablet: real 0.012s user 0.012s sys 0.001s
I20260809 12:14:55.874073 19878 consensus_queue.cc:260] T 3f43c98c79c3480295160d01baf8d5dc P b88447d41a7f4ca4b3f742239780061b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6a7d552ab0be491a8f544bb5c7f003d1" member_type: VOTER last_known_addr { host: "127.0.222.3" port: 41815 } } peers { permanent_uuid: "0053eca7b1ac45d8b5672e15b7724512" member_type: VOTER last_known_addr { host: "127.0.222.2" port: 40181 } } peers { permanent_uuid: "b88447d41a7f4ca4b3f742239780061b" member_type: VOTER last_known_addr { host: "127.0.222.1" port: 37951 } }
I20260809 12:14:55.874835 19880 raft_consensus.cc:348] T 3f43c98c79c3480295160d01baf8d5dc P 0053eca7b1ac45d8b5672e15b7724512 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6a7d552ab0be491a8f544bb5c7f003d1" member_type: VOTER last_known_addr { host: "127.0.222.3" port: 41815 } } peers { permanent_uuid: "0053eca7b1ac45d8b5672e15b7724512" member_type: VOTER last_known_addr { host: "127.0.222.2" port: 40181 } } peers { permanent_uuid: "b88447d41a7f4ca4b3f742239780061b" member_type: VOTER last_known_addr { host: "127.0.222.1" port: 37951 } }
I20260809 12:14:55.875319 19880 raft_consensus.cc:374] T 3f43c98c79c3480295160d01baf8d5dc P 0053eca7b1ac45d8b5672e15b7724512 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260809 12:14:55.875420 19880 raft_consensus.cc:729] T 3f43c98c79c3480295160d01baf8d5dc P 0053eca7b1ac45d8b5672e15b7724512 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0053eca7b1ac45d8b5672e15b7724512, State: Initialized, Role: FOLLOWER
I20260809 12:14:55.876155 19878 ts_tablet_manager.cc:1428] T 3f43c98c79c3480295160d01baf8d5dc P b88447d41a7f4ca4b3f742239780061b: Time spent starting tablet: real 0.014s user 0.008s sys 0.004s
I20260809 12:14:55.876017 19880 consensus_queue.cc:260] T 3f43c98c79c3480295160d01baf8d5dc P 0053eca7b1ac45d8b5672e15b7724512 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6a7d552ab0be491a8f544bb5c7f003d1" member_type: VOTER last_known_addr { host: "127.0.222.3" port: 41815 } } peers { permanent_uuid: "0053eca7b1ac45d8b5672e15b7724512" member_type: VOTER last_known_addr { host: "127.0.222.2" port: 40181 } } peers { permanent_uuid: "b88447d41a7f4ca4b3f742239780061b" member_type: VOTER last_known_addr { host: "127.0.222.1" port: 37951 } }
I20260809 12:14:55.877157 19709 heartbeater.cc:503] Master 127.0.222.62:45597 was elected leader, sending a full tablet report...
I20260809 12:14:55.877751 19880 ts_tablet_manager.cc:1428] T 3f43c98c79c3480295160d01baf8d5dc P 0053eca7b1ac45d8b5672e15b7724512: Time spent starting tablet: real 0.013s user 0.014s sys 0.000s
W20260809 12:14:55.902694 19562 tablet.cc:2369] T 3f43c98c79c3480295160d01baf8d5dc P b88447d41a7f4ca4b3f742239780061b: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260809 12:14:55.946398 19710 tablet.cc:2369] T 3f43c98c79c3480295160d01baf8d5dc P 0053eca7b1ac45d8b5672e15b7724512: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260809 12:14:55.995482 19859 tablet.cc:2369] T 3f43c98c79c3480295160d01baf8d5dc P 6a7d552ab0be491a8f544bb5c7f003d1: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260809 12:14:56.273010 19884 raft_consensus.cc:482] T 3f43c98c79c3480295160d01baf8d5dc P 6a7d552ab0be491a8f544bb5c7f003d1 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260809 12:14:56.273315 19884 raft_consensus.cc:504] T 3f43c98c79c3480295160d01baf8d5dc P 6a7d552ab0be491a8f544bb5c7f003d1 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6a7d552ab0be491a8f544bb5c7f003d1" member_type: VOTER last_known_addr { host: "127.0.222.3" port: 41815 } } peers { permanent_uuid: "0053eca7b1ac45d8b5672e15b7724512" member_type: VOTER last_known_addr { host: "127.0.222.2" port: 40181 } } peers { permanent_uuid: "b88447d41a7f4ca4b3f742239780061b" member_type: VOTER last_known_addr { host: "127.0.222.1" port: 37951 } }
I20260809 12:14:56.275089 19884 leader_election.cc:288] T 3f43c98c79c3480295160d01baf8d5dc P 6a7d552ab0be491a8f544bb5c7f003d1 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 0053eca7b1ac45d8b5672e15b7724512 (127.0.222.2:40181), b88447d41a7f4ca4b3f742239780061b (127.0.222.1:37951)
I20260809 12:14:56.275393 19885 raft_consensus.cc:482] T 3f43c98c79c3480295160d01baf8d5dc P b88447d41a7f4ca4b3f742239780061b [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260809 12:14:56.276885 19885 raft_consensus.cc:504] T 3f43c98c79c3480295160d01baf8d5dc P b88447d41a7f4ca4b3f742239780061b [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6a7d552ab0be491a8f544bb5c7f003d1" member_type: VOTER last_known_addr { host: "127.0.222.3" port: 41815 } } peers { permanent_uuid: "0053eca7b1ac45d8b5672e15b7724512" member_type: VOTER last_known_addr { host: "127.0.222.2" port: 40181 } } peers { permanent_uuid: "b88447d41a7f4ca4b3f742239780061b" member_type: VOTER last_known_addr { host: "127.0.222.1" port: 37951 } }
I20260809 12:14:56.283254 19875 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 12:14:56.275816 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.222.3:55035 (local address 127.0.222.2:40181)
0809 12:14:56.276026 (+ 210us) server_negotiation.cc:206] Beginning negotiation
0809 12:14:56.276033 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0809 12:14:56.276057 (+ 24us) server_negotiation.cc:407] Connection header received
0809 12:14:56.276144 (+ 87us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 12:14:56.276150 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 12:14:56.276236 (+ 86us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 12:14:56.276362 (+ 126us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0809 12:14:56.277027 (+ 665us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 12:14:56.278121 (+ 1094us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 12:14:56.280765 (+ 2644us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 12:14:56.282048 (+ 1283us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 12:14:56.282155 (+ 107us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 12:14:56.282256 (+ 101us) server_negotiation.cc:1035] Waiting for connection context
0809 12:14:56.282811 (+ 555us) server_negotiation.cc:299] Negotiation successful
0809 12:14:56.282976 (+ 165us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":71}
I20260809 12:14:56.284298 19887 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 12:14:56.275301 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.222.2:40181 (local address 127.0.222.3:55035)
0809 12:14:56.275834 (+ 533us) negotiation.cc:107] Waiting for socket to connect
0809 12:14:56.275859 (+ 25us) client_negotiation.cc:174] Beginning negotiation
0809 12:14:56.276000 (+ 141us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 12:14:56.276398 (+ 398us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 12:14:56.276404 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 12:14:56.276434 (+ 30us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0809 12:14:56.276787 (+ 353us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 12:14:56.276812 (+ 25us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 12:14:56.278268 (+ 1456us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 12:14:56.278273 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 12:14:56.280543 (+ 2270us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 12:14:56.280564 (+ 21us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 12:14:56.282393 (+ 1829us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 12:14:56.282401 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 12:14:56.282615 (+ 214us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 12:14:56.282648 (+ 33us) client_negotiation.cc:769] Sending connection context
0809 12:14:56.283791 (+ 1143us) client_negotiation.cc:240] Negotiation successful
0809 12:14:56.283941 (+ 150us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":378,"thread_start_us":1235,"threads_started":1}
I20260809 12:14:56.284561 19877 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 12:14:56.276362 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.222.3:51807 (local address 127.0.222.1:37951)
0809 12:14:56.276558 (+ 196us) server_negotiation.cc:206] Beginning negotiation
0809 12:14:56.276565 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0809 12:14:56.277281 (+ 716us) server_negotiation.cc:407] Connection header received
0809 12:14:56.277534 (+ 253us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 12:14:56.277539 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 12:14:56.277614 (+ 75us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 12:14:56.277715 (+ 101us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0809 12:14:56.278230 (+ 515us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 12:14:56.279412 (+ 1182us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 12:14:56.282041 (+ 2629us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 12:14:56.283359 (+ 1318us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 12:14:56.283490 (+ 131us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 12:14:56.283581 (+ 91us) server_negotiation.cc:1035] Waiting for connection context
0809 12:14:56.284169 (+ 588us) server_negotiation.cc:299] Negotiation successful
0809 12:14:56.284307 (+ 138us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":63}
I20260809 12:14:56.285240 19888 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 12:14:56.275912 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.222.1:37951 (local address 127.0.222.3:51807)
0809 12:14:56.277167 (+ 1255us) negotiation.cc:107] Waiting for socket to connect
0809 12:14:56.277181 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0809 12:14:56.277388 (+ 207us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 12:14:56.277730 (+ 342us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 12:14:56.277734 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 12:14:56.277756 (+ 22us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0809 12:14:56.278075 (+ 319us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 12:14:56.278083 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 12:14:56.279582 (+ 1499us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 12:14:56.279587 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 12:14:56.281832 (+ 2245us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 12:14:56.281850 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 12:14:56.283770 (+ 1920us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 12:14:56.283778 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 12:14:56.283964 (+ 186us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 12:14:56.284003 (+ 39us) client_negotiation.cc:769] Sending connection context
0809 12:14:56.284778 (+ 775us) client_negotiation.cc:240] Negotiation successful
0809 12:14:56.284923 (+ 145us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1149,"thread_start_us":987,"threads_started":1}
I20260809 12:14:56.285631 19661 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3f43c98c79c3480295160d01baf8d5dc" candidate_uuid: "6a7d552ab0be491a8f544bb5c7f003d1" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0053eca7b1ac45d8b5672e15b7724512" is_pre_election: true
I20260809 12:14:56.286258 19661 raft_consensus.cc:2454] T 3f43c98c79c3480295160d01baf8d5dc P 0053eca7b1ac45d8b5672e15b7724512 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6a7d552ab0be491a8f544bb5c7f003d1 in term 0.
I20260809 12:14:56.286346 19512 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3f43c98c79c3480295160d01baf8d5dc" candidate_uuid: "6a7d552ab0be491a8f544bb5c7f003d1" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b88447d41a7f4ca4b3f742239780061b" is_pre_election: true
I20260809 12:14:56.286855 19512 raft_consensus.cc:2454] T 3f43c98c79c3480295160d01baf8d5dc P b88447d41a7f4ca4b3f742239780061b [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6a7d552ab0be491a8f544bb5c7f003d1 in term 0.
I20260809 12:14:56.287137 19735 leader_election.cc:302] T 3f43c98c79c3480295160d01baf8d5dc P 6a7d552ab0be491a8f544bb5c7f003d1 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0053eca7b1ac45d8b5672e15b7724512, 6a7d552ab0be491a8f544bb5c7f003d1; no voters:
I20260809 12:14:56.287793 19884 raft_consensus.cc:2781] T 3f43c98c79c3480295160d01baf8d5dc P 6a7d552ab0be491a8f544bb5c7f003d1 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260809 12:14:56.287959 19884 raft_consensus.cc:482] T 3f43c98c79c3480295160d01baf8d5dc P 6a7d552ab0be491a8f544bb5c7f003d1 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260809 12:14:56.288070 19884 raft_consensus.cc:3037] T 3f43c98c79c3480295160d01baf8d5dc P 6a7d552ab0be491a8f544bb5c7f003d1 [term 0 FOLLOWER]: Advancing to term 1
I20260809 12:14:56.289330 19885 leader_election.cc:288] T 3f43c98c79c3480295160d01baf8d5dc P b88447d41a7f4ca4b3f742239780061b [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 6a7d552ab0be491a8f544bb5c7f003d1 (127.0.222.3:41815), 0053eca7b1ac45d8b5672e15b7724512 (127.0.222.2:40181)
I20260809 12:14:56.291661 19884 raft_consensus.cc:504] T 3f43c98c79c3480295160d01baf8d5dc P 6a7d552ab0be491a8f544bb5c7f003d1 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6a7d552ab0be491a8f544bb5c7f003d1" member_type: VOTER last_known_addr { host: "127.0.222.3" port: 41815 } } peers { permanent_uuid: "0053eca7b1ac45d8b5672e15b7724512" member_type: VOTER last_known_addr { host: "127.0.222.2" port: 40181 } } peers { permanent_uuid: "b88447d41a7f4ca4b3f742239780061b" member_type: VOTER last_known_addr { host: "127.0.222.1" port: 37951 } }
I20260809 12:14:56.292979 19661 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3f43c98c79c3480295160d01baf8d5dc" candidate_uuid: "6a7d552ab0be491a8f544bb5c7f003d1" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0053eca7b1ac45d8b5672e15b7724512"
I20260809 12:14:56.292985 19512 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3f43c98c79c3480295160d01baf8d5dc" candidate_uuid: "6a7d552ab0be491a8f544bb5c7f003d1" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b88447d41a7f4ca4b3f742239780061b"
I20260809 12:14:56.293208 19512 raft_consensus.cc:3037] T 3f43c98c79c3480295160d01baf8d5dc P b88447d41a7f4ca4b3f742239780061b [term 0 FOLLOWER]: Advancing to term 1
I20260809 12:14:56.293198 19661 raft_consensus.cc:3037] T 3f43c98c79c3480295160d01baf8d5dc P 0053eca7b1ac45d8b5672e15b7724512 [term 0 FOLLOWER]: Advancing to term 1
I20260809 12:14:56.296711 19661 raft_consensus.cc:2454] T 3f43c98c79c3480295160d01baf8d5dc P 0053eca7b1ac45d8b5672e15b7724512 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6a7d552ab0be491a8f544bb5c7f003d1 in term 1.
I20260809 12:14:56.296710 19512 raft_consensus.cc:2454] T 3f43c98c79c3480295160d01baf8d5dc P b88447d41a7f4ca4b3f742239780061b [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6a7d552ab0be491a8f544bb5c7f003d1 in term 1.
I20260809 12:14:56.297153 19735 leader_election.cc:302] T 3f43c98c79c3480295160d01baf8d5dc P 6a7d552ab0be491a8f544bb5c7f003d1 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0053eca7b1ac45d8b5672e15b7724512, 6a7d552ab0be491a8f544bb5c7f003d1; no voters:
I20260809 12:14:56.297338 19875 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 12:14:56.279536 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.222.1:41007 (local address 127.0.222.2:40181)
0809 12:14:56.283755 (+ 4219us) server_negotiation.cc:206] Beginning negotiation
0809 12:14:56.283760 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0809 12:14:56.289114 (+ 5354us) server_negotiation.cc:407] Connection header received
0809 12:14:56.289258 (+ 144us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 12:14:56.289266 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 12:14:56.289397 (+ 131us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 12:14:56.289510 (+ 113us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0809 12:14:56.290437 (+ 927us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 12:14:56.291520 (+ 1083us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 12:14:56.294356 (+ 2836us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 12:14:56.295737 (+ 1381us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 12:14:56.295844 (+ 107us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 12:14:56.296228 (+ 384us) server_negotiation.cc:1035] Waiting for connection context
0809 12:14:56.296902 (+ 674us) server_negotiation.cc:299] Negotiation successful
0809 12:14:56.297049 (+ 147us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":4111,"thread_start_us":8801,"threads_started":1}
I20260809 12:14:56.297519 19884 leader_election.cc:288] T 3f43c98c79c3480295160d01baf8d5dc P 6a7d552ab0be491a8f544bb5c7f003d1 [CANDIDATE]: Term 1 election: Requested vote from peers 0053eca7b1ac45d8b5672e15b7724512 (127.0.222.2:40181), b88447d41a7f4ca4b3f742239780061b (127.0.222.1:37951)
I20260809 12:14:56.297618 19872 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 12:14:56.279476 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.222.1:51479 (local address 127.0.222.3:41815)
0809 12:14:56.279668 (+ 192us) server_negotiation.cc:206] Beginning negotiation
0809 12:14:56.279673 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0809 12:14:56.290162 (+ 10489us) server_negotiation.cc:407] Connection header received
0809 12:14:56.290350 (+ 188us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 12:14:56.290358 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 12:14:56.290434 (+ 76us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 12:14:56.290544 (+ 110us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0809 12:14:56.291078 (+ 534us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 12:14:56.291946 (+ 868us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 12:14:56.294693 (+ 2747us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 12:14:56.295972 (+ 1279us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 12:14:56.296081 (+ 109us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 12:14:56.296175 (+ 94us) server_negotiation.cc:1035] Waiting for connection context
0809 12:14:56.297214 (+ 1039us) server_negotiation.cc:299] Negotiation successful
0809 12:14:56.297354 (+ 140us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":57}
I20260809 12:14:56.297919 19884 raft_consensus.cc:2781] T 3f43c98c79c3480295160d01baf8d5dc P 6a7d552ab0be491a8f544bb5c7f003d1 [term 1 FOLLOWER]: Leader election won for term 1
I20260809 12:14:56.298138 19884 raft_consensus.cc:686] T 3f43c98c79c3480295160d01baf8d5dc P 6a7d552ab0be491a8f544bb5c7f003d1 [term 1 LEADER]: Becoming Leader. State: Replica: 6a7d552ab0be491a8f544bb5c7f003d1, State: Running, Role: LEADER
I20260809 12:14:56.298367 19889 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 12:14:56.279083 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.222.3:41815 (local address 127.0.222.1:51479)
0809 12:14:56.290048 (+ 10965us) negotiation.cc:107] Waiting for socket to connect
0809 12:14:56.290066 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0809 12:14:56.290210 (+ 144us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 12:14:56.290558 (+ 348us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 12:14:56.290563 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 12:14:56.290589 (+ 26us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0809 12:14:56.290902 (+ 313us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 12:14:56.290910 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 12:14:56.292097 (+ 1187us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 12:14:56.292103 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 12:14:56.294500 (+ 2397us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 12:14:56.294516 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 12:14:56.296810 (+ 2294us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 12:14:56.296816 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 12:14:56.297017 (+ 201us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 12:14:56.297051 (+ 34us) client_negotiation.cc:769] Sending connection context
0809 12:14:56.297741 (+ 690us) client_negotiation.cc:240] Negotiation successful
0809 12:14:56.297884 (+ 143us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":10780,"mutex_wait_us":34,"thread_start_us":9576,"threads_started":1}
I20260809 12:14:56.298429 19890 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 12:14:56.279047 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.222.2:40181 (local address 127.0.222.1:41007)
0809 12:14:56.288976 (+ 9929us) negotiation.cc:107] Waiting for socket to connect
0809 12:14:56.289007 (+ 31us) client_negotiation.cc:174] Beginning negotiation
0809 12:14:56.289144 (+ 137us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 12:14:56.289712 (+ 568us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 12:14:56.289717 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 12:14:56.289749 (+ 32us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0809 12:14:56.290237 (+ 488us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 12:14:56.290256 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 12:14:56.291668 (+ 1412us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 12:14:56.291675 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 12:14:56.294146 (+ 2471us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 12:14:56.294160 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 12:14:56.296443 (+ 2283us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 12:14:56.296452 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 12:14:56.296653 (+ 201us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 12:14:56.296707 (+ 54us) client_negotiation.cc:769] Sending connection context
0809 12:14:56.297570 (+ 863us) client_negotiation.cc:240] Negotiation successful
0809 12:14:56.297760 (+ 190us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":9793,"mutex_wait_us":47,"thread_start_us":10656,"threads_started":1}
I20260809 12:14:56.298888 19884 consensus_queue.cc:237] T 3f43c98c79c3480295160d01baf8d5dc P 6a7d552ab0be491a8f544bb5c7f003d1 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6a7d552ab0be491a8f544bb5c7f003d1" member_type: VOTER last_known_addr { host: "127.0.222.3" port: 41815 } } peers { permanent_uuid: "0053eca7b1ac45d8b5672e15b7724512" member_type: VOTER last_known_addr { host: "127.0.222.2" port: 40181 } } peers { permanent_uuid: "b88447d41a7f4ca4b3f742239780061b" member_type: VOTER last_known_addr { host: "127.0.222.1" port: 37951 } }
I20260809 12:14:56.300204 19804 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3f43c98c79c3480295160d01baf8d5dc" candidate_uuid: "b88447d41a7f4ca4b3f742239780061b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6a7d552ab0be491a8f544bb5c7f003d1" is_pre_election: true
I20260809 12:14:56.301561 19661 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3f43c98c79c3480295160d01baf8d5dc" candidate_uuid: "b88447d41a7f4ca4b3f742239780061b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0053eca7b1ac45d8b5672e15b7724512" is_pre_election: true
I20260809 12:14:56.301846 19661 raft_consensus.cc:2379] T 3f43c98c79c3480295160d01baf8d5dc P 0053eca7b1ac45d8b5672e15b7724512 [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate b88447d41a7f4ca4b3f742239780061b in current term 1: Already voted for candidate 6a7d552ab0be491a8f544bb5c7f003d1 in this term.
I20260809 12:14:56.302524 19441 leader_election.cc:302] T 3f43c98c79c3480295160d01baf8d5dc P b88447d41a7f4ca4b3f742239780061b [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: b88447d41a7f4ca4b3f742239780061b; no voters: 0053eca7b1ac45d8b5672e15b7724512, 6a7d552ab0be491a8f544bb5c7f003d1
I20260809 12:14:56.302997 19885 raft_consensus.cc:2726] T 3f43c98c79c3480295160d01baf8d5dc P b88447d41a7f4ca4b3f742239780061b [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260809 12:14:56.304929 19359 catalog_manager.cc:5495] T 3f43c98c79c3480295160d01baf8d5dc P 6a7d552ab0be491a8f544bb5c7f003d1 reported cstate change: term changed from 0 to 1, leader changed from <none> to 6a7d552ab0be491a8f544bb5c7f003d1 (127.0.222.3). New cstate: current_term: 1 leader_uuid: "6a7d552ab0be491a8f544bb5c7f003d1" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6a7d552ab0be491a8f544bb5c7f003d1" member_type: VOTER last_known_addr { host: "127.0.222.3" port: 41815 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "0053eca7b1ac45d8b5672e15b7724512" member_type: VOTER last_known_addr { host: "127.0.222.2" port: 40181 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "b88447d41a7f4ca4b3f742239780061b" member_type: VOTER last_known_addr { host: "127.0.222.1" port: 37951 } health_report { overall_health: UNKNOWN } } }
I20260809 12:14:56.398924 19872 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 12:14:56.393192 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33476 (local address 127.0.222.3:41815)
0809 12:14:56.393413 (+ 221us) server_negotiation.cc:206] Beginning negotiation
0809 12:14:56.393418 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0809 12:14:56.393856 (+ 438us) server_negotiation.cc:407] Connection header received
0809 12:14:56.394058 (+ 202us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 12:14:56.394062 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 12:14:56.394155 (+ 93us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 12:14:56.394292 (+ 137us) server_negotiation.cc:226] Negotiated authn=TOKEN
0809 12:14:56.394689 (+ 397us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 12:14:56.395441 (+ 752us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 12:14:56.396912 (+ 1471us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 12:14:56.397305 (+ 393us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 12:14:56.397409 (+ 104us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 12:14:56.397856 (+ 447us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0809 12:14:56.398402 (+ 546us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0809 12:14:56.398524 (+ 122us) server_negotiation.cc:1035] Waiting for connection context
0809 12:14:56.398605 (+ 81us) server_negotiation.cc:299] Negotiation successful
0809 12:14:56.398673 (+ 68us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":98}
I20260809 12:14:56.406719 19661 raft_consensus.cc:1261] T 3f43c98c79c3480295160d01baf8d5dc P 0053eca7b1ac45d8b5672e15b7724512 [term 1 FOLLOWER]: Refusing update from remote peer 6a7d552ab0be491a8f544bb5c7f003d1: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260809 12:14:56.406713 19512 raft_consensus.cc:1261] T 3f43c98c79c3480295160d01baf8d5dc P b88447d41a7f4ca4b3f742239780061b [term 1 FOLLOWER]: Refusing update from remote peer 6a7d552ab0be491a8f544bb5c7f003d1: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260809 12:14:56.407661 19892 consensus_queue.cc:1035] T 3f43c98c79c3480295160d01baf8d5dc P 6a7d552ab0be491a8f544bb5c7f003d1 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b88447d41a7f4ca4b3f742239780061b" member_type: VOTER last_known_addr { host: "127.0.222.1" port: 37951 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260809 12:14:56.407989 19884 consensus_queue.cc:1035] T 3f43c98c79c3480295160d01baf8d5dc P 6a7d552ab0be491a8f544bb5c7f003d1 [LEADER]: Connected to new peer: Peer: permanent_uuid: "0053eca7b1ac45d8b5672e15b7724512" member_type: VOTER last_known_addr { host: "127.0.222.2" port: 40181 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260809 12:14:56.419420 19896 mvcc.cc:205] Tried to move back new op lower bound from 7316593444472479744 to 7316593444045705216. Current Snapshot: MvccSnapshot[applied={T|T < 7316593444472479744}]
I20260809 12:14:56.434249 19359 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:39168:
table { table_name: "test-table" } modify_external_catalogs: true
I20260809 12:14:56.434571 19359 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:39168:
table { table_name: "test-table" } modify_external_catalogs: true
I20260809 12:14:56.438560 19359 catalog_manager.cc:5782] T 00000000000000000000000000000000 P c7606c7b6c614665a58919c9f580fc7e: Sending DeleteTablet for 3 replicas of tablet 3f43c98c79c3480295160d01baf8d5dc
I20260809 12:14:56.439845 19783 tablet_service.cc:1514] Processing DeleteTablet for tablet 3f43c98c79c3480295160d01baf8d5dc with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-09 12:14:56 UTC) from {username='kudu', principal='kudu/127.0.222.62@KRBTEST.COM'} at 127.0.0.1:33466
I20260809 12:14:56.439905 19492 tablet_service.cc:1514] Processing DeleteTablet for tablet 3f43c98c79c3480295160d01baf8d5dc with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-09 12:14:56 UTC) from {username='kudu', principal='kudu/127.0.222.62@KRBTEST.COM'} at 127.0.0.1:39424
I20260809 12:14:56.440291 19640 tablet_service.cc:1514] Processing DeleteTablet for tablet 3f43c98c79c3480295160d01baf8d5dc with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-09 12:14:56 UTC) from {username='kudu', principal='kudu/127.0.222.62@KRBTEST.COM'} at 127.0.0.1:58190
I20260809 12:14:56.440943 19908 tablet_replica.cc:330] T 3f43c98c79c3480295160d01baf8d5dc P b88447d41a7f4ca4b3f742239780061b: stopping tablet replica
I20260809 12:14:56.441685 19908 raft_consensus.cc:2229] T 3f43c98c79c3480295160d01baf8d5dc P b88447d41a7f4ca4b3f742239780061b [term 1 FOLLOWER]: Raft consensus shutting down.
I20260809 12:14:56.442091 888 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskgH5Pjm/build/asan/bin/kudu pid 19423
I20260809 12:14:56.442319 19908 raft_consensus.cc:2258] T 3f43c98c79c3480295160d01baf8d5dc P b88447d41a7f4ca4b3f742239780061b [term 1 FOLLOWER]: Raft consensus is shut down!
I20260809 12:14:56.444579 19908 ts_tablet_manager.cc:1905] T 3f43c98c79c3480295160d01baf8d5dc P b88447d41a7f4ca4b3f742239780061b: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260809 12:14:56.451056 19909 tablet_replica.cc:330] T 3f43c98c79c3480295160d01baf8d5dc P 6a7d552ab0be491a8f544bb5c7f003d1: stopping tablet replica
I20260809 12:14:56.451587 19909 raft_consensus.cc:2229] T 3f43c98c79c3480295160d01baf8d5dc P 6a7d552ab0be491a8f544bb5c7f003d1 [term 1 LEADER]: Raft consensus shutting down.
I20260809 12:14:56.452338 19909 raft_consensus.cc:2258] T 3f43c98c79c3480295160d01baf8d5dc P 6a7d552ab0be491a8f544bb5c7f003d1 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260809 12:14:56.454391 19909 ts_tablet_manager.cc:1905] T 3f43c98c79c3480295160d01baf8d5dc P 6a7d552ab0be491a8f544bb5c7f003d1: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260809 12:14:56.454600 19908 ts_tablet_manager.cc:1918] T 3f43c98c79c3480295160d01baf8d5dc P b88447d41a7f4ca4b3f742239780061b: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260809 12:14:56.454864 19908 log.cc:1196] T 3f43c98c79c3480295160d01baf8d5dc P b88447d41a7f4ca4b3f742239780061b: Deleting WAL directory at /tmp/dist-test-taskgH5Pjm/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786277382073938-888-0/minicluster-data/ts-0/wal/wals/3f43c98c79c3480295160d01baf8d5dc
I20260809 12:14:56.455559 19908 ts_tablet_manager.cc:1939] T 3f43c98c79c3480295160d01baf8d5dc P b88447d41a7f4ca4b3f742239780061b: Deleting consensus metadata
I20260809 12:14:56.457749 19347 catalog_manager.cc:4841] TS b88447d41a7f4ca4b3f742239780061b (127.0.222.1:37951): tablet 3f43c98c79c3480295160d01baf8d5dc (table test-table [id=214415e2ad0a4acfacf747eb44be5143]) successfully deleted
I20260809 12:14:56.458609 19910 tablet_replica.cc:330] T 3f43c98c79c3480295160d01baf8d5dc P 0053eca7b1ac45d8b5672e15b7724512: stopping tablet replica
I20260809 12:14:56.459185 19910 raft_consensus.cc:2229] T 3f43c98c79c3480295160d01baf8d5dc P 0053eca7b1ac45d8b5672e15b7724512 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260809 12:14:56.459782 19910 raft_consensus.cc:2258] T 3f43c98c79c3480295160d01baf8d5dc P 0053eca7b1ac45d8b5672e15b7724512 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260809 12:14:56.461884 19910 ts_tablet_manager.cc:1905] T 3f43c98c79c3480295160d01baf8d5dc P 0053eca7b1ac45d8b5672e15b7724512: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260809 12:14:56.463158 19909 ts_tablet_manager.cc:1918] T 3f43c98c79c3480295160d01baf8d5dc P 6a7d552ab0be491a8f544bb5c7f003d1: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260809 12:14:56.463377 19909 log.cc:1196] T 3f43c98c79c3480295160d01baf8d5dc P 6a7d552ab0be491a8f544bb5c7f003d1: Deleting WAL directory at /tmp/dist-test-taskgH5Pjm/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786277382073938-888-0/minicluster-data/ts-2/wal/wals/3f43c98c79c3480295160d01baf8d5dc
I20260809 12:14:56.464022 19909 ts_tablet_manager.cc:1939] T 3f43c98c79c3480295160d01baf8d5dc P 6a7d552ab0be491a8f544bb5c7f003d1: Deleting consensus metadata
Aug 09 12:14:56 dist-test-slave-2w3w krb5kdc[19310](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786277693, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.222.1@KRBTEST.COM
I20260809 12:14:56.465890 19344 catalog_manager.cc:4841] TS 6a7d552ab0be491a8f544bb5c7f003d1 (127.0.222.3:41815): tablet 3f43c98c79c3480295160d01baf8d5dc (table test-table [id=214415e2ad0a4acfacf747eb44be5143]) successfully deleted
I20260809 12:14:56.468654 19910 ts_tablet_manager.cc:1918] T 3f43c98c79c3480295160d01baf8d5dc P 0053eca7b1ac45d8b5672e15b7724512: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260809 12:14:56.468830 19910 log.cc:1196] T 3f43c98c79c3480295160d01baf8d5dc P 0053eca7b1ac45d8b5672e15b7724512: Deleting WAL directory at /tmp/dist-test-taskgH5Pjm/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786277382073938-888-0/minicluster-data/ts-1/wal/wals/3f43c98c79c3480295160d01baf8d5dc
I20260809 12:14:56.469436 19910 ts_tablet_manager.cc:1939] T 3f43c98c79c3480295160d01baf8d5dc P 0053eca7b1ac45d8b5672e15b7724512: Deleting consensus metadata
I20260809 12:14:56.470912 19345 catalog_manager.cc:4841] TS 0053eca7b1ac45d8b5672e15b7724512 (127.0.222.2:40181): tablet 3f43c98c79c3480295160d01baf8d5dc (table test-table [id=214415e2ad0a4acfacf747eb44be5143]) successfully deleted
I20260809 12:14:56.474155 19877 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 12:14:56.443611 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39434 (local address 127.0.222.1:37951)
0809 12:14:56.443780 (+ 169us) server_negotiation.cc:206] Beginning negotiation
0809 12:14:56.443787 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0809 12:14:56.454204 (+ 10417us) server_negotiation.cc:407] Connection header received
0809 12:14:56.454731 (+ 527us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 12:14:56.454736 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 12:14:56.454872 (+ 136us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 12:14:56.454975 (+ 103us) server_negotiation.cc:226] Negotiated authn=SASL
0809 12:14:56.457028 (+ 2053us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 12:14:56.458190 (+ 1162us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 12:14:56.460185 (+ 1995us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 12:14:56.460828 (+ 643us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 12:14:56.460933 (+ 105us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 12:14:56.466817 (+ 5884us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 12:14:56.466834 (+ 17us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 12:14:56.466853 (+ 19us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0809 12:14:56.466897 (+ 44us) server_negotiation.cc:922] calling sasl_server_start()
0809 12:14:56.471099 (+ 4202us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 12:14:56.471810 (+ 711us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 12:14:56.471820 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 12:14:56.471823 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0809 12:14:56.471988 (+ 165us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 12:14:56.472352 (+ 364us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 12:14:56.472357 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 12:14:56.472361 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0809 12:14:56.472865 (+ 504us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 12:14:56.473173 (+ 308us) server_negotiation.cc:1035] Waiting for connection context
0809 12:14:56.473498 (+ 325us) server_negotiation.cc:299] Negotiation successful
0809 12:14:56.473734 (+ 236us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":61}
I20260809 12:14:56.475054 19556 generic_service.cc:196] Checking for leaks (request via RPC)
I20260809 12:14:56.587527 888 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskgH5Pjm/build/asan/bin/kudu with pid 19423
I20260809 12:14:56.615797 888 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskgH5Pjm/build/asan/bin/kudu pid 19567
Aug 09 12:14:56 dist-test-slave-2w3w krb5kdc[19310](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786277693, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.222.2@KRBTEST.COM
I20260809 12:14:56.634146 19875 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 12:14:56.616727 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58198 (local address 127.0.222.2:40181)
0809 12:14:56.616915 (+ 188us) server_negotiation.cc:206] Beginning negotiation
0809 12:14:56.616920 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0809 12:14:56.616939 (+ 19us) server_negotiation.cc:407] Connection header received
0809 12:14:56.617099 (+ 160us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 12:14:56.617103 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 12:14:56.617180 (+ 77us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 12:14:56.617283 (+ 103us) server_negotiation.cc:226] Negotiated authn=SASL
0809 12:14:56.619092 (+ 1809us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 12:14:56.619859 (+ 767us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 12:14:56.621797 (+ 1938us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 12:14:56.622308 (+ 511us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 12:14:56.622406 (+ 98us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 12:14:56.627750 (+ 5344us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 12:14:56.627767 (+ 17us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 12:14:56.627783 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0809 12:14:56.627819 (+ 36us) server_negotiation.cc:922] calling sasl_server_start()
0809 12:14:56.630861 (+ 3042us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 12:14:56.631776 (+ 915us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 12:14:56.631793 (+ 17us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 12:14:56.631795 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0809 12:14:56.631926 (+ 131us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 12:14:56.632321 (+ 395us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 12:14:56.632328 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 12:14:56.632331 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0809 12:14:56.632753 (+ 422us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 12:14:56.633170 (+ 417us) server_negotiation.cc:1035] Waiting for connection context
0809 12:14:56.633518 (+ 348us) server_negotiation.cc:299] Negotiation successful
0809 12:14:56.633789 (+ 271us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":60}
I20260809 12:14:56.634842 19703 generic_service.cc:196] Checking for leaks (request via RPC)
I20260809 12:14:56.727721 888 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskgH5Pjm/build/asan/bin/kudu with pid 19567
I20260809 12:14:56.751562 888 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskgH5Pjm/build/asan/bin/kudu pid 19716
Aug 09 12:14:56 dist-test-slave-2w3w krb5kdc[19310](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786277693, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.222.3@KRBTEST.COM
I20260809 12:14:56.773049 19872 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 12:14:56.752512 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33480 (local address 127.0.222.3:41815)
0809 12:14:56.752736 (+ 224us) server_negotiation.cc:206] Beginning negotiation
0809 12:14:56.752743 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0809 12:14:56.752770 (+ 27us) server_negotiation.cc:407] Connection header received
0809 12:14:56.752945 (+ 175us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 12:14:56.752951 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 12:14:56.753040 (+ 89us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 12:14:56.753159 (+ 119us) server_negotiation.cc:226] Negotiated authn=SASL
0809 12:14:56.754938 (+ 1779us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 12:14:56.756154 (+ 1216us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 12:14:56.758143 (+ 1989us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 12:14:56.758899 (+ 756us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 12:14:56.759224 (+ 325us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 12:14:56.765142 (+ 5918us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 12:14:56.765156 (+ 14us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 12:14:56.765175 (+ 19us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0809 12:14:56.765222 (+ 47us) server_negotiation.cc:922] calling sasl_server_start()
0809 12:14:56.769707 (+ 4485us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 12:14:56.770611 (+ 904us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 12:14:56.770620 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 12:14:56.770623 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0809 12:14:56.770774 (+ 151us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 12:14:56.771289 (+ 515us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 12:14:56.771294 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 12:14:56.771298 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0809 12:14:56.771757 (+ 459us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 12:14:56.772042 (+ 285us) server_negotiation.cc:1035] Waiting for connection context
0809 12:14:56.772425 (+ 383us) server_negotiation.cc:299] Negotiation successful
0809 12:14:56.772682 (+ 257us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":59}
I20260809 12:14:56.773945 19853 generic_service.cc:196] Checking for leaks (request via RPC)
I20260809 12:14:56.898005 888 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskgH5Pjm/build/asan/bin/kudu with pid 19716
I20260809 12:14:56.921916 888 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskgH5Pjm/build/asan/bin/kudu pid 19324
I20260809 12:14:56.923182 19392 generic_service.cc:196] Checking for leaks (request via RPC)
I20260809 12:14:57.051048 888 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskgH5Pjm/build/asan/bin/kudu with pid 19324
2026-08-09T12:14:57Z chronyd exiting
[ OK ] EncryptionPolicyTest.TestEncryptionPolicy/0 (5034 ms)
[ RUN ] EncryptionPolicyTest.TestEncryptionPolicy/1
2026-08-09T12:14:57Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-09T12:14:57Z Disabled control of system clock
I20260809 12:14:57.149050 888 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskgH5Pjm/build/asan/bin/kudu
/tmp/dist-test-taskgH5Pjm/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskgH5Pjm/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786277382073938-888-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskgH5Pjm/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786277382073938-888-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskgH5Pjm/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786277382073938-888-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskgH5Pjm/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786277382073938-888-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.222.62:37983
--webserver_interface=127.0.222.62
--webserver_port=0
--builtin_ntp_servers=127.0.222.20:41619
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.222.62:37983
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20260809 12:14:57.493569 19919 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260809 12:14:57.493971 19919 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260809 12:14:57.494144 19919 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260809 12:14:57.501734 19919 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260809 12:14:57.501941 19919 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260809 12:14:57.501984 19919 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260809 12:14:57.502045 19919 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260809 12:14:57.502084 19919 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260809 12:14:57.511298 19919 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.222.20:41619
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskgH5Pjm/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786277382073938-888-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskgH5Pjm/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786277382073938-888-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.222.62:37983
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.222.62:37983
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskgH5Pjm/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786277382073938-888-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.222.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskgH5Pjm/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786277382073938-888-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 09 Aug 2026 11:06:41 UTC on ccycloud.quasar-bwoogy.root.comops.site
build id 4575991
ASAN enabled
I20260809 12:14:57.512538 19919 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260809 12:14:57.514322 19919 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260809 12:14:57.526911 19928 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 12:14:57.526973 19927 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 12:14:57.527624 19930 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260809 12:14:57.527729 19919 server_base.cc:1034] running on GCE node
I20260809 12:14:57.528553 19919 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260809 12:14:57.529991 19919 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260809 12:14:57.531229 19919 hybrid_clock.cc:648] HybridClock initialized: now 1786277697531197 us; error 44 us; skew 500 ppm
I20260809 12:14:57.531582 19919 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260809 12:14:57.539381 19919 webserver.cc:460] Webserver started at http://127.0.222.62:44869/ using document root <none> and password file <none>
I20260809 12:14:57.540112 19919 fs_manager.cc:359] Metadata directory not provided
I20260809 12:14:57.540258 19919 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260809 12:14:57.540674 19919 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260809 12:14:57.543715 19919 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskgH5Pjm/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786277382073938-888-0/minicluster-data/master-0/data/instance:
uuid: "1ba08d3a915e4f35b5c89ed2df131ff8"
format_stamp: "Formatted at 2026-08-09 12:14:57 on dist-test-slave-2w3w"
server_key: "b3a5644768521d0d10ddea225e329913"
server_key_iv: "a715572f5c4596491d6f5ea3b5421742"
server_key_version: "encryptionkey@0"
I20260809 12:14:57.544549 19919 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskgH5Pjm/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786277382073938-888-0/minicluster-data/master-0/wal/instance:
uuid: "1ba08d3a915e4f35b5c89ed2df131ff8"
format_stamp: "Formatted at 2026-08-09 12:14:57 on dist-test-slave-2w3w"
server_key: "b3a5644768521d0d10ddea225e329913"
server_key_iv: "a715572f5c4596491d6f5ea3b5421742"
server_key_version: "encryptionkey@0"
I20260809 12:14:57.549983 19919 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.001s
I20260809 12:14:57.553505 19936 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260809 12:14:57.554939 19919 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260809 12:14:57.555199 19919 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskgH5Pjm/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786277382073938-888-0/minicluster-data/master-0/data,/tmp/dist-test-taskgH5Pjm/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786277382073938-888-0/minicluster-data/master-0/wal
uuid: "1ba08d3a915e4f35b5c89ed2df131ff8"
format_stamp: "Formatted at 2026-08-09 12:14:57 on dist-test-slave-2w3w"
server_key: "b3a5644768521d0d10ddea225e329913"
server_key_iv: "a715572f5c4596491d6f5ea3b5421742"
server_key_version: "encryptionkey@0"
I20260809 12:14:57.555423 19919 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskgH5Pjm/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786277382073938-888-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskgH5Pjm/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786277382073938-888-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskgH5Pjm/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786277382073938-888-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260809 12:14:57.605307 19919 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260809 12:14:57.606242 19919 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260809 12:14:57.606596 19919 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260809 12:14:57.626128 19919 rpc_server.cc:307] RPC server started. Bound to: 127.0.222.62:37983
I20260809 12:14:57.626160 19993 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.222.62:37983 every 8 connection(s)
I20260809 12:14:57.627549 19919 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskgH5Pjm/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786277382073938-888-0/minicluster-data/master-0/data/info.pb
I20260809 12:14:57.628979 888 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskgH5Pjm/build/asan/bin/kudu as pid 19919
I20260809 12:14:57.629185 888 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskgH5Pjm/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786277382073938-888-0/minicluster-data/master-0/wal/instance
I20260809 12:14:57.629760 888 external_mini_cluster.cc:1428] Setting key 998f4e6d427837273af7c0087418b339
I20260809 12:14:57.632337 19994 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260809 12:14:57.643415 19997 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 12:14:57.631773 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35056 (local address 127.0.222.62:37983)
0809 12:14:57.632862 (+ 1089us) server_negotiation.cc:206] Beginning negotiation
0809 12:14:57.632873 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0809 12:14:57.632915 (+ 42us) server_negotiation.cc:407] Connection header received
0809 12:14:57.634563 (+ 1648us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 12:14:57.634584 (+ 21us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 12:14:57.635112 (+ 528us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 12:14:57.635530 (+ 418us) server_negotiation.cc:226] Negotiated authn=SASL
0809 12:14:57.636077 (+ 547us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 12:14:57.637431 (+ 1354us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 12:14:57.638930 (+ 1499us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 12:14:57.639635 (+ 705us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 12:14:57.639847 (+ 212us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 12:14:57.641108 (+ 1261us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 12:14:57.641116 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 12:14:57.641131 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0809 12:14:57.641695 (+ 564us) server_negotiation.cc:922] calling sasl_server_start()
0809 12:14:57.641780 (+ 85us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0809 12:14:57.641802 (+ 22us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 12:14:57.641931 (+ 129us) server_negotiation.cc:1035] Waiting for connection context
0809 12:14:57.642160 (+ 229us) server_negotiation.cc:299] Negotiation successful
0809 12:14:57.642449 (+ 289us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":654,"thread_start_us":401,"threads_started":1}
I20260809 12:14:57.643471 19994 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 1ba08d3a915e4f35b5c89ed2df131ff8: Bootstrap starting.
I20260809 12:14:57.646581 19994 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 1ba08d3a915e4f35b5c89ed2df131ff8: Neither blocks nor log segments found. Creating new log.
I20260809 12:14:57.647651 19994 log.cc:824] T 00000000000000000000000000000000 P 1ba08d3a915e4f35b5c89ed2df131ff8: Log is configured to *not* fsync() on all Append() calls
I20260809 12:14:57.650349 19994 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 1ba08d3a915e4f35b5c89ed2df131ff8: No bootstrap required, opened a new log
I20260809 12:14:57.657866 19994 raft_consensus.cc:348] T 00000000000000000000000000000000 P 1ba08d3a915e4f35b5c89ed2df131ff8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1ba08d3a915e4f35b5c89ed2df131ff8" member_type: VOTER last_known_addr { host: "127.0.222.62" port: 37983 } }
I20260809 12:14:57.658216 19994 raft_consensus.cc:374] T 00000000000000000000000000000000 P 1ba08d3a915e4f35b5c89ed2df131ff8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260809 12:14:57.658314 19994 raft_consensus.cc:729] T 00000000000000000000000000000000 P 1ba08d3a915e4f35b5c89ed2df131ff8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1ba08d3a915e4f35b5c89ed2df131ff8, State: Initialized, Role: FOLLOWER
I20260809 12:14:57.659024 19994 consensus_queue.cc:260] T 00000000000000000000000000000000 P 1ba08d3a915e4f35b5c89ed2df131ff8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1ba08d3a915e4f35b5c89ed2df131ff8" member_type: VOTER last_known_addr { host: "127.0.222.62" port: 37983 } }
I20260809 12:14:57.659338 19994 raft_consensus.cc:388] T 00000000000000000000000000000000 P 1ba08d3a915e4f35b5c89ed2df131ff8 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260809 12:14:57.659497 19994 raft_consensus.cc:482] T 00000000000000000000000000000000 P 1ba08d3a915e4f35b5c89ed2df131ff8 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260809 12:14:57.659724 19994 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 1ba08d3a915e4f35b5c89ed2df131ff8 [term 0 FOLLOWER]: Advancing to term 1
I20260809 12:14:57.661876 19994 raft_consensus.cc:504] T 00000000000000000000000000000000 P 1ba08d3a915e4f35b5c89ed2df131ff8 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1ba08d3a915e4f35b5c89ed2df131ff8" member_type: VOTER last_known_addr { host: "127.0.222.62" port: 37983 } }
I20260809 12:14:57.662379 19994 leader_election.cc:302] T 00000000000000000000000000000000 P 1ba08d3a915e4f35b5c89ed2df131ff8 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 1ba08d3a915e4f35b5c89ed2df131ff8; no voters:
I20260809 12:14:57.662951 19994 leader_election.cc:288] T 00000000000000000000000000000000 P 1ba08d3a915e4f35b5c89ed2df131ff8 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260809 12:14:57.663164 19999 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 1ba08d3a915e4f35b5c89ed2df131ff8 [term 1 FOLLOWER]: Leader election won for term 1
I20260809 12:14:57.663903 19999 raft_consensus.cc:686] T 00000000000000000000000000000000 P 1ba08d3a915e4f35b5c89ed2df131ff8 [term 1 LEADER]: Becoming Leader. State: Replica: 1ba08d3a915e4f35b5c89ed2df131ff8, State: Running, Role: LEADER
I20260809 12:14:57.664513 19999 consensus_queue.cc:237] T 00000000000000000000000000000000 P 1ba08d3a915e4f35b5c89ed2df131ff8 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1ba08d3a915e4f35b5c89ed2df131ff8" member_type: VOTER last_known_addr { host: "127.0.222.62" port: 37983 } }
I20260809 12:14:57.665091 19994 sys_catalog.cc:564] T 00000000000000000000000000000000 P 1ba08d3a915e4f35b5c89ed2df131ff8 [sys.catalog]: configured and running, proceeding with master startup.
I20260809 12:14:57.667145 20001 sys_catalog.cc:455] T 00000000000000000000000000000000 P 1ba08d3a915e4f35b5c89ed2df131ff8 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 1ba08d3a915e4f35b5c89ed2df131ff8. Latest consensus state: current_term: 1 leader_uuid: "1ba08d3a915e4f35b5c89ed2df131ff8" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1ba08d3a915e4f35b5c89ed2df131ff8" member_type: VOTER last_known_addr { host: "127.0.222.62" port: 37983 } } }
I20260809 12:14:57.667522 20001 sys_catalog.cc:458] T 00000000000000000000000000000000 P 1ba08d3a915e4f35b5c89ed2df131ff8 [sys.catalog]: This master's current role is: LEADER
I20260809 12:14:57.667794 20000 sys_catalog.cc:455] T 00000000000000000000000000000000 P 1ba08d3a915e4f35b5c89ed2df131ff8 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "1ba08d3a915e4f35b5c89ed2df131ff8" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1ba08d3a915e4f35b5c89ed2df131ff8" member_type: VOTER last_known_addr { host: "127.0.222.62" port: 37983 } } }
I20260809 12:14:57.668286 20000 sys_catalog.cc:458] T 00000000000000000000000000000000 P 1ba08d3a915e4f35b5c89ed2df131ff8 [sys.catalog]: This master's current role is: LEADER
I20260809 12:14:57.674307 20004 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260809 12:14:57.679935 20004 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260809 12:14:57.689827 20004 catalog_manager.cc:1347] Generated new cluster ID: cdbec2836d6c4ea7a4014742952930c3
I20260809 12:14:57.689958 20004 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260809 12:14:57.715142 20004 catalog_manager.cc:1370] Generated new certificate authority record
I20260809 12:14:57.716528 20004 catalog_manager.cc:1504] Loading token signing keys...
I20260809 12:14:57.729023 20004 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 1ba08d3a915e4f35b5c89ed2df131ff8: Generated new TSK 0
I20260809 12:14:57.730070 20004 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260809 12:14:57.744563 888 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskgH5Pjm/build/asan/bin/kudu
/tmp/dist-test-taskgH5Pjm/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskgH5Pjm/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786277382073938-888-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskgH5Pjm/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786277382073938-888-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskgH5Pjm/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786277382073938-888-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskgH5Pjm/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786277382073938-888-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.222.1:0
--local_ip_for_outbound_sockets=127.0.222.1
--webserver_interface=127.0.222.1
--webserver_port=0
--tserver_master_addrs=127.0.222.62:37983
--builtin_ntp_servers=127.0.222.20:41619
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20260809 12:14:58.094448 20022 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260809 12:14:58.094806 20022 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260809 12:14:58.094995 20022 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260809 12:14:58.102351 20022 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260809 12:14:58.102489 20022 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260809 12:14:58.102667 20022 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.222.1
I20260809 12:14:58.112054 20022 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.222.20:41619
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskgH5Pjm/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786277382073938-888-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskgH5Pjm/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786277382073938-888-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.222.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskgH5Pjm/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786277382073938-888-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.222.1
--webserver_port=0
--tserver_master_addrs=127.0.222.62:37983
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.222.1
--log_dir=/tmp/dist-test-taskgH5Pjm/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786277382073938-888-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 09 Aug 2026 11:06:41 UTC on ccycloud.quasar-bwoogy.root.comops.site
build id 4575991
ASAN enabled
I20260809 12:14:58.113390 20022 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260809 12:14:58.115283 20022 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260809 12:14:58.128217 20029 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 12:14:58.128260 20031 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 12:14:58.128299 20028 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260809 12:14:58.129172 20022 server_base.cc:1034] running on GCE node
I20260809 12:14:58.129817 20022 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260809 12:14:58.130812 20022 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260809 12:14:58.132057 20022 hybrid_clock.cc:648] HybridClock initialized: now 1786277698132019 us; error 51 us; skew 500 ppm
I20260809 12:14:58.132383 20022 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260809 12:14:58.134650 20022 webserver.cc:460] Webserver started at http://127.0.222.1:34175/ using document root <none> and password file <none>
I20260809 12:14:58.135424 20022 fs_manager.cc:359] Metadata directory not provided
I20260809 12:14:58.135563 20022 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260809 12:14:58.135903 20022 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260809 12:14:58.138890 20022 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskgH5Pjm/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786277382073938-888-0/minicluster-data/ts-0/data/instance:
uuid: "94d4c13dd4e246a38799a6f0ef2947f8"
format_stamp: "Formatted at 2026-08-09 12:14:58 on dist-test-slave-2w3w"
server_key: "6a6357f3337a18d92d84798a4e1220e3"
server_key_iv: "15bd553abcf8583ea3d496941d3ae60f"
server_key_version: "encryptionkey@0"
I20260809 12:14:58.139778 20022 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskgH5Pjm/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786277382073938-888-0/minicluster-data/ts-0/wal/instance:
uuid: "94d4c13dd4e246a38799a6f0ef2947f8"
format_stamp: "Formatted at 2026-08-09 12:14:58 on dist-test-slave-2w3w"
server_key: "6a6357f3337a18d92d84798a4e1220e3"
server_key_iv: "15bd553abcf8583ea3d496941d3ae60f"
server_key_version: "encryptionkey@0"
I20260809 12:14:58.145013 20022 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.000s
I20260809 12:14:58.148584 20037 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260809 12:14:58.149978 20022 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.000s
I20260809 12:14:58.150182 20022 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskgH5Pjm/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786277382073938-888-0/minicluster-data/ts-0/data,/tmp/dist-test-taskgH5Pjm/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786277382073938-888-0/minicluster-data/ts-0/wal
uuid: "94d4c13dd4e246a38799a6f0ef2947f8"
format_stamp: "Formatted at 2026-08-09 12:14:58 on dist-test-slave-2w3w"
server_key: "6a6357f3337a18d92d84798a4e1220e3"
server_key_iv: "15bd553abcf8583ea3d496941d3ae60f"
server_key_version: "encryptionkey@0"
I20260809 12:14:58.150422 20022 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskgH5Pjm/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786277382073938-888-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskgH5Pjm/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786277382073938-888-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskgH5Pjm/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786277382073938-888-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260809 12:14:58.180208 20022 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260809 12:14:58.181200 20022 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260809 12:14:58.181486 20022 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260809 12:14:58.182467 20022 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260809 12:14:58.184033 20022 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260809 12:14:58.184160 20022 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260809 12:14:58.184288 20022 ts_tablet_manager.cc:610] Registered 0 tablets
I20260809 12:14:58.184379 20022 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260809 12:14:58.217799 20022 rpc_server.cc:307] RPC server started. Bound to: 127.0.222.1:45971
I20260809 12:14:58.217844 20156 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.222.1:45971 every 8 connection(s)
I20260809 12:14:58.219206 20022 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskgH5Pjm/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786277382073938-888-0/minicluster-data/ts-0/data/info.pb
I20260809 12:14:58.228341 888 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskgH5Pjm/build/asan/bin/kudu as pid 20022
I20260809 12:14:58.228524 888 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskgH5Pjm/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786277382073938-888-0/minicluster-data/ts-0/wal/instance
I20260809 12:14:58.229118 888 external_mini_cluster.cc:1428] Setting key 40497dd9195032f307ae53a064380ac9
I20260809 12:14:58.232182 888 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskgH5Pjm/build/asan/bin/kudu
/tmp/dist-test-taskgH5Pjm/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskgH5Pjm/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786277382073938-888-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskgH5Pjm/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786277382073938-888-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskgH5Pjm/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786277382073938-888-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskgH5Pjm/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786277382073938-888-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.222.2:0
--local_ip_for_outbound_sockets=127.0.222.2
--webserver_interface=127.0.222.2
--webserver_port=0
--tserver_master_addrs=127.0.222.62:37983
--builtin_ntp_servers=127.0.222.20:41619
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
I20260809 12:14:58.232331 20162 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 12:14:58.221823 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.222.1:59215 (local address 127.0.222.62:37983)
0809 12:14:58.222319 (+ 496us) server_negotiation.cc:206] Beginning negotiation
0809 12:14:58.222325 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0809 12:14:58.223557 (+ 1232us) server_negotiation.cc:407] Connection header received
0809 12:14:58.225284 (+ 1727us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 12:14:58.225288 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 12:14:58.225354 (+ 66us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 12:14:58.225455 (+ 101us) server_negotiation.cc:226] Negotiated authn=SASL
0809 12:14:58.226437 (+ 982us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 12:14:58.227188 (+ 751us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 12:14:58.229476 (+ 2288us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 12:14:58.229875 (+ 399us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 12:14:58.230051 (+ 176us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 12:14:58.231456 (+ 1405us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 12:14:58.231462 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 12:14:58.231466 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0809 12:14:58.231494 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0809 12:14:58.231532 (+ 38us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0809 12:14:58.231565 (+ 33us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 12:14:58.231659 (+ 94us) server_negotiation.cc:1035] Waiting for connection context
0809 12:14:58.231933 (+ 274us) server_negotiation.cc:299] Negotiation successful
0809 12:14:58.232017 (+ 84us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":362,"thread_start_us":243,"threads_started":1}
I20260809 12:14:58.233790 20163 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 12:14:58.222443 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.222.62:37983 (local address 127.0.222.1:59215)
0809 12:14:58.223412 (+ 969us) negotiation.cc:107] Waiting for socket to connect
0809 12:14:58.223448 (+ 36us) client_negotiation.cc:174] Beginning negotiation
0809 12:14:58.225047 (+ 1599us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 12:14:58.225649 (+ 602us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 12:14:58.225656 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 12:14:58.225736 (+ 80us) client_negotiation.cc:189] Negotiated authn=SASL
0809 12:14:58.226275 (+ 539us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 12:14:58.226289 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 12:14:58.227332 (+ 1043us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 12:14:58.227336 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 12:14:58.229332 (+ 1996us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 12:14:58.229349 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 12:14:58.230114 (+ 765us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 12:14:58.230121 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 12:14:58.230368 (+ 247us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 12:14:58.231151 (+ 783us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0809 12:14:58.231153 (+ 2us) client_negotiation.cc:656] calling sasl_client_start()
0809 12:14:58.231193 (+ 40us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0809 12:14:58.231269 (+ 76us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0809 12:14:58.231688 (+ 419us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0809 12:14:58.231700 (+ 12us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0809 12:14:58.231708 (+ 8us) client_negotiation.cc:769] Sending connection context
0809 12:14:58.231912 (+ 204us) client_negotiation.cc:240] Negotiation successful
0809 12:14:58.232352 (+ 440us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":475,"thread_start_us":288,"threads_started":1}
I20260809 12:14:58.235468 20157 heartbeater.cc:347] Connected to a master server at 127.0.222.62:37983
I20260809 12:14:58.235791 20157 heartbeater.cc:464] Registering TS with master...
I20260809 12:14:58.236549 20157 heartbeater.cc:511] Master 127.0.222.62:37983 requested a full tablet report, sending...
I20260809 12:14:58.238843 19955 ts_manager.cc:194] Registered new tserver with Master: 94d4c13dd4e246a38799a6f0ef2947f8 (127.0.222.1:45971)
I20260809 12:14:58.240984 19955 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.222.1:59215
W20260809 12:14:58.577723 20164 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260809 12:14:58.577997 20164 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260809 12:14:58.578135 20164 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260809 12:14:58.585695 20164 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260809 12:14:58.585798 20164 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260809 12:14:58.585963 20164 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.222.2
I20260809 12:14:58.595299 20164 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.222.20:41619
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskgH5Pjm/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786277382073938-888-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskgH5Pjm/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786277382073938-888-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.222.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskgH5Pjm/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786277382073938-888-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.222.2
--webserver_port=0
--tserver_master_addrs=127.0.222.62:37983
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.222.2
--log_dir=/tmp/dist-test-taskgH5Pjm/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786277382073938-888-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 09 Aug 2026 11:06:41 UTC on ccycloud.quasar-bwoogy.root.comops.site
build id 4575991
ASAN enabled
I20260809 12:14:58.596630 20164 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260809 12:14:58.598920 20164 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260809 12:14:58.612010 20169 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 12:14:58.611979 20173 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 12:14:58.612066 20171 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260809 12:14:58.612937 20164 server_base.cc:1034] running on GCE node
I20260809 12:14:58.613477 20164 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260809 12:14:58.614454 20164 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260809 12:14:58.615705 20164 hybrid_clock.cc:648] HybridClock initialized: now 1786277698615631 us; error 67 us; skew 500 ppm
I20260809 12:14:58.616072 20164 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260809 12:14:58.618288 20164 webserver.cc:460] Webserver started at http://127.0.222.2:45817/ using document root <none> and password file <none>
I20260809 12:14:58.619045 20164 fs_manager.cc:359] Metadata directory not provided
I20260809 12:14:58.619176 20164 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260809 12:14:58.619529 20164 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260809 12:14:58.622306 20164 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskgH5Pjm/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786277382073938-888-0/minicluster-data/ts-1/data/instance:
uuid: "3ea81fb4f9564a12b0b50550947cfb60"
format_stamp: "Formatted at 2026-08-09 12:14:58 on dist-test-slave-2w3w"
server_key: "2b1ee6da44a16de05fe3e71a69557f24"
server_key_iv: "5b295e54e375896e92e7c9a465ce35a9"
server_key_version: "encryptionkey@0"
I20260809 12:14:58.623196 20164 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskgH5Pjm/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786277382073938-888-0/minicluster-data/ts-1/wal/instance:
uuid: "3ea81fb4f9564a12b0b50550947cfb60"
format_stamp: "Formatted at 2026-08-09 12:14:58 on dist-test-slave-2w3w"
server_key: "2b1ee6da44a16de05fe3e71a69557f24"
server_key_iv: "5b295e54e375896e92e7c9a465ce35a9"
server_key_version: "encryptionkey@0"
I20260809 12:14:58.628584 20164 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.004s sys 0.000s
I20260809 12:14:58.632169 20180 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260809 12:14:58.633517 20164 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.001s
I20260809 12:14:58.633702 20164 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskgH5Pjm/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786277382073938-888-0/minicluster-data/ts-1/data,/tmp/dist-test-taskgH5Pjm/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786277382073938-888-0/minicluster-data/ts-1/wal
uuid: "3ea81fb4f9564a12b0b50550947cfb60"
format_stamp: "Formatted at 2026-08-09 12:14:58 on dist-test-slave-2w3w"
server_key: "2b1ee6da44a16de05fe3e71a69557f24"
server_key_iv: "5b295e54e375896e92e7c9a465ce35a9"
server_key_version: "encryptionkey@0"
I20260809 12:14:58.633920 20164 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskgH5Pjm/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786277382073938-888-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskgH5Pjm/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786277382073938-888-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskgH5Pjm/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786277382073938-888-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260809 12:14:58.650952 20164 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260809 12:14:58.651983 20164 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260809 12:14:58.652230 20164 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260809 12:14:58.653198 20164 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260809 12:14:58.654851 20164 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260809 12:14:58.654992 20164 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260809 12:14:58.655102 20164 ts_tablet_manager.cc:610] Registered 0 tablets
I20260809 12:14:58.655169 20164 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260809 12:14:58.690552 20164 rpc_server.cc:307] RPC server started. Bound to: 127.0.222.2:37547
I20260809 12:14:58.690611 20299 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.222.2:37547 every 8 connection(s)
I20260809 12:14:58.692003 20164 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskgH5Pjm/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786277382073938-888-0/minicluster-data/ts-1/data/info.pb
I20260809 12:14:58.695645 888 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskgH5Pjm/build/asan/bin/kudu as pid 20164
I20260809 12:14:58.695861 888 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskgH5Pjm/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786277382073938-888-0/minicluster-data/ts-1/wal/instance
I20260809 12:14:58.696345 888 external_mini_cluster.cc:1428] Setting key 0134ccf06e8b47ca75c9cd30437f550e
I20260809 12:14:58.699481 888 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskgH5Pjm/build/asan/bin/kudu
/tmp/dist-test-taskgH5Pjm/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskgH5Pjm/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786277382073938-888-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskgH5Pjm/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786277382073938-888-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskgH5Pjm/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786277382073938-888-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskgH5Pjm/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786277382073938-888-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.222.3:0
--local_ip_for_outbound_sockets=127.0.222.3
--webserver_interface=127.0.222.3
--webserver_port=0
--tserver_master_addrs=127.0.222.62:37983
--builtin_ntp_servers=127.0.222.20:41619
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
I20260809 12:14:58.705829 20162 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 12:14:58.694735 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.222.2:32933 (local address 127.0.222.62:37983)
0809 12:14:58.694999 (+ 264us) server_negotiation.cc:206] Beginning negotiation
0809 12:14:58.695006 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0809 12:14:58.696599 (+ 1593us) server_negotiation.cc:407] Connection header received
0809 12:14:58.698598 (+ 1999us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 12:14:58.698603 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 12:14:58.698685 (+ 82us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 12:14:58.698800 (+ 115us) server_negotiation.cc:226] Negotiated authn=SASL
0809 12:14:58.699976 (+ 1176us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 12:14:58.701184 (+ 1208us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 12:14:58.702782 (+ 1598us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 12:14:58.703491 (+ 709us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 12:14:58.703637 (+ 146us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 12:14:58.704933 (+ 1296us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 12:14:58.704941 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 12:14:58.704949 (+ 8us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0809 12:14:58.704974 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0809 12:14:58.705019 (+ 45us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0809 12:14:58.705037 (+ 18us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 12:14:58.705133 (+ 96us) server_negotiation.cc:1035] Waiting for connection context
0809 12:14:58.705337 (+ 204us) server_negotiation.cc:299] Negotiation successful
0809 12:14:58.705447 (+ 110us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":79}
I20260809 12:14:58.706683 20302 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 12:14:58.695323 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.222.62:37983 (local address 127.0.222.2:32933)
0809 12:14:58.696412 (+ 1089us) negotiation.cc:107] Waiting for socket to connect
0809 12:14:58.696458 (+ 46us) client_negotiation.cc:174] Beginning negotiation
0809 12:14:58.698345 (+ 1887us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 12:14:58.699075 (+ 730us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 12:14:58.699083 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 12:14:58.699169 (+ 86us) client_negotiation.cc:189] Negotiated authn=SASL
0809 12:14:58.699806 (+ 637us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 12:14:58.699825 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 12:14:58.701331 (+ 1506us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 12:14:58.701335 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 12:14:58.702633 (+ 1298us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 12:14:58.702646 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 12:14:58.703696 (+ 1050us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 12:14:58.703701 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 12:14:58.703897 (+ 196us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 12:14:58.704616 (+ 719us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0809 12:14:58.704618 (+ 2us) client_negotiation.cc:656] calling sasl_client_start()
0809 12:14:58.704662 (+ 44us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0809 12:14:58.704736 (+ 74us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0809 12:14:58.705136 (+ 400us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0809 12:14:58.705144 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0809 12:14:58.705151 (+ 7us) client_negotiation.cc:769] Sending connection context
0809 12:14:58.705288 (+ 137us) client_negotiation.cc:240] Negotiation successful
0809 12:14:58.705568 (+ 280us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":568,"thread_start_us":384,"threads_started":1}
I20260809 12:14:58.708184 20300 heartbeater.cc:347] Connected to a master server at 127.0.222.62:37983
I20260809 12:14:58.708529 20300 heartbeater.cc:464] Registering TS with master...
I20260809 12:14:58.709308 20300 heartbeater.cc:511] Master 127.0.222.62:37983 requested a full tablet report, sending...
I20260809 12:14:58.710767 19955 ts_manager.cc:194] Registered new tserver with Master: 3ea81fb4f9564a12b0b50550947cfb60 (127.0.222.2:37547)
I20260809 12:14:58.711647 19955 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.222.2:32933
W20260809 12:14:59.038215 20303 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260809 12:14:59.038543 20303 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260809 12:14:59.038704 20303 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260809 12:14:59.047785 20303 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260809 12:14:59.047909 20303 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260809 12:14:59.048085 20303 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.222.3
I20260809 12:14:59.058391 20303 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.222.20:41619
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskgH5Pjm/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786277382073938-888-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskgH5Pjm/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786277382073938-888-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.222.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskgH5Pjm/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786277382073938-888-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.222.3
--webserver_port=0
--tserver_master_addrs=127.0.222.62:37983
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.222.3
--log_dir=/tmp/dist-test-taskgH5Pjm/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786277382073938-888-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 09 Aug 2026 11:06:41 UTC on ccycloud.quasar-bwoogy.root.comops.site
build id 4575991
ASAN enabled
I20260809 12:14:59.059677 20303 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260809 12:14:59.061479 20303 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260809 12:14:59.074563 20308 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 12:14:59.074563 20311 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 12:14:59.074689 20313 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260809 12:14:59.080363 20303 server_base.cc:1034] running on GCE node
I20260809 12:14:59.080937 20303 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260809 12:14:59.081821 20303 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260809 12:14:59.083068 20303 hybrid_clock.cc:648] HybridClock initialized: now 1786277699083018 us; error 29 us; skew 500 ppm
I20260809 12:14:59.083359 20303 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260809 12:14:59.085577 20303 webserver.cc:460] Webserver started at http://127.0.222.3:35473/ using document root <none> and password file <none>
I20260809 12:14:59.086216 20303 fs_manager.cc:359] Metadata directory not provided
I20260809 12:14:59.086311 20303 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260809 12:14:59.086581 20303 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260809 12:14:59.089462 20303 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskgH5Pjm/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786277382073938-888-0/minicluster-data/ts-2/data/instance:
uuid: "40aa5586957145c5902f46c34b2d1625"
format_stamp: "Formatted at 2026-08-09 12:14:59 on dist-test-slave-2w3w"
server_key: "4fb3d2b5c7042e1840459c0ca9a850b1"
server_key_iv: "6e2a192d4899086ba5a31678e3429558"
server_key_version: "encryptionkey@0"
I20260809 12:14:59.090197 20303 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskgH5Pjm/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786277382073938-888-0/minicluster-data/ts-2/wal/instance:
uuid: "40aa5586957145c5902f46c34b2d1625"
format_stamp: "Formatted at 2026-08-09 12:14:59 on dist-test-slave-2w3w"
server_key: "4fb3d2b5c7042e1840459c0ca9a850b1"
server_key_iv: "6e2a192d4899086ba5a31678e3429558"
server_key_version: "encryptionkey@0"
I20260809 12:14:59.095407 20303 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.003s sys 0.004s
I20260809 12:14:59.098698 20319 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260809 12:14:59.100080 20303 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260809 12:14:59.100257 20303 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskgH5Pjm/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786277382073938-888-0/minicluster-data/ts-2/data,/tmp/dist-test-taskgH5Pjm/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786277382073938-888-0/minicluster-data/ts-2/wal
uuid: "40aa5586957145c5902f46c34b2d1625"
format_stamp: "Formatted at 2026-08-09 12:14:59 on dist-test-slave-2w3w"
server_key: "4fb3d2b5c7042e1840459c0ca9a850b1"
server_key_iv: "6e2a192d4899086ba5a31678e3429558"
server_key_version: "encryptionkey@0"
I20260809 12:14:59.100481 20303 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskgH5Pjm/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786277382073938-888-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskgH5Pjm/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786277382073938-888-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskgH5Pjm/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786277382073938-888-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260809 12:14:59.117656 20303 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260809 12:14:59.118479 20303 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260809 12:14:59.118765 20303 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260809 12:14:59.119822 20303 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260809 12:14:59.121225 20303 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260809 12:14:59.121343 20303 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260809 12:14:59.121470 20303 ts_tablet_manager.cc:610] Registered 0 tablets
I20260809 12:14:59.121567 20303 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260809 12:14:59.155722 20303 rpc_server.cc:307] RPC server started. Bound to: 127.0.222.3:45649
I20260809 12:14:59.155809 20438 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.222.3:45649 every 8 connection(s)
I20260809 12:14:59.157047 20303 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskgH5Pjm/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786277382073938-888-0/minicluster-data/ts-2/data/info.pb
I20260809 12:14:59.163233 888 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskgH5Pjm/build/asan/bin/kudu as pid 20303
I20260809 12:14:59.163430 888 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskgH5Pjm/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786277382073938-888-0/minicluster-data/ts-2/wal/instance
I20260809 12:14:59.163988 888 external_mini_cluster.cc:1428] Setting key 6599f89fed2e04326a6fb62683827a9b
I20260809 12:14:59.169994 20162 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 12:14:59.159587 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.222.3:48359 (local address 127.0.222.62:37983)
0809 12:14:59.159815 (+ 228us) server_negotiation.cc:206] Beginning negotiation
0809 12:14:59.159824 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0809 12:14:59.161175 (+ 1351us) server_negotiation.cc:407] Connection header received
0809 12:14:59.162932 (+ 1757us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 12:14:59.162936 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 12:14:59.163065 (+ 129us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 12:14:59.163148 (+ 83us) server_negotiation.cc:226] Negotiated authn=SASL
0809 12:14:59.164601 (+ 1453us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 12:14:59.165543 (+ 942us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 12:14:59.167722 (+ 2179us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 12:14:59.168108 (+ 386us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 12:14:59.168202 (+ 94us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 12:14:59.169368 (+ 1166us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 12:14:59.169373 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 12:14:59.169376 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0809 12:14:59.169392 (+ 16us) server_negotiation.cc:922] calling sasl_server_start()
0809 12:14:59.169422 (+ 30us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0809 12:14:59.169434 (+ 12us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 12:14:59.169501 (+ 67us) server_negotiation.cc:1035] Waiting for connection context
0809 12:14:59.169668 (+ 167us) server_negotiation.cc:299] Negotiation successful
0809 12:14:59.169741 (+ 73us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":47}
I20260809 12:14:59.171043 20441 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 12:14:59.160144 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.222.62:37983 (local address 127.0.222.3:48359)
0809 12:14:59.161018 (+ 874us) negotiation.cc:107] Waiting for socket to connect
0809 12:14:59.161061 (+ 43us) client_negotiation.cc:174] Beginning negotiation
0809 12:14:59.162622 (+ 1561us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 12:14:59.163445 (+ 823us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 12:14:59.163459 (+ 14us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 12:14:59.163575 (+ 116us) client_negotiation.cc:189] Negotiated authn=SASL
0809 12:14:59.164393 (+ 818us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 12:14:59.164421 (+ 28us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 12:14:59.165710 (+ 1289us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 12:14:59.165714 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 12:14:59.167595 (+ 1881us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 12:14:59.167608 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 12:14:59.168290 (+ 682us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 12:14:59.168294 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 12:14:59.168484 (+ 190us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 12:14:59.169088 (+ 604us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0809 12:14:59.169090 (+ 2us) client_negotiation.cc:656] calling sasl_client_start()
0809 12:14:59.169126 (+ 36us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0809 12:14:59.169198 (+ 72us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0809 12:14:59.169500 (+ 302us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0809 12:14:59.169505 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0809 12:14:59.169510 (+ 5us) client_negotiation.cc:769] Sending connection context
0809 12:14:59.169638 (+ 128us) client_negotiation.cc:240] Negotiation successful
0809 12:14:59.169914 (+ 276us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":461,"thread_start_us":280,"threads_started":1}
I20260809 12:14:59.172363 20439 heartbeater.cc:347] Connected to a master server at 127.0.222.62:37983
I20260809 12:14:59.172628 20439 heartbeater.cc:464] Registering TS with master...
I20260809 12:14:59.173326 20439 heartbeater.cc:511] Master 127.0.222.62:37983 requested a full tablet report, sending...
I20260809 12:14:59.174649 19955 ts_manager.cc:194] Registered new tserver with Master: 40aa5586957145c5902f46c34b2d1625 (127.0.222.3:45649)
I20260809 12:14:59.175581 19955 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.222.3:48359
I20260809 12:14:59.179350 888 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260809 12:14:59.188974 20162 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 12:14:59.183533 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37942 (local address 127.0.222.62:37983)
0809 12:14:59.183696 (+ 163us) server_negotiation.cc:206] Beginning negotiation
0809 12:14:59.183700 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0809 12:14:59.184129 (+ 429us) server_negotiation.cc:407] Connection header received
0809 12:14:59.184280 (+ 151us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 12:14:59.184284 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 12:14:59.184356 (+ 72us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 12:14:59.184434 (+ 78us) server_negotiation.cc:226] Negotiated authn=SASL
0809 12:14:59.184884 (+ 450us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 12:14:59.185574 (+ 690us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 12:14:59.186875 (+ 1301us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 12:14:59.187342 (+ 467us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 12:14:59.187435 (+ 93us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 12:14:59.188402 (+ 967us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 12:14:59.188406 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 12:14:59.188409 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0809 12:14:59.188426 (+ 17us) server_negotiation.cc:922] calling sasl_server_start()
0809 12:14:59.188458 (+ 32us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0809 12:14:59.188470 (+ 12us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 12:14:59.188538 (+ 68us) server_negotiation.cc:1035] Waiting for connection context
0809 12:14:59.188660 (+ 122us) server_negotiation.cc:299] Negotiation successful
0809 12:14:59.188730 (+ 70us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":55}
I20260809 12:14:59.192606 19955 catalog_manager.cc:1908] Servicing CreateTable request from {username='slave'} at 127.0.0.1:37942:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260809 12:14:59.199676 19955 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260809 12:14:59.244105 20157 heartbeater.cc:503] Master 127.0.222.62:37983 was elected leader, sending a full tablet report...
I20260809 12:14:59.246017 20457 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 12:14:59.213918 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50050 (local address 127.0.222.1:45971)
0809 12:14:59.234400 (+ 20482us) server_negotiation.cc:206] Beginning negotiation
0809 12:14:59.234414 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0809 12:14:59.235481 (+ 1067us) server_negotiation.cc:407] Connection header received
0809 12:14:59.235797 (+ 316us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 12:14:59.235810 (+ 13us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 12:14:59.236168 (+ 358us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 12:14:59.236450 (+ 282us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0809 12:14:59.238238 (+ 1788us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 12:14:59.239800 (+ 1562us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 12:14:59.242624 (+ 2824us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 12:14:59.244201 (+ 1577us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 12:14:59.244337 (+ 136us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 12:14:59.244636 (+ 299us) server_negotiation.cc:1035] Waiting for connection context
0809 12:14:59.245467 (+ 831us) server_negotiation.cc:299] Negotiation successful
0809 12:14:59.245630 (+ 163us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":20232,"thread_start_us":20068,"threads_started":1}
I20260809 12:14:59.246025 20459 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 12:14:59.214425 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44054 (local address 127.0.222.2:37547)
0809 12:14:59.234647 (+ 20222us) server_negotiation.cc:206] Beginning negotiation
0809 12:14:59.234658 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0809 12:14:59.236202 (+ 1544us) server_negotiation.cc:407] Connection header received
0809 12:14:59.236411 (+ 209us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 12:14:59.236418 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 12:14:59.236667 (+ 249us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 12:14:59.236873 (+ 206us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0809 12:14:59.237886 (+ 1013us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 12:14:59.239578 (+ 1692us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 12:14:59.242728 (+ 3150us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 12:14:59.244189 (+ 1461us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 12:14:59.244333 (+ 144us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 12:14:59.244587 (+ 254us) server_negotiation.cc:1035] Waiting for connection context
0809 12:14:59.245461 (+ 874us) server_negotiation.cc:299] Negotiation successful
0809 12:14:59.245634 (+ 173us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":17785,"thread_start_us":17656,"threads_started":1}
I20260809 12:14:59.246942 20452 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 12:14:59.212607 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.222.3:45649 (local address 127.0.0.1:48086)
0809 12:14:59.228791 (+ 16184us) negotiation.cc:107] Waiting for socket to connect
0809 12:14:59.228831 (+ 40us) client_negotiation.cc:174] Beginning negotiation
0809 12:14:59.229052 (+ 221us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 12:14:59.238625 (+ 9573us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 12:14:59.238630 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 12:14:59.238656 (+ 26us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0809 12:14:59.239048 (+ 392us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 12:14:59.239066 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 12:14:59.240954 (+ 1888us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 12:14:59.240959 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 12:14:59.243117 (+ 2158us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 12:14:59.243128 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 12:14:59.244875 (+ 1747us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 12:14:59.244880 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 12:14:59.245067 (+ 187us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 12:14:59.245141 (+ 74us) client_negotiation.cc:769] Sending connection context
0809 12:14:59.245283 (+ 142us) client_negotiation.cc:240] Negotiation successful
0809 12:14:59.245514 (+ 231us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":15911,"mutex_wait_us":40,"thread_start_us":15744,"threads_started":1}
I20260809 12:14:59.247064 20453 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 12:14:59.212918 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48086 (local address 127.0.222.3:45649)
0809 12:14:59.238048 (+ 25130us) server_negotiation.cc:206] Beginning negotiation
0809 12:14:59.238057 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0809 12:14:59.238092 (+ 35us) server_negotiation.cc:407] Connection header received
0809 12:14:59.238190 (+ 98us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 12:14:59.238197 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 12:14:59.238405 (+ 208us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 12:14:59.238634 (+ 229us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0809 12:14:59.239242 (+ 608us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 12:14:59.240777 (+ 1535us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 12:14:59.243295 (+ 2518us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 12:14:59.244559 (+ 1264us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 12:14:59.244701 (+ 142us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 12:14:59.244967 (+ 266us) server_negotiation.cc:1035] Waiting for connection context
0809 12:14:59.246488 (+ 1521us) server_negotiation.cc:299] Negotiation successful
0809 12:14:59.246655 (+ 167us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":24859,"thread_start_us":24687,"threads_started":1}
I20260809 12:14:59.247027 20454 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 12:14:59.213031 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.222.1:45971 (local address 127.0.0.1:50050)
0809 12:14:59.235304 (+ 22273us) negotiation.cc:107] Waiting for socket to connect
0809 12:14:59.235331 (+ 27us) client_negotiation.cc:174] Beginning negotiation
0809 12:14:59.235527 (+ 196us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 12:14:59.236837 (+ 1310us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 12:14:59.236848 (+ 11us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 12:14:59.236902 (+ 54us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0809 12:14:59.238047 (+ 1145us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 12:14:59.238061 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 12:14:59.239935 (+ 1874us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 12:14:59.239943 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 12:14:59.242407 (+ 2464us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 12:14:59.242420 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 12:14:59.244751 (+ 2331us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 12:14:59.244756 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 12:14:59.245237 (+ 481us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 12:14:59.245277 (+ 40us) client_negotiation.cc:769] Sending connection context
0809 12:14:59.245351 (+ 74us) client_negotiation.cc:240] Negotiation successful
0809 12:14:59.246314 (+ 963us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":22063,"mutex_wait_us":1247,"thread_start_us":21840,"threads_started":1}
I20260809 12:14:59.247566 20455 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 12:14:59.213370 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.222.2:37547 (local address 127.0.0.1:44054)
0809 12:14:59.236067 (+ 22697us) negotiation.cc:107] Waiting for socket to connect
0809 12:14:59.236089 (+ 22us) client_negotiation.cc:174] Beginning negotiation
0809 12:14:59.236233 (+ 144us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 12:14:59.237146 (+ 913us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 12:14:59.237151 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 12:14:59.237178 (+ 27us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0809 12:14:59.237670 (+ 492us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 12:14:59.237685 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 12:14:59.239723 (+ 2038us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 12:14:59.239728 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 12:14:59.242526 (+ 2798us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 12:14:59.242547 (+ 21us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 12:14:59.244739 (+ 2192us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 12:14:59.244746 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 12:14:59.244999 (+ 253us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 12:14:59.245140 (+ 141us) client_negotiation.cc:769] Sending connection context
0809 12:14:59.245279 (+ 139us) client_negotiation.cc:240] Negotiation successful
0809 12:14:59.246350 (+ 1071us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":22578,"mutex_wait_us":1025,"thread_start_us":22433,"threads_started":1}
I20260809 12:14:59.250604 20087 tablet_service.cc:1467] Processing CreateTablet for tablet 419742dc15be4d27a58b0993b6730a3d (DEFAULT_TABLE table=test-table [id=957620529e6f464782b80785e091c3b7]), partition=RANGE (key) PARTITION UNBOUNDED
I20260809 12:14:59.251004 20230 tablet_service.cc:1467] Processing CreateTablet for tablet 419742dc15be4d27a58b0993b6730a3d (DEFAULT_TABLE table=test-table [id=957620529e6f464782b80785e091c3b7]), partition=RANGE (key) PARTITION UNBOUNDED
I20260809 12:14:59.251016 20368 tablet_service.cc:1467] Processing CreateTablet for tablet 419742dc15be4d27a58b0993b6730a3d (DEFAULT_TABLE table=test-table [id=957620529e6f464782b80785e091c3b7]), partition=RANGE (key) PARTITION UNBOUNDED
I20260809 12:14:59.251969 20087 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 419742dc15be4d27a58b0993b6730a3d. 1 dirs total, 0 dirs full, 0 dirs failed
I20260809 12:14:59.252379 20230 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 419742dc15be4d27a58b0993b6730a3d. 1 dirs total, 0 dirs full, 0 dirs failed
I20260809 12:14:59.252419 20368 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 419742dc15be4d27a58b0993b6730a3d. 1 dirs total, 0 dirs full, 0 dirs failed
I20260809 12:14:59.263847 20460 tablet_bootstrap.cc:492] T 419742dc15be4d27a58b0993b6730a3d P 94d4c13dd4e246a38799a6f0ef2947f8: Bootstrap starting.
I20260809 12:14:59.267132 20461 tablet_bootstrap.cc:492] T 419742dc15be4d27a58b0993b6730a3d P 3ea81fb4f9564a12b0b50550947cfb60: Bootstrap starting.
I20260809 12:14:59.267601 20460 tablet_bootstrap.cc:654] T 419742dc15be4d27a58b0993b6730a3d P 94d4c13dd4e246a38799a6f0ef2947f8: Neither blocks nor log segments found. Creating new log.
I20260809 12:14:59.269408 20460 log.cc:824] T 419742dc15be4d27a58b0993b6730a3d P 94d4c13dd4e246a38799a6f0ef2947f8: Log is configured to *not* fsync() on all Append() calls
I20260809 12:14:59.270838 20461 tablet_bootstrap.cc:654] T 419742dc15be4d27a58b0993b6730a3d P 3ea81fb4f9564a12b0b50550947cfb60: Neither blocks nor log segments found. Creating new log.
I20260809 12:14:59.270895 20462 tablet_bootstrap.cc:492] T 419742dc15be4d27a58b0993b6730a3d P 40aa5586957145c5902f46c34b2d1625: Bootstrap starting.
I20260809 12:14:59.272078 20461 log.cc:824] T 419742dc15be4d27a58b0993b6730a3d P 3ea81fb4f9564a12b0b50550947cfb60: Log is configured to *not* fsync() on all Append() calls
I20260809 12:14:59.272912 20460 tablet_bootstrap.cc:492] T 419742dc15be4d27a58b0993b6730a3d P 94d4c13dd4e246a38799a6f0ef2947f8: No bootstrap required, opened a new log
I20260809 12:14:59.273178 20460 ts_tablet_manager.cc:1397] T 419742dc15be4d27a58b0993b6730a3d P 94d4c13dd4e246a38799a6f0ef2947f8: Time spent bootstrapping tablet: real 0.010s user 0.007s sys 0.000s
I20260809 12:14:59.274719 20462 tablet_bootstrap.cc:654] T 419742dc15be4d27a58b0993b6730a3d P 40aa5586957145c5902f46c34b2d1625: Neither blocks nor log segments found. Creating new log.
I20260809 12:14:59.276115 20462 log.cc:824] T 419742dc15be4d27a58b0993b6730a3d P 40aa5586957145c5902f46c34b2d1625: Log is configured to *not* fsync() on all Append() calls
I20260809 12:14:59.276722 20461 tablet_bootstrap.cc:492] T 419742dc15be4d27a58b0993b6730a3d P 3ea81fb4f9564a12b0b50550947cfb60: No bootstrap required, opened a new log
I20260809 12:14:59.277153 20461 ts_tablet_manager.cc:1397] T 419742dc15be4d27a58b0993b6730a3d P 3ea81fb4f9564a12b0b50550947cfb60: Time spent bootstrapping tablet: real 0.010s user 0.005s sys 0.004s
I20260809 12:14:59.279486 20462 tablet_bootstrap.cc:492] T 419742dc15be4d27a58b0993b6730a3d P 40aa5586957145c5902f46c34b2d1625: No bootstrap required, opened a new log
I20260809 12:14:59.279764 20462 ts_tablet_manager.cc:1397] T 419742dc15be4d27a58b0993b6730a3d P 40aa5586957145c5902f46c34b2d1625: Time spent bootstrapping tablet: real 0.009s user 0.003s sys 0.004s
I20260809 12:14:59.284636 20460 raft_consensus.cc:348] T 419742dc15be4d27a58b0993b6730a3d P 94d4c13dd4e246a38799a6f0ef2947f8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "40aa5586957145c5902f46c34b2d1625" member_type: VOTER last_known_addr { host: "127.0.222.3" port: 45649 } } peers { permanent_uuid: "94d4c13dd4e246a38799a6f0ef2947f8" member_type: VOTER last_known_addr { host: "127.0.222.1" port: 45971 } } peers { permanent_uuid: "3ea81fb4f9564a12b0b50550947cfb60" member_type: VOTER last_known_addr { host: "127.0.222.2" port: 37547 } }
I20260809 12:14:59.285090 20460 raft_consensus.cc:374] T 419742dc15be4d27a58b0993b6730a3d P 94d4c13dd4e246a38799a6f0ef2947f8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260809 12:14:59.285188 20460 raft_consensus.cc:729] T 419742dc15be4d27a58b0993b6730a3d P 94d4c13dd4e246a38799a6f0ef2947f8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 94d4c13dd4e246a38799a6f0ef2947f8, State: Initialized, Role: FOLLOWER
I20260809 12:14:59.285835 20460 consensus_queue.cc:260] T 419742dc15be4d27a58b0993b6730a3d P 94d4c13dd4e246a38799a6f0ef2947f8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "40aa5586957145c5902f46c34b2d1625" member_type: VOTER last_known_addr { host: "127.0.222.3" port: 45649 } } peers { permanent_uuid: "94d4c13dd4e246a38799a6f0ef2947f8" member_type: VOTER last_known_addr { host: "127.0.222.1" port: 45971 } } peers { permanent_uuid: "3ea81fb4f9564a12b0b50550947cfb60" member_type: VOTER last_known_addr { host: "127.0.222.2" port: 37547 } }
I20260809 12:14:59.287719 20460 ts_tablet_manager.cc:1428] T 419742dc15be4d27a58b0993b6730a3d P 94d4c13dd4e246a38799a6f0ef2947f8: Time spent starting tablet: real 0.014s user 0.015s sys 0.000s
I20260809 12:14:59.287626 20461 raft_consensus.cc:348] T 419742dc15be4d27a58b0993b6730a3d P 3ea81fb4f9564a12b0b50550947cfb60 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "40aa5586957145c5902f46c34b2d1625" member_type: VOTER last_known_addr { host: "127.0.222.3" port: 45649 } } peers { permanent_uuid: "94d4c13dd4e246a38799a6f0ef2947f8" member_type: VOTER last_known_addr { host: "127.0.222.1" port: 45971 } } peers { permanent_uuid: "3ea81fb4f9564a12b0b50550947cfb60" member_type: VOTER last_known_addr { host: "127.0.222.2" port: 37547 } }
I20260809 12:14:59.287760 20462 raft_consensus.cc:348] T 419742dc15be4d27a58b0993b6730a3d P 40aa5586957145c5902f46c34b2d1625 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "40aa5586957145c5902f46c34b2d1625" member_type: VOTER last_known_addr { host: "127.0.222.3" port: 45649 } } peers { permanent_uuid: "94d4c13dd4e246a38799a6f0ef2947f8" member_type: VOTER last_known_addr { host: "127.0.222.1" port: 45971 } } peers { permanent_uuid: "3ea81fb4f9564a12b0b50550947cfb60" member_type: VOTER last_known_addr { host: "127.0.222.2" port: 37547 } }
I20260809 12:14:59.288074 20461 raft_consensus.cc:374] T 419742dc15be4d27a58b0993b6730a3d P 3ea81fb4f9564a12b0b50550947cfb60 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260809 12:14:59.288203 20461 raft_consensus.cc:729] T 419742dc15be4d27a58b0993b6730a3d P 3ea81fb4f9564a12b0b50550947cfb60 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3ea81fb4f9564a12b0b50550947cfb60, State: Initialized, Role: FOLLOWER
I20260809 12:14:59.288182 20462 raft_consensus.cc:374] T 419742dc15be4d27a58b0993b6730a3d P 40aa5586957145c5902f46c34b2d1625 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260809 12:14:59.288313 20462 raft_consensus.cc:729] T 419742dc15be4d27a58b0993b6730a3d P 40aa5586957145c5902f46c34b2d1625 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 40aa5586957145c5902f46c34b2d1625, State: Initialized, Role: FOLLOWER
I20260809 12:14:59.288873 20462 consensus_queue.cc:260] T 419742dc15be4d27a58b0993b6730a3d P 40aa5586957145c5902f46c34b2d1625 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "40aa5586957145c5902f46c34b2d1625" member_type: VOTER last_known_addr { host: "127.0.222.3" port: 45649 } } peers { permanent_uuid: "94d4c13dd4e246a38799a6f0ef2947f8" member_type: VOTER last_known_addr { host: "127.0.222.1" port: 45971 } } peers { permanent_uuid: "3ea81fb4f9564a12b0b50550947cfb60" member_type: VOTER last_known_addr { host: "127.0.222.2" port: 37547 } }
I20260809 12:14:59.288810 20461 consensus_queue.cc:260] T 419742dc15be4d27a58b0993b6730a3d P 3ea81fb4f9564a12b0b50550947cfb60 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "40aa5586957145c5902f46c34b2d1625" member_type: VOTER last_known_addr { host: "127.0.222.3" port: 45649 } } peers { permanent_uuid: "94d4c13dd4e246a38799a6f0ef2947f8" member_type: VOTER last_known_addr { host: "127.0.222.1" port: 45971 } } peers { permanent_uuid: "3ea81fb4f9564a12b0b50550947cfb60" member_type: VOTER last_known_addr { host: "127.0.222.2" port: 37547 } }
I20260809 12:14:59.289952 20439 heartbeater.cc:503] Master 127.0.222.62:37983 was elected leader, sending a full tablet report...
I20260809 12:14:59.290617 20462 ts_tablet_manager.cc:1428] T 419742dc15be4d27a58b0993b6730a3d P 40aa5586957145c5902f46c34b2d1625: Time spent starting tablet: real 0.011s user 0.008s sys 0.003s
I20260809 12:14:59.291623 20300 heartbeater.cc:503] Master 127.0.222.62:37983 was elected leader, sending a full tablet report...
I20260809 12:14:59.292228 20461 ts_tablet_manager.cc:1428] T 419742dc15be4d27a58b0993b6730a3d P 3ea81fb4f9564a12b0b50550947cfb60: Time spent starting tablet: real 0.015s user 0.007s sys 0.006s
W20260809 12:14:59.408738 20440 tablet.cc:2369] T 419742dc15be4d27a58b0993b6730a3d P 40aa5586957145c5902f46c34b2d1625: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260809 12:14:59.444219 20301 tablet.cc:2369] T 419742dc15be4d27a58b0993b6730a3d P 3ea81fb4f9564a12b0b50550947cfb60: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260809 12:14:59.471761 20159 tablet.cc:2369] T 419742dc15be4d27a58b0993b6730a3d P 94d4c13dd4e246a38799a6f0ef2947f8: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260809 12:14:59.526567 20469 raft_consensus.cc:482] T 419742dc15be4d27a58b0993b6730a3d P 3ea81fb4f9564a12b0b50550947cfb60 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260809 12:14:59.526891 20469 raft_consensus.cc:504] T 419742dc15be4d27a58b0993b6730a3d P 3ea81fb4f9564a12b0b50550947cfb60 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "40aa5586957145c5902f46c34b2d1625" member_type: VOTER last_known_addr { host: "127.0.222.3" port: 45649 } } peers { permanent_uuid: "94d4c13dd4e246a38799a6f0ef2947f8" member_type: VOTER last_known_addr { host: "127.0.222.1" port: 45971 } } peers { permanent_uuid: "3ea81fb4f9564a12b0b50550947cfb60" member_type: VOTER last_known_addr { host: "127.0.222.2" port: 37547 } }
I20260809 12:14:59.528697 20469 leader_election.cc:288] T 419742dc15be4d27a58b0993b6730a3d P 3ea81fb4f9564a12b0b50550947cfb60 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 40aa5586957145c5902f46c34b2d1625 (127.0.222.3:45649), 94d4c13dd4e246a38799a6f0ef2947f8 (127.0.222.1:45971)
I20260809 12:14:59.535795 20453 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 12:14:59.529156 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.222.2:45309 (local address 127.0.222.3:45649)
0809 12:14:59.529392 (+ 236us) server_negotiation.cc:206] Beginning negotiation
0809 12:14:59.529403 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0809 12:14:59.530346 (+ 943us) server_negotiation.cc:407] Connection header received
0809 12:14:59.530532 (+ 186us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 12:14:59.530537 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 12:14:59.530597 (+ 60us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 12:14:59.530698 (+ 101us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0809 12:14:59.531402 (+ 704us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 12:14:59.532092 (+ 690us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 12:14:59.534058 (+ 1966us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 12:14:59.534850 (+ 792us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 12:14:59.535014 (+ 164us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 12:14:59.535139 (+ 125us) server_negotiation.cc:1035] Waiting for connection context
0809 12:14:59.535431 (+ 292us) server_negotiation.cc:299] Negotiation successful
0809 12:14:59.535555 (+ 124us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":72}
I20260809 12:14:59.535833 20457 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 12:14:59.529282 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.222.2:49193 (local address 127.0.222.1:45971)
0809 12:14:59.529631 (+ 349us) server_negotiation.cc:206] Beginning negotiation
0809 12:14:59.529640 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0809 12:14:59.530225 (+ 585us) server_negotiation.cc:407] Connection header received
0809 12:14:59.530408 (+ 183us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 12:14:59.530413 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 12:14:59.530488 (+ 75us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 12:14:59.530591 (+ 103us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0809 12:14:59.531257 (+ 666us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 12:14:59.531992 (+ 735us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 12:14:59.534065 (+ 2073us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 12:14:59.534851 (+ 786us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 12:14:59.535023 (+ 172us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 12:14:59.535156 (+ 133us) server_negotiation.cc:1035] Waiting for connection context
0809 12:14:59.535465 (+ 309us) server_negotiation.cc:299] Negotiation successful
0809 12:14:59.535594 (+ 129us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":157}
I20260809 12:14:59.535915 20470 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 12:14:59.529094 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.222.3:45649 (local address 127.0.222.2:45309)
0809 12:14:59.530206 (+ 1112us) negotiation.cc:107] Waiting for socket to connect
0809 12:14:59.530225 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0809 12:14:59.530370 (+ 145us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 12:14:59.530723 (+ 353us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 12:14:59.530727 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 12:14:59.530765 (+ 38us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0809 12:14:59.531209 (+ 444us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 12:14:59.531217 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 12:14:59.532240 (+ 1023us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 12:14:59.532246 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 12:14:59.533834 (+ 1588us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 12:14:59.533847 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 12:14:59.535101 (+ 1254us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 12:14:59.535108 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 12:14:59.535270 (+ 162us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 12:14:59.535294 (+ 24us) client_negotiation.cc:769] Sending connection context
0809 12:14:59.535402 (+ 108us) client_negotiation.cc:240] Negotiation successful
0809 12:14:59.535530 (+ 128us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":980,"mutex_wait_us":12,"thread_start_us":532,"threads_started":1}
I20260809 12:14:59.536394 20471 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 12:14:59.528967 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.222.1:45971 (local address 127.0.222.2:49193)
0809 12:14:59.530091 (+ 1124us) negotiation.cc:107] Waiting for socket to connect
0809 12:14:59.530118 (+ 27us) client_negotiation.cc:174] Beginning negotiation
0809 12:14:59.530246 (+ 128us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 12:14:59.530608 (+ 362us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 12:14:59.530612 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 12:14:59.530637 (+ 25us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0809 12:14:59.531103 (+ 466us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 12:14:59.531116 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 12:14:59.532122 (+ 1006us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 12:14:59.532130 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 12:14:59.533840 (+ 1710us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 12:14:59.533853 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 12:14:59.535144 (+ 1291us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 12:14:59.535151 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 12:14:59.535312 (+ 161us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 12:14:59.535334 (+ 22us) client_negotiation.cc:769] Sending connection context
0809 12:14:59.535438 (+ 104us) client_negotiation.cc:240] Negotiation successful
0809 12:14:59.535567 (+ 129us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":986,"mutex_wait_us":99,"thread_start_us":927,"threads_started":1}
I20260809 12:14:59.536983 20390 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "419742dc15be4d27a58b0993b6730a3d" candidate_uuid: "3ea81fb4f9564a12b0b50550947cfb60" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "40aa5586957145c5902f46c34b2d1625" is_pre_election: true
I20260809 12:14:59.537370 20108 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "419742dc15be4d27a58b0993b6730a3d" candidate_uuid: "3ea81fb4f9564a12b0b50550947cfb60" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "94d4c13dd4e246a38799a6f0ef2947f8" is_pre_election: true
I20260809 12:14:59.537580 20390 raft_consensus.cc:2454] T 419742dc15be4d27a58b0993b6730a3d P 40aa5586957145c5902f46c34b2d1625 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3ea81fb4f9564a12b0b50550947cfb60 in term 0.
I20260809 12:14:59.537890 20108 raft_consensus.cc:2454] T 419742dc15be4d27a58b0993b6730a3d P 94d4c13dd4e246a38799a6f0ef2947f8 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3ea81fb4f9564a12b0b50550947cfb60 in term 0.
I20260809 12:14:59.538308 20184 leader_election.cc:302] T 419742dc15be4d27a58b0993b6730a3d P 3ea81fb4f9564a12b0b50550947cfb60 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3ea81fb4f9564a12b0b50550947cfb60, 40aa5586957145c5902f46c34b2d1625; no voters:
I20260809 12:14:59.538729 20469 raft_consensus.cc:2781] T 419742dc15be4d27a58b0993b6730a3d P 3ea81fb4f9564a12b0b50550947cfb60 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260809 12:14:59.538911 20469 raft_consensus.cc:482] T 419742dc15be4d27a58b0993b6730a3d P 3ea81fb4f9564a12b0b50550947cfb60 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260809 12:14:59.539048 20469 raft_consensus.cc:3037] T 419742dc15be4d27a58b0993b6730a3d P 3ea81fb4f9564a12b0b50550947cfb60 [term 0 FOLLOWER]: Advancing to term 1
I20260809 12:14:59.541227 20469 raft_consensus.cc:504] T 419742dc15be4d27a58b0993b6730a3d P 3ea81fb4f9564a12b0b50550947cfb60 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "40aa5586957145c5902f46c34b2d1625" member_type: VOTER last_known_addr { host: "127.0.222.3" port: 45649 } } peers { permanent_uuid: "94d4c13dd4e246a38799a6f0ef2947f8" member_type: VOTER last_known_addr { host: "127.0.222.1" port: 45971 } } peers { permanent_uuid: "3ea81fb4f9564a12b0b50550947cfb60" member_type: VOTER last_known_addr { host: "127.0.222.2" port: 37547 } }
I20260809 12:14:59.541821 20469 leader_election.cc:288] T 419742dc15be4d27a58b0993b6730a3d P 3ea81fb4f9564a12b0b50550947cfb60 [CANDIDATE]: Term 1 election: Requested vote from peers 40aa5586957145c5902f46c34b2d1625 (127.0.222.3:45649), 94d4c13dd4e246a38799a6f0ef2947f8 (127.0.222.1:45971)
I20260809 12:14:59.542300 20108 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "419742dc15be4d27a58b0993b6730a3d" candidate_uuid: "3ea81fb4f9564a12b0b50550947cfb60" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "94d4c13dd4e246a38799a6f0ef2947f8"
I20260809 12:14:59.542467 20108 raft_consensus.cc:3037] T 419742dc15be4d27a58b0993b6730a3d P 94d4c13dd4e246a38799a6f0ef2947f8 [term 0 FOLLOWER]: Advancing to term 1
I20260809 12:14:59.544749 20108 raft_consensus.cc:2454] T 419742dc15be4d27a58b0993b6730a3d P 94d4c13dd4e246a38799a6f0ef2947f8 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3ea81fb4f9564a12b0b50550947cfb60 in term 1.
I20260809 12:14:59.545153 20182 leader_election.cc:302] T 419742dc15be4d27a58b0993b6730a3d P 3ea81fb4f9564a12b0b50550947cfb60 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3ea81fb4f9564a12b0b50550947cfb60, 94d4c13dd4e246a38799a6f0ef2947f8; no voters:
I20260809 12:14:59.545154 20390 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "419742dc15be4d27a58b0993b6730a3d" candidate_uuid: "3ea81fb4f9564a12b0b50550947cfb60" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "40aa5586957145c5902f46c34b2d1625"
I20260809 12:14:59.545362 20390 raft_consensus.cc:3037] T 419742dc15be4d27a58b0993b6730a3d P 40aa5586957145c5902f46c34b2d1625 [term 0 FOLLOWER]: Advancing to term 1
I20260809 12:14:59.545451 20469 raft_consensus.cc:2781] T 419742dc15be4d27a58b0993b6730a3d P 3ea81fb4f9564a12b0b50550947cfb60 [term 1 FOLLOWER]: Leader election won for term 1
I20260809 12:14:59.546006 20469 raft_consensus.cc:686] T 419742dc15be4d27a58b0993b6730a3d P 3ea81fb4f9564a12b0b50550947cfb60 [term 1 LEADER]: Becoming Leader. State: Replica: 3ea81fb4f9564a12b0b50550947cfb60, State: Running, Role: LEADER
I20260809 12:14:59.546568 20469 consensus_queue.cc:237] T 419742dc15be4d27a58b0993b6730a3d P 3ea81fb4f9564a12b0b50550947cfb60 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "40aa5586957145c5902f46c34b2d1625" member_type: VOTER last_known_addr { host: "127.0.222.3" port: 45649 } } peers { permanent_uuid: "94d4c13dd4e246a38799a6f0ef2947f8" member_type: VOTER last_known_addr { host: "127.0.222.1" port: 45971 } } peers { permanent_uuid: "3ea81fb4f9564a12b0b50550947cfb60" member_type: VOTER last_known_addr { host: "127.0.222.2" port: 37547 } }
I20260809 12:14:59.547845 20390 raft_consensus.cc:2454] T 419742dc15be4d27a58b0993b6730a3d P 40aa5586957145c5902f46c34b2d1625 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3ea81fb4f9564a12b0b50550947cfb60 in term 1.
I20260809 12:14:59.551844 19955 catalog_manager.cc:5495] T 419742dc15be4d27a58b0993b6730a3d P 3ea81fb4f9564a12b0b50550947cfb60 reported cstate change: term changed from 0 to 1, leader changed from <none> to 3ea81fb4f9564a12b0b50550947cfb60 (127.0.222.2). New cstate: current_term: 1 leader_uuid: "3ea81fb4f9564a12b0b50550947cfb60" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "40aa5586957145c5902f46c34b2d1625" member_type: VOTER last_known_addr { host: "127.0.222.3" port: 45649 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "94d4c13dd4e246a38799a6f0ef2947f8" member_type: VOTER last_known_addr { host: "127.0.222.1" port: 45971 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "3ea81fb4f9564a12b0b50550947cfb60" member_type: VOTER last_known_addr { host: "127.0.222.2" port: 37547 } health_report { overall_health: HEALTHY } } }
I20260809 12:14:59.607671 20459 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 12:14:59.601889 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44060 (local address 127.0.222.2:37547)
0809 12:14:59.602133 (+ 244us) server_negotiation.cc:206] Beginning negotiation
0809 12:14:59.602139 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0809 12:14:59.602238 (+ 99us) server_negotiation.cc:407] Connection header received
0809 12:14:59.602493 (+ 255us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 12:14:59.602497 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 12:14:59.602597 (+ 100us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 12:14:59.602755 (+ 158us) server_negotiation.cc:226] Negotiated authn=TOKEN
0809 12:14:59.603222 (+ 467us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 12:14:59.604009 (+ 787us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 12:14:59.605514 (+ 1505us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 12:14:59.605938 (+ 424us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 12:14:59.606079 (+ 141us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 12:14:59.606573 (+ 494us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0809 12:14:59.607093 (+ 520us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0809 12:14:59.607265 (+ 172us) server_negotiation.cc:1035] Waiting for connection context
0809 12:14:59.607318 (+ 53us) server_negotiation.cc:299] Negotiation successful
0809 12:14:59.607395 (+ 77us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":85}
I20260809 12:14:59.615841 20108 raft_consensus.cc:1261] T 419742dc15be4d27a58b0993b6730a3d P 94d4c13dd4e246a38799a6f0ef2947f8 [term 1 FOLLOWER]: Refusing update from remote peer 3ea81fb4f9564a12b0b50550947cfb60: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260809 12:14:59.615860 20390 raft_consensus.cc:1261] T 419742dc15be4d27a58b0993b6730a3d P 40aa5586957145c5902f46c34b2d1625 [term 1 FOLLOWER]: Refusing update from remote peer 3ea81fb4f9564a12b0b50550947cfb60: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260809 12:14:59.616837 20469 consensus_queue.cc:1035] T 419742dc15be4d27a58b0993b6730a3d P 3ea81fb4f9564a12b0b50550947cfb60 [LEADER]: Connected to new peer: Peer: permanent_uuid: "40aa5586957145c5902f46c34b2d1625" member_type: VOTER last_known_addr { host: "127.0.222.3" port: 45649 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260809 12:14:59.617154 20472 consensus_queue.cc:1035] T 419742dc15be4d27a58b0993b6730a3d P 3ea81fb4f9564a12b0b50550947cfb60 [LEADER]: Connected to new peer: Peer: permanent_uuid: "94d4c13dd4e246a38799a6f0ef2947f8" member_type: VOTER last_known_addr { host: "127.0.222.1" port: 45971 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260809 12:14:59.635095 20475 mvcc.cc:205] Tried to move back new op lower bound from 7316593457616445440 to 7316593457347358720. Current Snapshot: MvccSnapshot[applied={T|T < 7316593457616445440}]
I20260809 12:14:59.650667 19953 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:37942:
table { table_name: "test-table" } modify_external_catalogs: true
I20260809 12:14:59.651054 19953 catalog_manager.cc:2645] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:37942:
table { table_name: "test-table" } modify_external_catalogs: true
I20260809 12:14:59.655470 19953 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 1ba08d3a915e4f35b5c89ed2df131ff8: Sending DeleteTablet for 3 replicas of tablet 419742dc15be4d27a58b0993b6730a3d
I20260809 12:14:59.656715 20368 tablet_service.cc:1514] Processing DeleteTablet for tablet 419742dc15be4d27a58b0993b6730a3d with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-09 12:14:59 UTC) from {username='slave'} at 127.0.0.1:48086
I20260809 12:14:59.656715 20087 tablet_service.cc:1514] Processing DeleteTablet for tablet 419742dc15be4d27a58b0993b6730a3d with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-09 12:14:59 UTC) from {username='slave'} at 127.0.0.1:50050
I20260809 12:14:59.658089 20230 tablet_service.cc:1514] Processing DeleteTablet for tablet 419742dc15be4d27a58b0993b6730a3d with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-09 12:14:59 UTC) from {username='slave'} at 127.0.0.1:44054
I20260809 12:14:59.658339 20484 tablet_replica.cc:330] T 419742dc15be4d27a58b0993b6730a3d P 40aa5586957145c5902f46c34b2d1625: stopping tablet replica
I20260809 12:14:59.658809 20484 raft_consensus.cc:2229] T 419742dc15be4d27a58b0993b6730a3d P 40aa5586957145c5902f46c34b2d1625 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260809 12:14:59.659540 20485 tablet_replica.cc:330] T 419742dc15be4d27a58b0993b6730a3d P 94d4c13dd4e246a38799a6f0ef2947f8: stopping tablet replica
I20260809 12:14:59.659706 888 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskgH5Pjm/build/asan/bin/kudu pid 20022
I20260809 12:14:59.659971 20485 raft_consensus.cc:2229] T 419742dc15be4d27a58b0993b6730a3d P 94d4c13dd4e246a38799a6f0ef2947f8 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260809 12:14:59.660200 20486 tablet_replica.cc:330] T 419742dc15be4d27a58b0993b6730a3d P 3ea81fb4f9564a12b0b50550947cfb60: stopping tablet replica
I20260809 12:14:59.660480 20485 raft_consensus.cc:2258] T 419742dc15be4d27a58b0993b6730a3d P 94d4c13dd4e246a38799a6f0ef2947f8 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260809 12:14:59.660775 20486 raft_consensus.cc:2229] T 419742dc15be4d27a58b0993b6730a3d P 3ea81fb4f9564a12b0b50550947cfb60 [term 1 LEADER]: Raft consensus shutting down.
I20260809 12:14:59.661168 20484 raft_consensus.cc:2258] T 419742dc15be4d27a58b0993b6730a3d P 40aa5586957145c5902f46c34b2d1625 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260809 12:14:59.661542 20486 raft_consensus.cc:2258] T 419742dc15be4d27a58b0993b6730a3d P 3ea81fb4f9564a12b0b50550947cfb60 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260809 12:14:59.663412 20485 ts_tablet_manager.cc:1905] T 419742dc15be4d27a58b0993b6730a3d P 94d4c13dd4e246a38799a6f0ef2947f8: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260809 12:14:59.663842 20484 ts_tablet_manager.cc:1905] T 419742dc15be4d27a58b0993b6730a3d P 40aa5586957145c5902f46c34b2d1625: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260809 12:14:59.666639 20486 ts_tablet_manager.cc:1905] T 419742dc15be4d27a58b0993b6730a3d P 3ea81fb4f9564a12b0b50550947cfb60: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260809 12:14:59.670879 20485 ts_tablet_manager.cc:1918] T 419742dc15be4d27a58b0993b6730a3d P 94d4c13dd4e246a38799a6f0ef2947f8: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260809 12:14:59.671092 20485 log.cc:1196] T 419742dc15be4d27a58b0993b6730a3d P 94d4c13dd4e246a38799a6f0ef2947f8: Deleting WAL directory at /tmp/dist-test-taskgH5Pjm/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786277382073938-888-0/minicluster-data/ts-0/wal/wals/419742dc15be4d27a58b0993b6730a3d
I20260809 12:14:59.671931 20485 ts_tablet_manager.cc:1939] T 419742dc15be4d27a58b0993b6730a3d P 94d4c13dd4e246a38799a6f0ef2947f8: Deleting consensus metadata
I20260809 12:14:59.674199 20486 ts_tablet_manager.cc:1918] T 419742dc15be4d27a58b0993b6730a3d P 3ea81fb4f9564a12b0b50550947cfb60: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260809 12:14:59.674360 20486 log.cc:1196] T 419742dc15be4d27a58b0993b6730a3d P 3ea81fb4f9564a12b0b50550947cfb60: Deleting WAL directory at /tmp/dist-test-taskgH5Pjm/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786277382073938-888-0/minicluster-data/ts-1/wal/wals/419742dc15be4d27a58b0993b6730a3d
I20260809 12:14:59.674930 20486 ts_tablet_manager.cc:1939] T 419742dc15be4d27a58b0993b6730a3d P 3ea81fb4f9564a12b0b50550947cfb60: Deleting consensus metadata
I20260809 12:14:59.675122 19938 catalog_manager.cc:4841] TS 94d4c13dd4e246a38799a6f0ef2947f8 (127.0.222.1:45971): tablet 419742dc15be4d27a58b0993b6730a3d (table test-table [id=957620529e6f464782b80785e091c3b7]) successfully deleted
I20260809 12:14:59.676680 19940 catalog_manager.cc:4841] TS 3ea81fb4f9564a12b0b50550947cfb60 (127.0.222.2:37547): tablet 419742dc15be4d27a58b0993b6730a3d (table test-table [id=957620529e6f464782b80785e091c3b7]) successfully deleted
I20260809 12:14:59.679415 20484 ts_tablet_manager.cc:1918] T 419742dc15be4d27a58b0993b6730a3d P 40aa5586957145c5902f46c34b2d1625: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260809 12:14:59.679600 20484 log.cc:1196] T 419742dc15be4d27a58b0993b6730a3d P 40aa5586957145c5902f46c34b2d1625: Deleting WAL directory at /tmp/dist-test-taskgH5Pjm/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786277382073938-888-0/minicluster-data/ts-2/wal/wals/419742dc15be4d27a58b0993b6730a3d
I20260809 12:14:59.680204 20484 ts_tablet_manager.cc:1939] T 419742dc15be4d27a58b0993b6730a3d P 40aa5586957145c5902f46c34b2d1625: Deleting consensus metadata
I20260809 12:14:59.681792 20457 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 12:14:59.661238 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50062 (local address 127.0.222.1:45971)
0809 12:14:59.661452 (+ 214us) server_negotiation.cc:206] Beginning negotiation
0809 12:14:59.661458 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0809 12:14:59.671641 (+ 10183us) server_negotiation.cc:407] Connection header received
0809 12:14:59.671974 (+ 333us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 12:14:59.671980 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 12:14:59.672091 (+ 111us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 12:14:59.672185 (+ 94us) server_negotiation.cc:226] Negotiated authn=SASL
0809 12:14:59.673111 (+ 926us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 12:14:59.674447 (+ 1336us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 12:14:59.677723 (+ 3276us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 12:14:59.678370 (+ 647us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 12:14:59.678517 (+ 147us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 12:14:59.680002 (+ 1485us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 12:14:59.680015 (+ 13us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 12:14:59.680034 (+ 19us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0809 12:14:59.680915 (+ 881us) server_negotiation.cc:922] calling sasl_server_start()
0809 12:14:59.680977 (+ 62us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0809 12:14:59.681003 (+ 26us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 12:14:59.681124 (+ 121us) server_negotiation.cc:1035] Waiting for connection context
0809 12:14:59.681305 (+ 181us) server_negotiation.cc:299] Negotiation successful
0809 12:14:59.681424 (+ 119us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":68}
I20260809 12:14:59.682112 19941 catalog_manager.cc:4841] TS 40aa5586957145c5902f46c34b2d1625 (127.0.222.3:45649): tablet 419742dc15be4d27a58b0993b6730a3d (table test-table [id=957620529e6f464782b80785e091c3b7]) successfully deleted
I20260809 12:14:59.682502 20151 generic_service.cc:196] Checking for leaks (request via RPC)
I20260809 12:14:59.791333 888 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskgH5Pjm/build/asan/bin/kudu with pid 20022
I20260809 12:14:59.813483 888 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskgH5Pjm/build/asan/bin/kudu pid 20164
I20260809 12:14:59.821444 20459 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 12:14:59.814353 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44072 (local address 127.0.222.2:37547)
0809 12:14:59.814547 (+ 194us) server_negotiation.cc:206] Beginning negotiation
0809 12:14:59.814551 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0809 12:14:59.814570 (+ 19us) server_negotiation.cc:407] Connection header received
0809 12:14:59.814770 (+ 200us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 12:14:59.814774 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 12:14:59.814856 (+ 82us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 12:14:59.814944 (+ 88us) server_negotiation.cc:226] Negotiated authn=SASL
0809 12:14:59.815470 (+ 526us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 12:14:59.816276 (+ 806us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 12:14:59.818187 (+ 1911us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 12:14:59.818630 (+ 443us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 12:14:59.818727 (+ 97us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 12:14:59.819951 (+ 1224us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 12:14:59.819959 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 12:14:59.819975 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0809 12:14:59.820613 (+ 638us) server_negotiation.cc:922] calling sasl_server_start()
0809 12:14:59.820667 (+ 54us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0809 12:14:59.820686 (+ 19us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 12:14:59.820780 (+ 94us) server_negotiation.cc:1035] Waiting for connection context
0809 12:14:59.820948 (+ 168us) server_negotiation.cc:299] Negotiation successful
0809 12:14:59.821029 (+ 81us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":68}
I20260809 12:14:59.822207 20295 generic_service.cc:196] Checking for leaks (request via RPC)
I20260809 12:14:59.936208 888 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskgH5Pjm/build/asan/bin/kudu with pid 20164
I20260809 12:14:59.960160 888 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskgH5Pjm/build/asan/bin/kudu pid 20303
I20260809 12:14:59.967605 20453 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 12:14:59.960951 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48102 (local address 127.0.222.3:45649)
0809 12:14:59.961163 (+ 212us) server_negotiation.cc:206] Beginning negotiation
0809 12:14:59.961171 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0809 12:14:59.961193 (+ 22us) server_negotiation.cc:407] Connection header received
0809 12:14:59.961431 (+ 238us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 12:14:59.961436 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 12:14:59.961521 (+ 85us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 12:14:59.961636 (+ 115us) server_negotiation.cc:226] Negotiated authn=SASL
0809 12:14:59.962058 (+ 422us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 12:14:59.962789 (+ 731us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 12:14:59.964230 (+ 1441us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 12:14:59.964631 (+ 401us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 12:14:59.964828 (+ 197us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 12:14:59.966033 (+ 1205us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 12:14:59.966043 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 12:14:59.966059 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0809 12:14:59.966642 (+ 583us) server_negotiation.cc:922] calling sasl_server_start()
0809 12:14:59.966690 (+ 48us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0809 12:14:59.966709 (+ 19us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 12:14:59.966875 (+ 166us) server_negotiation.cc:1035] Waiting for connection context
0809 12:14:59.967074 (+ 199us) server_negotiation.cc:299] Negotiation successful
0809 12:14:59.967202 (+ 128us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":82}
I20260809 12:14:59.968331 20434 generic_service.cc:196] Checking for leaks (request via RPC)
I20260809 12:15:00.058012 888 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskgH5Pjm/build/asan/bin/kudu with pid 20303
I20260809 12:15:00.078516 888 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskgH5Pjm/build/asan/bin/kudu pid 19919
I20260809 12:15:00.079743 19988 generic_service.cc:196] Checking for leaks (request via RPC)
I20260809 12:15:00.192260 888 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskgH5Pjm/build/asan/bin/kudu with pid 19919
2026-08-09T12:15:00Z chronyd exiting
[ OK ] EncryptionPolicyTest.TestEncryptionPolicy/1 (3137 ms)
[ RUN ] EncryptionPolicyTest.TestEncryptionPolicy/2
Loading random data
Initializing database '/tmp/dist-test-taskgH5Pjm/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786277382073938-888-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 09 12:15:00 dist-test-slave-2w3w krb5kdc[20495](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 09 12:15:00 dist-test-slave-2w3w krb5kdc[20495](info): set up 2 sockets
Aug 09 12:15:00 dist-test-slave-2w3w krb5kdc[20495](info): commencing operation
krb5kdc: starting...
W20260809 12:15:01.375689 888 mini_kdc.cc:121] Time spent starting KDC: real 1.136s user 0.006s sys 0.040s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskgH5Pjm/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786277382073938-888-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskgH5Pjm/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786277382073938-888-0/krb5kdc/test-user.keytab.
Aug 09 12:15:01 dist-test-slave-2w3w krb5kdc[20495](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786277701, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-09T12:15:01Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-09T12:15:01Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.222.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.222.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.222.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskgH5Pjm/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786277382073938-888-0/krb5kdc/kudu_127.0.222.62.keytab.
Entry for principal kudu/127.0.222.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskgH5Pjm/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786277382073938-888-0/krb5kdc/kudu_127.0.222.62.keytab.
I20260809 12:15:01.694643 888 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskgH5Pjm/build/asan/bin/kudu
/tmp/dist-test-taskgH5Pjm/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskgH5Pjm/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786277382073938-888-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskgH5Pjm/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786277382073938-888-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskgH5Pjm/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786277382073938-888-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskgH5Pjm/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786277382073938-888-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.222.62:33101
--webserver_interface=127.0.222.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskgH5Pjm/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786277382073938-888-0/krb5kdc/kudu_127.0.222.62.keytab
--principal=kudu/127.0.222.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.222.20:38559
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.222.62:33101
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskgH5Pjm/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786277382073938-888-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskgH5Pjm/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786277382073938-888-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskgH5Pjm/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786277382073938-888-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260809 12:15:02.039143 20508 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260809 12:15:02.039489 20508 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260809 12:15:02.039660 20508 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260809 12:15:02.048317 20508 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260809 12:15:02.048458 20508 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260809 12:15:02.048539 20508 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260809 12:15:02.048674 20508 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260809 12:15:02.048774 20508 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260809 12:15:02.063872 20508 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.222.20:38559
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskgH5Pjm/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786277382073938-888-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskgH5Pjm/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786277382073938-888-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.222.62:33101
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskgH5Pjm/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786277382073938-888-0/krb5kdc/kudu_127.0.222.62.keytab
--principal=kudu/127.0.222.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.222.62:33101
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskgH5Pjm/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786277382073938-888-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.222.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskgH5Pjm/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786277382073938-888-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 09 Aug 2026 11:06:41 UTC on ccycloud.quasar-bwoogy.root.comops.site
build id 4575991
ASAN enabled
I20260809 12:15:02.065457 20508 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260809 12:15:02.067140 20508 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260809 12:15:02.080610 20513 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 12:15:02.080617 20514 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 12:15:02.080751 20516 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260809 12:15:02.081404 20508 server_base.cc:1034] running on GCE node
I20260809 12:15:02.082165 20508 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260809 12:15:02.083600 20508 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260809 12:15:02.084828 20508 hybrid_clock.cc:648] HybridClock initialized: now 1786277702084783 us; error 30 us; skew 500 ppm
I20260809 12:15:02.085188 20508 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 09 12:15:02 dist-test-slave-2w3w krb5kdc[20495](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786277702, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.222.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260809 12:15:02.088555 20508 init.cc:377] Logged in from keytab as kudu/127.0.222.62@KRBTEST.COM (short username kudu)
I20260809 12:15:02.090063 20508 webserver.cc:460] Webserver started at http://127.0.222.62:40119/ using document root <none> and password file <none>
I20260809 12:15:02.090757 20508 fs_manager.cc:359] Metadata directory not provided
I20260809 12:15:02.090899 20508 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260809 12:15:02.091285 20508 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260809 12:15:02.094295 20508 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskgH5Pjm/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786277382073938-888-0/minicluster-data/master-0/data/instance:
uuid: "88d10fc7ebd243ce8011d352fe6d15f0"
format_stamp: "Formatted at 2026-08-09 12:15:02 on dist-test-slave-2w3w"
server_key: "4d421a5d5fe75651ec3116bcf2441ef6"
server_key_iv: "f9a799dcf8eda22bac0412606d814ba6"
server_key_version: "encryptionkey@0"
I20260809 12:15:02.095134 20508 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskgH5Pjm/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786277382073938-888-0/minicluster-data/master-0/wal/instance:
uuid: "88d10fc7ebd243ce8011d352fe6d15f0"
format_stamp: "Formatted at 2026-08-09 12:15:02 on dist-test-slave-2w3w"
server_key: "4d421a5d5fe75651ec3116bcf2441ef6"
server_key_iv: "f9a799dcf8eda22bac0412606d814ba6"
server_key_version: "encryptionkey@0"
I20260809 12:15:02.100471 20508 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.004s sys 0.000s
I20260809 12:15:02.104051 20524 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260809 12:15:02.105512 20508 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260809 12:15:02.105741 20508 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskgH5Pjm/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786277382073938-888-0/minicluster-data/master-0/data,/tmp/dist-test-taskgH5Pjm/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786277382073938-888-0/minicluster-data/master-0/wal
uuid: "88d10fc7ebd243ce8011d352fe6d15f0"
format_stamp: "Formatted at 2026-08-09 12:15:02 on dist-test-slave-2w3w"
server_key: "4d421a5d5fe75651ec3116bcf2441ef6"
server_key_iv: "f9a799dcf8eda22bac0412606d814ba6"
server_key_version: "encryptionkey@0"
I20260809 12:15:02.105991 20508 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskgH5Pjm/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786277382073938-888-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskgH5Pjm/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786277382073938-888-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskgH5Pjm/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786277382073938-888-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260809 12:15:02.127796 20508 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260809 12:15:02.132402 20508 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260809 12:15:02.132720 20508 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260809 12:15:02.151852 20508 rpc_server.cc:307] RPC server started. Bound to: 127.0.222.62:33101
I20260809 12:15:02.151888 20583 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.222.62:33101 every 8 connection(s)
I20260809 12:15:02.153353 20508 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskgH5Pjm/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786277382073938-888-0/minicluster-data/master-0/data/info.pb
I20260809 12:15:02.157270 20584 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260809 12:15:02.158048 888 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskgH5Pjm/build/asan/bin/kudu as pid 20508
I20260809 12:15:02.158285 888 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskgH5Pjm/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786277382073938-888-0/minicluster-data/master-0/wal/instance
I20260809 12:15:02.158784 888 external_mini_cluster.cc:1428] Setting key 6768307775cd7c7bc61b3c96d86e34dc
I20260809 12:15:02.171459 20584 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 88d10fc7ebd243ce8011d352fe6d15f0: Bootstrap starting.
Aug 09 12:15:02 dist-test-slave-2w3w krb5kdc[20495](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786277701, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.222.62@KRBTEST.COM
I20260809 12:15:02.175420 20584 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 88d10fc7ebd243ce8011d352fe6d15f0: Neither blocks nor log segments found. Creating new log.
I20260809 12:15:02.176908 20584 log.cc:824] T 00000000000000000000000000000000 P 88d10fc7ebd243ce8011d352fe6d15f0: Log is configured to *not* fsync() on all Append() calls
I20260809 12:15:02.180428 20584 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 88d10fc7ebd243ce8011d352fe6d15f0: No bootstrap required, opened a new log
I20260809 12:15:02.184619 20587 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 12:15:02.160826 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53462 (local address 127.0.222.62:33101)
0809 12:15:02.161623 (+ 797us) server_negotiation.cc:206] Beginning negotiation
0809 12:15:02.161633 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0809 12:15:02.161674 (+ 41us) server_negotiation.cc:407] Connection header received
0809 12:15:02.163446 (+ 1772us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 12:15:02.163468 (+ 22us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 12:15:02.163867 (+ 399us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 12:15:02.164202 (+ 335us) server_negotiation.cc:226] Negotiated authn=SASL
0809 12:15:02.166103 (+ 1901us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 12:15:02.167547 (+ 1444us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 12:15:02.169498 (+ 1951us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 12:15:02.170187 (+ 689us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 12:15:02.170318 (+ 131us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 12:15:02.175673 (+ 5355us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 12:15:02.175687 (+ 14us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 12:15:02.175707 (+ 20us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0809 12:15:02.175757 (+ 50us) server_negotiation.cc:922] calling sasl_server_start()
0809 12:15:02.179785 (+ 4028us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 12:15:02.180569 (+ 784us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 12:15:02.180577 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 12:15:02.180585 (+ 8us) server_negotiation.cc:967] calling sasl_server_step()
0809 12:15:02.180803 (+ 218us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 12:15:02.181237 (+ 434us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 12:15:02.181242 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 12:15:02.181245 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0809 12:15:02.181964 (+ 719us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 12:15:02.182266 (+ 302us) server_negotiation.cc:1035] Waiting for connection context
0809 12:15:02.182658 (+ 392us) server_negotiation.cc:299] Negotiation successful
0809 12:15:02.183170 (+ 512us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":579,"thread_start_us":387,"threads_started":1}
I20260809 12:15:02.190267 20584 raft_consensus.cc:348] T 00000000000000000000000000000000 P 88d10fc7ebd243ce8011d352fe6d15f0 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "88d10fc7ebd243ce8011d352fe6d15f0" member_type: VOTER last_known_addr { host: "127.0.222.62" port: 33101 } }
I20260809 12:15:02.190711 20584 raft_consensus.cc:374] T 00000000000000000000000000000000 P 88d10fc7ebd243ce8011d352fe6d15f0 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260809 12:15:02.190831 20584 raft_consensus.cc:729] T 00000000000000000000000000000000 P 88d10fc7ebd243ce8011d352fe6d15f0 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 88d10fc7ebd243ce8011d352fe6d15f0, State: Initialized, Role: FOLLOWER
I20260809 12:15:02.191556 20584 consensus_queue.cc:260] T 00000000000000000000000000000000 P 88d10fc7ebd243ce8011d352fe6d15f0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "88d10fc7ebd243ce8011d352fe6d15f0" member_type: VOTER last_known_addr { host: "127.0.222.62" port: 33101 } }
I20260809 12:15:02.191790 20584 raft_consensus.cc:388] T 00000000000000000000000000000000 P 88d10fc7ebd243ce8011d352fe6d15f0 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260809 12:15:02.191956 20584 raft_consensus.cc:482] T 00000000000000000000000000000000 P 88d10fc7ebd243ce8011d352fe6d15f0 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260809 12:15:02.192143 20584 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 88d10fc7ebd243ce8011d352fe6d15f0 [term 0 FOLLOWER]: Advancing to term 1
I20260809 12:15:02.194402 20584 raft_consensus.cc:504] T 00000000000000000000000000000000 P 88d10fc7ebd243ce8011d352fe6d15f0 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "88d10fc7ebd243ce8011d352fe6d15f0" member_type: VOTER last_known_addr { host: "127.0.222.62" port: 33101 } }
I20260809 12:15:02.194941 20584 leader_election.cc:302] T 00000000000000000000000000000000 P 88d10fc7ebd243ce8011d352fe6d15f0 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 88d10fc7ebd243ce8011d352fe6d15f0; no voters:
I20260809 12:15:02.195636 20584 leader_election.cc:288] T 00000000000000000000000000000000 P 88d10fc7ebd243ce8011d352fe6d15f0 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260809 12:15:02.195811 20590 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 88d10fc7ebd243ce8011d352fe6d15f0 [term 1 FOLLOWER]: Leader election won for term 1
I20260809 12:15:02.196632 20590 raft_consensus.cc:686] T 00000000000000000000000000000000 P 88d10fc7ebd243ce8011d352fe6d15f0 [term 1 LEADER]: Becoming Leader. State: Replica: 88d10fc7ebd243ce8011d352fe6d15f0, State: Running, Role: LEADER
I20260809 12:15:02.197216 20590 consensus_queue.cc:237] T 00000000000000000000000000000000 P 88d10fc7ebd243ce8011d352fe6d15f0 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "88d10fc7ebd243ce8011d352fe6d15f0" member_type: VOTER last_known_addr { host: "127.0.222.62" port: 33101 } }
I20260809 12:15:02.197991 20584 sys_catalog.cc:564] T 00000000000000000000000000000000 P 88d10fc7ebd243ce8011d352fe6d15f0 [sys.catalog]: configured and running, proceeding with master startup.
I20260809 12:15:02.200573 20591 sys_catalog.cc:455] T 00000000000000000000000000000000 P 88d10fc7ebd243ce8011d352fe6d15f0 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "88d10fc7ebd243ce8011d352fe6d15f0" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "88d10fc7ebd243ce8011d352fe6d15f0" member_type: VOTER last_known_addr { host: "127.0.222.62" port: 33101 } } }
I20260809 12:15:02.200963 20591 sys_catalog.cc:458] T 00000000000000000000000000000000 P 88d10fc7ebd243ce8011d352fe6d15f0 [sys.catalog]: This master's current role is: LEADER
I20260809 12:15:02.203052 20592 sys_catalog.cc:455] T 00000000000000000000000000000000 P 88d10fc7ebd243ce8011d352fe6d15f0 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 88d10fc7ebd243ce8011d352fe6d15f0. Latest consensus state: current_term: 1 leader_uuid: "88d10fc7ebd243ce8011d352fe6d15f0" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "88d10fc7ebd243ce8011d352fe6d15f0" member_type: VOTER last_known_addr { host: "127.0.222.62" port: 33101 } } }
I20260809 12:15:02.203473 20592 sys_catalog.cc:458] T 00000000000000000000000000000000 P 88d10fc7ebd243ce8011d352fe6d15f0 [sys.catalog]: This master's current role is: LEADER
I20260809 12:15:02.203510 20597 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260809 12:15:02.209197 20597 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260809 12:15:02.218925 20597 catalog_manager.cc:1347] Generated new cluster ID: 18edf39e74664f72b4567eb5435cd05f
I20260809 12:15:02.219098 20597 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260809 12:15:02.236213 20597 catalog_manager.cc:1370] Generated new certificate authority record
I20260809 12:15:02.238256 20597 catalog_manager.cc:1504] Loading token signing keys...
I20260809 12:15:02.247576 20597 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 88d10fc7ebd243ce8011d352fe6d15f0: Generated new TSK 0
I20260809 12:15:02.248670 20597 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.0.222.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.222.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.222.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskgH5Pjm/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786277382073938-888-0/krb5kdc/kudu_127.0.222.1.keytab.
Entry for principal kudu/127.0.222.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskgH5Pjm/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786277382073938-888-0/krb5kdc/kudu_127.0.222.1.keytab.
I20260809 12:15:02.327132 888 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskgH5Pjm/build/asan/bin/kudu
/tmp/dist-test-taskgH5Pjm/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskgH5Pjm/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786277382073938-888-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskgH5Pjm/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786277382073938-888-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskgH5Pjm/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786277382073938-888-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskgH5Pjm/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786277382073938-888-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.222.1:0
--local_ip_for_outbound_sockets=127.0.222.1
--webserver_interface=127.0.222.1
--webserver_port=0
--tserver_master_addrs=127.0.222.62:33101
--keytab_file=/tmp/dist-test-taskgH5Pjm/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786277382073938-888-0/krb5kdc/kudu_127.0.222.1.keytab
--principal=kudu/127.0.222.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.222.20:38559
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskgH5Pjm/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786277382073938-888-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskgH5Pjm/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786277382073938-888-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskgH5Pjm/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786277382073938-888-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260809 12:15:02.675745 20611 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260809 12:15:02.676016 20611 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260809 12:15:02.676184 20611 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260809 12:15:02.683667 20611 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260809 12:15:02.683773 20611 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260809 12:15:02.683983 20611 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.222.1
I20260809 12:15:02.693853 20611 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.222.20:38559
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskgH5Pjm/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786277382073938-888-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskgH5Pjm/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786277382073938-888-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskgH5Pjm/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786277382073938-888-0/krb5kdc/kudu_127.0.222.1.keytab
--principal=kudu/127.0.222.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.222.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskgH5Pjm/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786277382073938-888-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.222.1
--webserver_port=0
--tserver_master_addrs=127.0.222.62:33101
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.222.1
--log_dir=/tmp/dist-test-taskgH5Pjm/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786277382073938-888-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 09 Aug 2026 11:06:41 UTC on ccycloud.quasar-bwoogy.root.comops.site
build id 4575991
ASAN enabled
I20260809 12:15:02.695196 20611 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260809 12:15:02.696511 20611 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260809 12:15:02.710032 20618 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 12:15:02.710314 20617 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 12:15:02.710438 20620 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260809 12:15:02.711373 20611 server_base.cc:1034] running on GCE node
I20260809 12:15:02.712111 20611 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260809 12:15:02.713085 20611 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260809 12:15:02.714286 20611 hybrid_clock.cc:648] HybridClock initialized: now 1786277702714221 us; error 60 us; skew 500 ppm
I20260809 12:15:02.714586 20611 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 09 12:15:02 dist-test-slave-2w3w krb5kdc[20495](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786277702, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.222.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260809 12:15:02.720875 20611 init.cc:377] Logged in from keytab as kudu/127.0.222.1@KRBTEST.COM (short username kudu)
I20260809 12:15:02.722347 20611 webserver.cc:460] Webserver started at http://127.0.222.1:42719/ using document root <none> and password file <none>
I20260809 12:15:02.723132 20611 fs_manager.cc:359] Metadata directory not provided
I20260809 12:15:02.723244 20611 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260809 12:15:02.723513 20611 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260809 12:15:02.726434 20611 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskgH5Pjm/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786277382073938-888-0/minicluster-data/ts-0/data/instance:
uuid: "11d77089e61e4fc68c02fe1642510c46"
format_stamp: "Formatted at 2026-08-09 12:15:02 on dist-test-slave-2w3w"
server_key: "52a3f889a8d39a0fca7a6d5e23329f27"
server_key_iv: "a300c89210da2e56b6619e71669a44bb"
server_key_version: "encryptionkey@0"
I20260809 12:15:02.727289 20611 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskgH5Pjm/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786277382073938-888-0/minicluster-data/ts-0/wal/instance:
uuid: "11d77089e61e4fc68c02fe1642510c46"
format_stamp: "Formatted at 2026-08-09 12:15:02 on dist-test-slave-2w3w"
server_key: "52a3f889a8d39a0fca7a6d5e23329f27"
server_key_iv: "a300c89210da2e56b6619e71669a44bb"
server_key_version: "encryptionkey@0"
I20260809 12:15:02.732373 20611 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.004s sys 0.000s
I20260809 12:15:02.735818 20627 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260809 12:15:02.737164 20611 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260809 12:15:02.737341 20611 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskgH5Pjm/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786277382073938-888-0/minicluster-data/ts-0/data,/tmp/dist-test-taskgH5Pjm/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786277382073938-888-0/minicluster-data/ts-0/wal
uuid: "11d77089e61e4fc68c02fe1642510c46"
format_stamp: "Formatted at 2026-08-09 12:15:02 on dist-test-slave-2w3w"
server_key: "52a3f889a8d39a0fca7a6d5e23329f27"
server_key_iv: "a300c89210da2e56b6619e71669a44bb"
server_key_version: "encryptionkey@0"
I20260809 12:15:02.737550 20611 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskgH5Pjm/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786277382073938-888-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskgH5Pjm/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786277382073938-888-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskgH5Pjm/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786277382073938-888-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260809 12:15:02.755404 20611 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260809 12:15:02.759971 20611 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260809 12:15:02.760321 20611 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260809 12:15:02.761337 20611 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260809 12:15:02.762712 20611 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260809 12:15:02.762789 20611 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260809 12:15:02.763001 20611 ts_tablet_manager.cc:610] Registered 0 tablets
I20260809 12:15:02.763069 20611 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.001s sys 0.000s
I20260809 12:15:02.796620 20611 rpc_server.cc:307] RPC server started. Bound to: 127.0.222.1:43611
I20260809 12:15:02.796674 20749 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.222.1:43611 every 8 connection(s)
I20260809 12:15:02.798069 20611 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskgH5Pjm/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786277382073938-888-0/minicluster-data/ts-0/data/info.pb
I20260809 12:15:02.799937 888 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskgH5Pjm/build/asan/bin/kudu as pid 20611
I20260809 12:15:02.800125 888 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskgH5Pjm/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786277382073938-888-0/minicluster-data/ts-0/wal/instance
I20260809 12:15:02.800748 888 external_mini_cluster.cc:1428] Setting key 7889d2a382f9b025e05047740918b50d
Aug 09 12:15:02 dist-test-slave-2w3w krb5kdc[20495](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786277702, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.222.1@KRBTEST.COM for kudu/127.0.222.62@KRBTEST.COM
I20260809 12:15:02.820775 20753 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 12:15:02.800880 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.222.1:57589 (local address 127.0.222.62:33101)
0809 12:15:02.801396 (+ 516us) server_negotiation.cc:206] Beginning negotiation
0809 12:15:02.801404 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0809 12:15:02.803048 (+ 1644us) server_negotiation.cc:407] Connection header received
0809 12:15:02.804786 (+ 1738us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 12:15:02.804794 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 12:15:02.804888 (+ 94us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 12:15:02.805012 (+ 124us) server_negotiation.cc:226] Negotiated authn=SASL
0809 12:15:02.806492 (+ 1480us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 12:15:02.807662 (+ 1170us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 12:15:02.809180 (+ 1518us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 12:15:02.809906 (+ 726us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 12:15:02.810054 (+ 148us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 12:15:02.813677 (+ 3623us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 12:15:02.813686 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 12:15:02.813693 (+ 7us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0809 12:15:02.813740 (+ 47us) server_negotiation.cc:922] calling sasl_server_start()
0809 12:15:02.817380 (+ 3640us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 12:15:02.818354 (+ 974us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 12:15:02.818358 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 12:15:02.818360 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0809 12:15:02.818493 (+ 133us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 12:15:02.819057 (+ 564us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 12:15:02.819062 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 12:15:02.819066 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0809 12:15:02.819344 (+ 278us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 12:15:02.819460 (+ 116us) server_negotiation.cc:1035] Waiting for connection context
0809 12:15:02.820346 (+ 886us) server_negotiation.cc:299] Negotiation successful
0809 12:15:02.820526 (+ 180us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":366,"thread_start_us":253,"threads_started":1}
I20260809 12:15:02.822283 20754 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 12:15:02.801513 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.222.62:33101 (local address 127.0.222.1:57589)
0809 12:15:02.802389 (+ 876us) negotiation.cc:107] Waiting for socket to connect
0809 12:15:02.802444 (+ 55us) client_negotiation.cc:174] Beginning negotiation
0809 12:15:02.804486 (+ 2042us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 12:15:02.805206 (+ 720us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 12:15:02.805213 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 12:15:02.805734 (+ 521us) client_negotiation.cc:189] Negotiated authn=SASL
0809 12:15:02.806290 (+ 556us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 12:15:02.806320 (+ 30us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 12:15:02.807818 (+ 1498us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 12:15:02.807823 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 12:15:02.809027 (+ 1204us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 12:15:02.809040 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 12:15:02.810115 (+ 1075us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 12:15:02.810120 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 12:15:02.810324 (+ 204us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 12:15:02.811059 (+ 735us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0809 12:15:02.811088 (+ 29us) client_negotiation.cc:656] calling sasl_client_start()
0809 12:15:02.813451 (+ 2363us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0809 12:15:02.817537 (+ 4086us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0809 12:15:02.817548 (+ 11us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0809 12:15:02.817568 (+ 20us) client_negotiation.cc:760] calling sasl_client_step()
0809 12:15:02.818235 (+ 667us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0809 12:15:02.818606 (+ 371us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0809 12:15:02.818614 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0809 12:15:02.818620 (+ 6us) client_negotiation.cc:760] calling sasl_client_step()
0809 12:15:02.818927 (+ 307us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0809 12:15:02.819478 (+ 551us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0809 12:15:02.819489 (+ 11us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0809 12:15:02.819946 (+ 457us) client_negotiation.cc:769] Sending connection context
0809 12:15:02.820249 (+ 303us) client_negotiation.cc:240] Negotiation successful
0809 12:15:02.820825 (+ 576us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":602,"thread_start_us":396,"threads_started":1}
I20260809 12:15:02.823863 20750 heartbeater.cc:347] Connected to a master server at 127.0.222.62:33101
I20260809 12:15:02.824179 20750 heartbeater.cc:464] Registering TS with master...
I20260809 12:15:02.824905 20750 heartbeater.cc:511] Master 127.0.222.62:33101 requested a full tablet report, sending...
I20260809 12:15:02.827014 20543 ts_manager.cc:194] Registered new tserver with Master: 11d77089e61e4fc68c02fe1642510c46 (127.0.222.1:43611)
I20260809 12:15:02.828439 20543 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.222.1@KRBTEST.COM'} at 127.0.222.1:57589
WARNING: no policy specified for kudu/127.0.222.2@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.222.2@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.222.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskgH5Pjm/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786277382073938-888-0/krb5kdc/kudu_127.0.222.2.keytab.
Entry for principal kudu/127.0.222.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskgH5Pjm/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786277382073938-888-0/krb5kdc/kudu_127.0.222.2.keytab.
I20260809 12:15:02.874477 888 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskgH5Pjm/build/asan/bin/kudu
/tmp/dist-test-taskgH5Pjm/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskgH5Pjm/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786277382073938-888-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskgH5Pjm/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786277382073938-888-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskgH5Pjm/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786277382073938-888-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskgH5Pjm/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786277382073938-888-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.222.2:0
--local_ip_for_outbound_sockets=127.0.222.2
--webserver_interface=127.0.222.2
--webserver_port=0
--tserver_master_addrs=127.0.222.62:33101
--keytab_file=/tmp/dist-test-taskgH5Pjm/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786277382073938-888-0/krb5kdc/kudu_127.0.222.2.keytab
--principal=kudu/127.0.222.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.222.20:38559
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskgH5Pjm/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786277382073938-888-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskgH5Pjm/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786277382073938-888-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskgH5Pjm/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786277382073938-888-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260809 12:15:03.215270 20758 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260809 12:15:03.215544 20758 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260809 12:15:03.215735 20758 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260809 12:15:03.223091 20758 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260809 12:15:03.223227 20758 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260809 12:15:03.223430 20758 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.222.2
I20260809 12:15:03.233090 20758 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.222.20:38559
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskgH5Pjm/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786277382073938-888-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskgH5Pjm/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786277382073938-888-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskgH5Pjm/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786277382073938-888-0/krb5kdc/kudu_127.0.222.2.keytab
--principal=kudu/127.0.222.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.222.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskgH5Pjm/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786277382073938-888-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.222.2
--webserver_port=0
--tserver_master_addrs=127.0.222.62:33101
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.222.2
--log_dir=/tmp/dist-test-taskgH5Pjm/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786277382073938-888-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 09 Aug 2026 11:06:41 UTC on ccycloud.quasar-bwoogy.root.comops.site
build id 4575991
ASAN enabled
I20260809 12:15:03.234462 20758 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260809 12:15:03.236061 20758 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260809 12:15:03.249064 20765 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 12:15:03.249155 20768 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 12:15:03.249194 20766 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260809 12:15:03.250315 20758 server_base.cc:1034] running on GCE node
I20260809 12:15:03.251019 20758 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260809 12:15:03.251988 20758 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260809 12:15:03.253196 20758 hybrid_clock.cc:648] HybridClock initialized: now 1786277703253145 us; error 40 us; skew 500 ppm
I20260809 12:15:03.253520 20758 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 09 12:15:03 dist-test-slave-2w3w krb5kdc[20495](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786277703, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.222.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260809 12:15:03.256860 20758 init.cc:377] Logged in from keytab as kudu/127.0.222.2@KRBTEST.COM (short username kudu)
I20260809 12:15:03.258342 20758 webserver.cc:460] Webserver started at http://127.0.222.2:45427/ using document root <none> and password file <none>
I20260809 12:15:03.259094 20758 fs_manager.cc:359] Metadata directory not provided
I20260809 12:15:03.259231 20758 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260809 12:15:03.259585 20758 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260809 12:15:03.262643 20758 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskgH5Pjm/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786277382073938-888-0/minicluster-data/ts-1/data/instance:
uuid: "0d0b0cb4a03a4e33b9f37441b0d4a3ef"
format_stamp: "Formatted at 2026-08-09 12:15:03 on dist-test-slave-2w3w"
server_key: "49cb1bb7cedf026276f5153fbc18a290"
server_key_iv: "b9ca053653af3ceeaf0411a79a210dac"
server_key_version: "encryptionkey@0"
I20260809 12:15:03.263564 20758 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskgH5Pjm/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786277382073938-888-0/minicluster-data/ts-1/wal/instance:
uuid: "0d0b0cb4a03a4e33b9f37441b0d4a3ef"
format_stamp: "Formatted at 2026-08-09 12:15:03 on dist-test-slave-2w3w"
server_key: "49cb1bb7cedf026276f5153fbc18a290"
server_key_iv: "b9ca053653af3ceeaf0411a79a210dac"
server_key_version: "encryptionkey@0"
I20260809 12:15:03.268560 20758 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.004s sys 0.000s
I20260809 12:15:03.271926 20775 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260809 12:15:03.273176 20758 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.001s
I20260809 12:15:03.273344 20758 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskgH5Pjm/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786277382073938-888-0/minicluster-data/ts-1/data,/tmp/dist-test-taskgH5Pjm/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786277382073938-888-0/minicluster-data/ts-1/wal
uuid: "0d0b0cb4a03a4e33b9f37441b0d4a3ef"
format_stamp: "Formatted at 2026-08-09 12:15:03 on dist-test-slave-2w3w"
server_key: "49cb1bb7cedf026276f5153fbc18a290"
server_key_iv: "b9ca053653af3ceeaf0411a79a210dac"
server_key_version: "encryptionkey@0"
I20260809 12:15:03.273516 20758 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskgH5Pjm/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786277382073938-888-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskgH5Pjm/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786277382073938-888-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskgH5Pjm/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786277382073938-888-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260809 12:15:03.313359 20758 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260809 12:15:03.317862 20758 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260809 12:15:03.318213 20758 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260809 12:15:03.319380 20758 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260809 12:15:03.320856 20758 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260809 12:15:03.320940 20758 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260809 12:15:03.321152 20758 ts_tablet_manager.cc:610] Registered 0 tablets
I20260809 12:15:03.321241 20758 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260809 12:15:03.357870 20758 rpc_server.cc:307] RPC server started. Bound to: 127.0.222.2:35123
I20260809 12:15:03.357910 20899 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.222.2:35123 every 8 connection(s)
I20260809 12:15:03.359341 20758 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskgH5Pjm/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786277382073938-888-0/minicluster-data/ts-1/data/info.pb
I20260809 12:15:03.364975 888 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskgH5Pjm/build/asan/bin/kudu as pid 20758
I20260809 12:15:03.365190 888 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskgH5Pjm/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786277382073938-888-0/minicluster-data/ts-1/wal/instance
I20260809 12:15:03.365693 888 external_mini_cluster.cc:1428] Setting key 63e1319de4f528485cdf3f15963288ba
Aug 09 12:15:03 dist-test-slave-2w3w krb5kdc[20495](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786277703, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.222.2@KRBTEST.COM for kudu/127.0.222.62@KRBTEST.COM
I20260809 12:15:03.383279 20902 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 12:15:03.362002 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.222.2:58079 (local address 127.0.222.62:33101)
0809 12:15:03.362537 (+ 535us) server_negotiation.cc:206] Beginning negotiation
0809 12:15:03.362542 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0809 12:15:03.363590 (+ 1048us) server_negotiation.cc:407] Connection header received
0809 12:15:03.365588 (+ 1998us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 12:15:03.365594 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 12:15:03.365693 (+ 99us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 12:15:03.365818 (+ 125us) server_negotiation.cc:226] Negotiated authn=SASL
0809 12:15:03.367821 (+ 2003us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 12:15:03.369090 (+ 1269us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 12:15:03.370656 (+ 1566us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 12:15:03.371540 (+ 884us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 12:15:03.371700 (+ 160us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 12:15:03.375528 (+ 3828us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 12:15:03.375537 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 12:15:03.375546 (+ 9us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0809 12:15:03.375601 (+ 55us) server_negotiation.cc:922] calling sasl_server_start()
0809 12:15:03.379761 (+ 4160us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 12:15:03.380535 (+ 774us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 12:15:03.380544 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 12:15:03.380551 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0809 12:15:03.380739 (+ 188us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 12:15:03.381186 (+ 447us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 12:15:03.381195 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 12:15:03.381201 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0809 12:15:03.381645 (+ 444us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 12:15:03.381822 (+ 177us) server_negotiation.cc:1035] Waiting for connection context
0809 12:15:03.382524 (+ 702us) server_negotiation.cc:299] Negotiation successful
0809 12:15:03.382816 (+ 292us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":371,"thread_start_us":264,"threads_started":1}
I20260809 12:15:03.384337 20903 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 12:15:03.362636 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.222.62:33101 (local address 127.0.222.2:58079)
0809 12:15:03.363409 (+ 773us) negotiation.cc:107] Waiting for socket to connect
0809 12:15:03.363448 (+ 39us) client_negotiation.cc:174] Beginning negotiation
0809 12:15:03.365321 (+ 1873us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 12:15:03.366066 (+ 745us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 12:15:03.366075 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 12:15:03.366761 (+ 686us) client_negotiation.cc:189] Negotiated authn=SASL
0809 12:15:03.367632 (+ 871us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 12:15:03.367662 (+ 30us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 12:15:03.369258 (+ 1596us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 12:15:03.369262 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 12:15:03.370493 (+ 1231us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 12:15:03.370506 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 12:15:03.371758 (+ 1252us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 12:15:03.371763 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 12:15:03.371964 (+ 201us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 12:15:03.372723 (+ 759us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0809 12:15:03.372750 (+ 27us) client_negotiation.cc:656] calling sasl_client_start()
0809 12:15:03.375276 (+ 2526us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0809 12:15:03.379909 (+ 4633us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0809 12:15:03.379916 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0809 12:15:03.379929 (+ 13us) client_negotiation.cc:760] calling sasl_client_step()
0809 12:15:03.380390 (+ 461us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0809 12:15:03.380857 (+ 467us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0809 12:15:03.380860 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0809 12:15:03.380862 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0809 12:15:03.381061 (+ 199us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0809 12:15:03.381770 (+ 709us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0809 12:15:03.381776 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0809 12:15:03.382117 (+ 341us) client_negotiation.cc:769] Sending connection context
0809 12:15:03.382351 (+ 234us) client_negotiation.cc:240] Negotiation successful
0809 12:15:03.382764 (+ 413us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":549,"thread_start_us":359,"threads_started":1}
I20260809 12:15:03.385960 20900 heartbeater.cc:347] Connected to a master server at 127.0.222.62:33101
I20260809 12:15:03.386297 20900 heartbeater.cc:464] Registering TS with master...
I20260809 12:15:03.387217 20900 heartbeater.cc:511] Master 127.0.222.62:33101 requested a full tablet report, sending...
I20260809 12:15:03.388960 20543 ts_manager.cc:194] Registered new tserver with Master: 0d0b0cb4a03a4e33b9f37441b0d4a3ef (127.0.222.2:35123)
I20260809 12:15:03.389940 20543 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.222.2@KRBTEST.COM'} at 127.0.222.2:58079
WARNING: no policy specified for kudu/127.0.222.3@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.222.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.222.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskgH5Pjm/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786277382073938-888-0/krb5kdc/kudu_127.0.222.3.keytab.
Entry for principal kudu/127.0.222.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskgH5Pjm/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786277382073938-888-0/krb5kdc/kudu_127.0.222.3.keytab.
I20260809 12:15:03.442512 888 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskgH5Pjm/build/asan/bin/kudu
/tmp/dist-test-taskgH5Pjm/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskgH5Pjm/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786277382073938-888-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskgH5Pjm/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786277382073938-888-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskgH5Pjm/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786277382073938-888-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskgH5Pjm/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786277382073938-888-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.222.3:0
--local_ip_for_outbound_sockets=127.0.222.3
--webserver_interface=127.0.222.3
--webserver_port=0
--tserver_master_addrs=127.0.222.62:33101
--keytab_file=/tmp/dist-test-taskgH5Pjm/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786277382073938-888-0/krb5kdc/kudu_127.0.222.3.keytab
--principal=kudu/127.0.222.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.222.20:38559
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskgH5Pjm/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786277382073938-888-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskgH5Pjm/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786277382073938-888-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskgH5Pjm/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786277382073938-888-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260809 12:15:03.787809 20907 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260809 12:15:03.788152 20907 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260809 12:15:03.788321 20907 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260809 12:15:03.795720 20907 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260809 12:15:03.795902 20907 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260809 12:15:03.796084 20907 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.222.3
I20260809 12:15:03.806474 20907 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.222.20:38559
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskgH5Pjm/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786277382073938-888-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskgH5Pjm/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786277382073938-888-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskgH5Pjm/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786277382073938-888-0/krb5kdc/kudu_127.0.222.3.keytab
--principal=kudu/127.0.222.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.222.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskgH5Pjm/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786277382073938-888-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.222.3
--webserver_port=0
--tserver_master_addrs=127.0.222.62:33101
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.222.3
--log_dir=/tmp/dist-test-taskgH5Pjm/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786277382073938-888-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 09 Aug 2026 11:06:41 UTC on ccycloud.quasar-bwoogy.root.comops.site
build id 4575991
ASAN enabled
I20260809 12:15:03.807837 20907 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260809 12:15:03.809089 20907 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260809 12:15:03.822014 20912 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 12:15:03.822069 20915 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 12:15:03.822530 20913 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260809 12:15:03.823203 20907 server_base.cc:1034] running on GCE node
I20260809 12:15:03.823875 20907 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260809 12:15:03.824843 20907 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260809 12:15:03.826085 20907 hybrid_clock.cc:648] HybridClock initialized: now 1786277703826032 us; error 46 us; skew 500 ppm
I20260809 12:15:03.826408 20907 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 09 12:15:03 dist-test-slave-2w3w krb5kdc[20495](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786277703, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.222.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260809 12:15:03.829761 20907 init.cc:377] Logged in from keytab as kudu/127.0.222.3@KRBTEST.COM (short username kudu)
I20260809 12:15:03.831488 20750 heartbeater.cc:503] Master 127.0.222.62:33101 was elected leader, sending a full tablet report...
I20260809 12:15:03.831571 20907 webserver.cc:460] Webserver started at http://127.0.222.3:44465/ using document root <none> and password file <none>
I20260809 12:15:03.832337 20907 fs_manager.cc:359] Metadata directory not provided
I20260809 12:15:03.832500 20907 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260809 12:15:03.832839 20907 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260809 12:15:03.836078 20907 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskgH5Pjm/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786277382073938-888-0/minicluster-data/ts-2/data/instance:
uuid: "663b94b3d1d0498dbfb7a86eb2dc5794"
format_stamp: "Formatted at 2026-08-09 12:15:03 on dist-test-slave-2w3w"
server_key: "a6f4fe08e41f8d14ceb8a2db85345d57"
server_key_iv: "898db4473b320ef0f86bd436c3b89051"
server_key_version: "encryptionkey@0"
I20260809 12:15:03.836896 20907 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskgH5Pjm/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786277382073938-888-0/minicluster-data/ts-2/wal/instance:
uuid: "663b94b3d1d0498dbfb7a86eb2dc5794"
format_stamp: "Formatted at 2026-08-09 12:15:03 on dist-test-slave-2w3w"
server_key: "a6f4fe08e41f8d14ceb8a2db85345d57"
server_key_iv: "898db4473b320ef0f86bd436c3b89051"
server_key_version: "encryptionkey@0"
I20260809 12:15:03.842552 20907 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.002s sys 0.004s
I20260809 12:15:03.846330 20922 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260809 12:15:03.847738 20907 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260809 12:15:03.847923 20907 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskgH5Pjm/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786277382073938-888-0/minicluster-data/ts-2/data,/tmp/dist-test-taskgH5Pjm/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786277382073938-888-0/minicluster-data/ts-2/wal
uuid: "663b94b3d1d0498dbfb7a86eb2dc5794"
format_stamp: "Formatted at 2026-08-09 12:15:03 on dist-test-slave-2w3w"
server_key: "a6f4fe08e41f8d14ceb8a2db85345d57"
server_key_iv: "898db4473b320ef0f86bd436c3b89051"
server_key_version: "encryptionkey@0"
I20260809 12:15:03.848182 20907 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskgH5Pjm/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786277382073938-888-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskgH5Pjm/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786277382073938-888-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskgH5Pjm/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786277382073938-888-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260809 12:15:03.867115 20907 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260809 12:15:03.871575 20907 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260809 12:15:03.871902 20907 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260809 12:15:03.872951 20907 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260809 12:15:03.874375 20907 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260809 12:15:03.874477 20907 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260809 12:15:03.874639 20907 ts_tablet_manager.cc:610] Registered 0 tablets
I20260809 12:15:03.874713 20907 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260809 12:15:03.909421 20907 rpc_server.cc:307] RPC server started. Bound to: 127.0.222.3:39667
I20260809 12:15:03.909472 21043 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.222.3:39667 every 8 connection(s)
I20260809 12:15:03.910775 20907 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskgH5Pjm/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786277382073938-888-0/minicluster-data/ts-2/data/info.pb
I20260809 12:15:03.915205 888 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskgH5Pjm/build/asan/bin/kudu as pid 20907
I20260809 12:15:03.915431 888 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskgH5Pjm/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786277382073938-888-0/minicluster-data/ts-2/wal/instance
I20260809 12:15:03.916021 888 external_mini_cluster.cc:1428] Setting key 8cded422ce35a73ee49288f1af1e777d
Aug 09 12:15:03 dist-test-slave-2w3w krb5kdc[20495](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786277703, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.222.3@KRBTEST.COM for kudu/127.0.222.62@KRBTEST.COM
I20260809 12:15:03.932673 21046 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 12:15:03.913482 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.222.3:38073 (local address 127.0.222.62:33101)
0809 12:15:03.913986 (+ 504us) server_negotiation.cc:206] Beginning negotiation
0809 12:15:03.913993 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0809 12:15:03.915118 (+ 1125us) server_negotiation.cc:407] Connection header received
0809 12:15:03.917668 (+ 2550us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 12:15:03.917672 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 12:15:03.917834 (+ 162us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 12:15:03.917975 (+ 141us) server_negotiation.cc:226] Negotiated authn=SASL
0809 12:15:03.919827 (+ 1852us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 12:15:03.920619 (+ 792us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 12:15:03.922228 (+ 1609us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 12:15:03.922651 (+ 423us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 12:15:03.922760 (+ 109us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 12:15:03.926366 (+ 3606us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 12:15:03.926372 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 12:15:03.926375 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0809 12:15:03.926412 (+ 37us) server_negotiation.cc:922] calling sasl_server_start()
0809 12:15:03.929244 (+ 2832us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 12:15:03.930069 (+ 825us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 12:15:03.930077 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 12:15:03.930086 (+ 9us) server_negotiation.cc:967] calling sasl_server_step()
0809 12:15:03.930214 (+ 128us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 12:15:03.930686 (+ 472us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 12:15:03.930691 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 12:15:03.930693 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0809 12:15:03.931064 (+ 371us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 12:15:03.931247 (+ 183us) server_negotiation.cc:1035] Waiting for connection context
0809 12:15:03.932001 (+ 754us) server_negotiation.cc:299] Negotiation successful
0809 12:15:03.932241 (+ 240us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":365,"thread_start_us":216,"threads_started":1}
I20260809 12:15:03.933734 21047 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 12:15:03.914047 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.222.62:33101 (local address 127.0.222.3:38073)
0809 12:15:03.914873 (+ 826us) negotiation.cc:107] Waiting for socket to connect
0809 12:15:03.914940 (+ 67us) client_negotiation.cc:174] Beginning negotiation
0809 12:15:03.917296 (+ 2356us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 12:15:03.918287 (+ 991us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 12:15:03.918298 (+ 11us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 12:15:03.918876 (+ 578us) client_negotiation.cc:189] Negotiated authn=SASL
0809 12:15:03.919631 (+ 755us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 12:15:03.919649 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 12:15:03.920773 (+ 1124us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 12:15:03.920777 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 12:15:03.922069 (+ 1292us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 12:15:03.922081 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 12:15:03.922878 (+ 797us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 12:15:03.922884 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 12:15:03.923161 (+ 277us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 12:15:03.923826 (+ 665us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0809 12:15:03.923854 (+ 28us) client_negotiation.cc:656] calling sasl_client_start()
0809 12:15:03.926147 (+ 2293us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0809 12:15:03.929432 (+ 3285us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0809 12:15:03.929441 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0809 12:15:03.929455 (+ 14us) client_negotiation.cc:760] calling sasl_client_step()
0809 12:15:03.929918 (+ 463us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0809 12:15:03.930349 (+ 431us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0809 12:15:03.930354 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0809 12:15:03.930359 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0809 12:15:03.930550 (+ 191us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0809 12:15:03.931208 (+ 658us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0809 12:15:03.931217 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0809 12:15:03.931592 (+ 375us) client_negotiation.cc:769] Sending connection context
0809 12:15:03.931879 (+ 287us) client_negotiation.cc:240] Negotiation successful
0809 12:15:03.932521 (+ 642us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":484,"thread_start_us":281,"threads_started":1}
I20260809 12:15:03.935271 21044 heartbeater.cc:347] Connected to a master server at 127.0.222.62:33101
I20260809 12:15:03.935606 21044 heartbeater.cc:464] Registering TS with master...
I20260809 12:15:03.936375 21044 heartbeater.cc:511] Master 127.0.222.62:33101 requested a full tablet report, sending...
I20260809 12:15:03.937834 20543 ts_manager.cc:194] Registered new tserver with Master: 663b94b3d1d0498dbfb7a86eb2dc5794 (127.0.222.3:39667)
I20260809 12:15:03.938658 20543 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.222.3@KRBTEST.COM'} at 127.0.222.3:38073
I20260809 12:15:03.944118 888 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260809 12:15:03.961891 21046 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 12:15:03.948255 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53474 (local address 127.0.222.62:33101)
0809 12:15:03.948436 (+ 181us) server_negotiation.cc:206] Beginning negotiation
0809 12:15:03.948441 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0809 12:15:03.948918 (+ 477us) server_negotiation.cc:407] Connection header received
0809 12:15:03.949124 (+ 206us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 12:15:03.949128 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 12:15:03.949194 (+ 66us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 12:15:03.949302 (+ 108us) server_negotiation.cc:226] Negotiated authn=SASL
0809 12:15:03.950736 (+ 1434us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 12:15:03.951577 (+ 841us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 12:15:03.953006 (+ 1429us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 12:15:03.953408 (+ 402us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 12:15:03.953546 (+ 138us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 12:15:03.956469 (+ 2923us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 12:15:03.956483 (+ 14us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 12:15:03.956488 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0809 12:15:03.956521 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0809 12:15:03.958911 (+ 2390us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 12:15:03.959765 (+ 854us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 12:15:03.959771 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 12:15:03.959774 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0809 12:15:03.959913 (+ 139us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 12:15:03.960396 (+ 483us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 12:15:03.960401 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 12:15:03.960405 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0809 12:15:03.960722 (+ 317us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 12:15:03.961013 (+ 291us) server_negotiation.cc:1035] Waiting for connection context
0809 12:15:03.961391 (+ 378us) server_negotiation.cc:299] Negotiation successful
0809 12:15:03.961591 (+ 200us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":73}
I20260809 12:15:03.966120 20543 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:53474:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260809 12:15:03.973188 20543 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260809 12:15:04.002632 21061 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 12:15:03.993058 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59496 (local address 127.0.222.1:43611)
0809 12:15:03.995684 (+ 2626us) server_negotiation.cc:206] Beginning negotiation
0809 12:15:03.995695 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0809 12:15:03.995723 (+ 28us) server_negotiation.cc:407] Connection header received
0809 12:15:03.995817 (+ 94us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 12:15:03.995824 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 12:15:03.996069 (+ 245us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 12:15:03.996252 (+ 183us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0809 12:15:03.996850 (+ 598us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 12:15:03.998231 (+ 1381us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 12:15:04.000452 (+ 2221us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 12:15:04.001399 (+ 947us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 12:15:04.001562 (+ 163us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 12:15:04.001814 (+ 252us) server_negotiation.cc:1035] Waiting for connection context
0809 12:15:04.002094 (+ 280us) server_negotiation.cc:299] Negotiation successful
0809 12:15:04.002265 (+ 171us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":2438,"thread_start_us":2270,"threads_started":1}
I20260809 12:15:04.002797 21062 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 12:15:03.993782 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.222.1:43611 (local address 127.0.0.1:59496)
0809 12:15:03.994900 (+ 1118us) negotiation.cc:107] Waiting for socket to connect
0809 12:15:03.994936 (+ 36us) client_negotiation.cc:174] Beginning negotiation
0809 12:15:03.995088 (+ 152us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 12:15:03.996277 (+ 1189us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 12:15:03.996282 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 12:15:03.996308 (+ 26us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0809 12:15:03.996656 (+ 348us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 12:15:03.996668 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 12:15:03.998424 (+ 1756us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 12:15:03.998435 (+ 11us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 12:15:04.000274 (+ 1839us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 12:15:04.000290 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 12:15:04.001615 (+ 1325us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 12:15:04.001622 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 12:15:04.001818 (+ 196us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 12:15:04.001876 (+ 58us) client_negotiation.cc:769] Sending connection context
0809 12:15:04.002038 (+ 162us) client_negotiation.cc:240] Negotiation successful
0809 12:15:04.002257 (+ 219us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":973,"mutex_wait_us":4,"thread_start_us":797,"threads_started":1}
I20260809 12:15:04.003199 21057 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 12:15:03.985593 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.222.2:35123 (local address 127.0.0.1:51640)
0809 12:15:03.992108 (+ 6515us) negotiation.cc:107] Waiting for socket to connect
0809 12:15:03.992161 (+ 53us) client_negotiation.cc:174] Beginning negotiation
0809 12:15:03.992385 (+ 224us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 12:15:03.994125 (+ 1740us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 12:15:03.994137 (+ 12us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 12:15:03.994179 (+ 42us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0809 12:15:03.994848 (+ 669us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 12:15:03.994872 (+ 24us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 12:15:03.996934 (+ 2062us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 12:15:03.996942 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 12:15:04.000104 (+ 3162us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 12:15:04.000123 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 12:15:04.002171 (+ 2048us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 12:15:04.002177 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 12:15:04.002381 (+ 204us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 12:15:04.002421 (+ 40us) client_negotiation.cc:769] Sending connection context
0809 12:15:04.002487 (+ 66us) client_negotiation.cc:240] Negotiation successful
0809 12:15:04.002643 (+ 156us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":6169,"mutex_wait_us":411,"thread_start_us":7127,"threads_started":1}
I20260809 12:15:04.004159 21060 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 12:15:03.986270 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59440 (local address 127.0.222.3:39667)
0809 12:15:03.991395 (+ 5125us) server_negotiation.cc:206] Beginning negotiation
0809 12:15:03.991409 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0809 12:15:03.993195 (+ 1786us) server_negotiation.cc:407] Connection header received
0809 12:15:03.993725 (+ 530us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 12:15:03.993733 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 12:15:03.993968 (+ 235us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 12:15:03.994147 (+ 179us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0809 12:15:03.995370 (+ 1223us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 12:15:03.996961 (+ 1591us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 12:15:04.000292 (+ 3331us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 12:15:04.001736 (+ 1444us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 12:15:04.001886 (+ 150us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 12:15:04.002128 (+ 242us) server_negotiation.cc:1035] Waiting for connection context
0809 12:15:04.002644 (+ 516us) server_negotiation.cc:299] Negotiation successful
0809 12:15:04.003845 (+ 1201us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":4880,"thread_start_us":4703,"threads_started":1}
I20260809 12:15:04.004320 21059 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 12:15:03.985712 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51640 (local address 127.0.222.2:35123)
0809 12:15:03.993357 (+ 7645us) server_negotiation.cc:206] Beginning negotiation
0809 12:15:03.993370 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0809 12:15:03.993408 (+ 38us) server_negotiation.cc:407] Connection header received
0809 12:15:03.993548 (+ 140us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 12:15:03.993559 (+ 11us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 12:15:03.993832 (+ 273us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 12:15:03.995170 (+ 1338us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0809 12:15:03.995444 (+ 274us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 12:15:03.996753 (+ 1309us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 12:15:04.000502 (+ 3749us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 12:15:04.001934 (+ 1432us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 12:15:04.003531 (+ 1597us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 12:15:04.003738 (+ 207us) server_negotiation.cc:1035] Waiting for connection context
0809 12:15:04.003805 (+ 67us) server_negotiation.cc:299] Negotiation successful
0809 12:15:04.003967 (+ 162us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":7449,"thread_start_us":7344,"threads_started":1}
I20260809 12:15:04.004957 21058 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 12:15:03.985645 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.222.3:39667 (local address 127.0.0.1:59440)
0809 12:15:03.993083 (+ 7438us) negotiation.cc:107] Waiting for socket to connect
0809 12:15:03.993101 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0809 12:15:03.993508 (+ 407us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 12:15:03.994492 (+ 984us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 12:15:03.994499 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 12:15:03.994542 (+ 43us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0809 12:15:03.995167 (+ 625us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 12:15:03.995180 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 12:15:03.997104 (+ 1924us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 12:15:03.997109 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 12:15:04.000104 (+ 2995us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 12:15:04.000123 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 12:15:04.002262 (+ 2139us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 12:15:04.002268 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 12:15:04.002442 (+ 174us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 12:15:04.002481 (+ 39us) client_negotiation.cc:769] Sending connection context
0809 12:15:04.004376 (+ 1895us) client_negotiation.cc:240] Negotiation successful
0809 12:15:04.004591 (+ 215us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":7296,"mutex_wait_us":227,"thread_start_us":5984,"threads_started":1}
I20260809 12:15:04.007215 20681 tablet_service.cc:1467] Processing CreateTablet for tablet 9882fab77be443f6b78f213d71cc9860 (DEFAULT_TABLE table=test-table [id=9f70105361c842e1975726244d94a22e]), partition=RANGE (key) PARTITION UNBOUNDED
I20260809 12:15:04.007941 20826 tablet_service.cc:1467] Processing CreateTablet for tablet 9882fab77be443f6b78f213d71cc9860 (DEFAULT_TABLE table=test-table [id=9f70105361c842e1975726244d94a22e]), partition=RANGE (key) PARTITION UNBOUNDED
I20260809 12:15:04.008121 20973 tablet_service.cc:1467] Processing CreateTablet for tablet 9882fab77be443f6b78f213d71cc9860 (DEFAULT_TABLE table=test-table [id=9f70105361c842e1975726244d94a22e]), partition=RANGE (key) PARTITION UNBOUNDED
I20260809 12:15:04.008859 20681 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 9882fab77be443f6b78f213d71cc9860. 1 dirs total, 0 dirs full, 0 dirs failed
I20260809 12:15:04.009342 20973 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 9882fab77be443f6b78f213d71cc9860. 1 dirs total, 0 dirs full, 0 dirs failed
I20260809 12:15:04.009879 20826 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 9882fab77be443f6b78f213d71cc9860. 1 dirs total, 0 dirs full, 0 dirs failed
I20260809 12:15:04.020607 21063 tablet_bootstrap.cc:492] T 9882fab77be443f6b78f213d71cc9860 P 663b94b3d1d0498dbfb7a86eb2dc5794: Bootstrap starting.
I20260809 12:15:04.023671 21064 tablet_bootstrap.cc:492] T 9882fab77be443f6b78f213d71cc9860 P 11d77089e61e4fc68c02fe1642510c46: Bootstrap starting.
I20260809 12:15:04.024042 21063 tablet_bootstrap.cc:654] T 9882fab77be443f6b78f213d71cc9860 P 663b94b3d1d0498dbfb7a86eb2dc5794: Neither blocks nor log segments found. Creating new log.
I20260809 12:15:04.025794 21065 tablet_bootstrap.cc:492] T 9882fab77be443f6b78f213d71cc9860 P 0d0b0cb4a03a4e33b9f37441b0d4a3ef: Bootstrap starting.
I20260809 12:15:04.026723 21063 log.cc:824] T 9882fab77be443f6b78f213d71cc9860 P 663b94b3d1d0498dbfb7a86eb2dc5794: Log is configured to *not* fsync() on all Append() calls
I20260809 12:15:04.027340 21064 tablet_bootstrap.cc:654] T 9882fab77be443f6b78f213d71cc9860 P 11d77089e61e4fc68c02fe1642510c46: Neither blocks nor log segments found. Creating new log.
I20260809 12:15:04.028448 21064 log.cc:824] T 9882fab77be443f6b78f213d71cc9860 P 11d77089e61e4fc68c02fe1642510c46: Log is configured to *not* fsync() on all Append() calls
I20260809 12:15:04.029175 21065 tablet_bootstrap.cc:654] T 9882fab77be443f6b78f213d71cc9860 P 0d0b0cb4a03a4e33b9f37441b0d4a3ef: Neither blocks nor log segments found. Creating new log.
I20260809 12:15:04.030138 21063 tablet_bootstrap.cc:492] T 9882fab77be443f6b78f213d71cc9860 P 663b94b3d1d0498dbfb7a86eb2dc5794: No bootstrap required, opened a new log
I20260809 12:15:04.030354 21065 log.cc:824] T 9882fab77be443f6b78f213d71cc9860 P 0d0b0cb4a03a4e33b9f37441b0d4a3ef: Log is configured to *not* fsync() on all Append() calls
I20260809 12:15:04.030424 21063 ts_tablet_manager.cc:1397] T 9882fab77be443f6b78f213d71cc9860 P 663b94b3d1d0498dbfb7a86eb2dc5794: Time spent bootstrapping tablet: real 0.010s user 0.006s sys 0.000s
I20260809 12:15:04.031615 21064 tablet_bootstrap.cc:492] T 9882fab77be443f6b78f213d71cc9860 P 11d77089e61e4fc68c02fe1642510c46: No bootstrap required, opened a new log
I20260809 12:15:04.031911 21064 ts_tablet_manager.cc:1397] T 9882fab77be443f6b78f213d71cc9860 P 11d77089e61e4fc68c02fe1642510c46: Time spent bootstrapping tablet: real 0.008s user 0.007s sys 0.000s
I20260809 12:15:04.032907 21065 tablet_bootstrap.cc:492] T 9882fab77be443f6b78f213d71cc9860 P 0d0b0cb4a03a4e33b9f37441b0d4a3ef: No bootstrap required, opened a new log
I20260809 12:15:04.033174 21065 ts_tablet_manager.cc:1397] T 9882fab77be443f6b78f213d71cc9860 P 0d0b0cb4a03a4e33b9f37441b0d4a3ef: Time spent bootstrapping tablet: real 0.008s user 0.006s sys 0.000s
I20260809 12:15:04.041121 21065 raft_consensus.cc:348] T 9882fab77be443f6b78f213d71cc9860 P 0d0b0cb4a03a4e33b9f37441b0d4a3ef [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0d0b0cb4a03a4e33b9f37441b0d4a3ef" member_type: VOTER last_known_addr { host: "127.0.222.2" port: 35123 } } peers { permanent_uuid: "663b94b3d1d0498dbfb7a86eb2dc5794" member_type: VOTER last_known_addr { host: "127.0.222.3" port: 39667 } } peers { permanent_uuid: "11d77089e61e4fc68c02fe1642510c46" member_type: VOTER last_known_addr { host: "127.0.222.1" port: 43611 } }
I20260809 12:15:04.041589 21065 raft_consensus.cc:374] T 9882fab77be443f6b78f213d71cc9860 P 0d0b0cb4a03a4e33b9f37441b0d4a3ef [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260809 12:15:04.041347 21063 raft_consensus.cc:348] T 9882fab77be443f6b78f213d71cc9860 P 663b94b3d1d0498dbfb7a86eb2dc5794 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0d0b0cb4a03a4e33b9f37441b0d4a3ef" member_type: VOTER last_known_addr { host: "127.0.222.2" port: 35123 } } peers { permanent_uuid: "663b94b3d1d0498dbfb7a86eb2dc5794" member_type: VOTER last_known_addr { host: "127.0.222.3" port: 39667 } } peers { permanent_uuid: "11d77089e61e4fc68c02fe1642510c46" member_type: VOTER last_known_addr { host: "127.0.222.1" port: 43611 } }
I20260809 12:15:04.041689 21065 raft_consensus.cc:729] T 9882fab77be443f6b78f213d71cc9860 P 0d0b0cb4a03a4e33b9f37441b0d4a3ef [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0d0b0cb4a03a4e33b9f37441b0d4a3ef, State: Initialized, Role: FOLLOWER
I20260809 12:15:04.041793 21063 raft_consensus.cc:374] T 9882fab77be443f6b78f213d71cc9860 P 663b94b3d1d0498dbfb7a86eb2dc5794 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260809 12:15:04.041913 21063 raft_consensus.cc:729] T 9882fab77be443f6b78f213d71cc9860 P 663b94b3d1d0498dbfb7a86eb2dc5794 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 663b94b3d1d0498dbfb7a86eb2dc5794, State: Initialized, Role: FOLLOWER
I20260809 12:15:04.042513 21065 consensus_queue.cc:260] T 9882fab77be443f6b78f213d71cc9860 P 0d0b0cb4a03a4e33b9f37441b0d4a3ef [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0d0b0cb4a03a4e33b9f37441b0d4a3ef" member_type: VOTER last_known_addr { host: "127.0.222.2" port: 35123 } } peers { permanent_uuid: "663b94b3d1d0498dbfb7a86eb2dc5794" member_type: VOTER last_known_addr { host: "127.0.222.3" port: 39667 } } peers { permanent_uuid: "11d77089e61e4fc68c02fe1642510c46" member_type: VOTER last_known_addr { host: "127.0.222.1" port: 43611 } }
I20260809 12:15:04.042524 21063 consensus_queue.cc:260] T 9882fab77be443f6b78f213d71cc9860 P 663b94b3d1d0498dbfb7a86eb2dc5794 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0d0b0cb4a03a4e33b9f37441b0d4a3ef" member_type: VOTER last_known_addr { host: "127.0.222.2" port: 35123 } } peers { permanent_uuid: "663b94b3d1d0498dbfb7a86eb2dc5794" member_type: VOTER last_known_addr { host: "127.0.222.3" port: 39667 } } peers { permanent_uuid: "11d77089e61e4fc68c02fe1642510c46" member_type: VOTER last_known_addr { host: "127.0.222.1" port: 43611 } }
I20260809 12:15:04.042703 21064 raft_consensus.cc:348] T 9882fab77be443f6b78f213d71cc9860 P 11d77089e61e4fc68c02fe1642510c46 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0d0b0cb4a03a4e33b9f37441b0d4a3ef" member_type: VOTER last_known_addr { host: "127.0.222.2" port: 35123 } } peers { permanent_uuid: "663b94b3d1d0498dbfb7a86eb2dc5794" member_type: VOTER last_known_addr { host: "127.0.222.3" port: 39667 } } peers { permanent_uuid: "11d77089e61e4fc68c02fe1642510c46" member_type: VOTER last_known_addr { host: "127.0.222.1" port: 43611 } }
I20260809 12:15:04.043210 21064 raft_consensus.cc:374] T 9882fab77be443f6b78f213d71cc9860 P 11d77089e61e4fc68c02fe1642510c46 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260809 12:15:04.043351 21064 raft_consensus.cc:729] T 9882fab77be443f6b78f213d71cc9860 P 11d77089e61e4fc68c02fe1642510c46 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 11d77089e61e4fc68c02fe1642510c46, State: Initialized, Role: FOLLOWER
I20260809 12:15:04.044035 20900 heartbeater.cc:503] Master 127.0.222.62:33101 was elected leader, sending a full tablet report...
I20260809 12:15:04.044431 21065 ts_tablet_manager.cc:1428] T 9882fab77be443f6b78f213d71cc9860 P 0d0b0cb4a03a4e33b9f37441b0d4a3ef: Time spent starting tablet: real 0.011s user 0.010s sys 0.001s
I20260809 12:15:04.045101 21064 consensus_queue.cc:260] T 9882fab77be443f6b78f213d71cc9860 P 11d77089e61e4fc68c02fe1642510c46 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0d0b0cb4a03a4e33b9f37441b0d4a3ef" member_type: VOTER last_known_addr { host: "127.0.222.2" port: 35123 } } peers { permanent_uuid: "663b94b3d1d0498dbfb7a86eb2dc5794" member_type: VOTER last_known_addr { host: "127.0.222.3" port: 39667 } } peers { permanent_uuid: "11d77089e61e4fc68c02fe1642510c46" member_type: VOTER last_known_addr { host: "127.0.222.1" port: 43611 } }
I20260809 12:15:04.046922 21064 ts_tablet_manager.cc:1428] T 9882fab77be443f6b78f213d71cc9860 P 11d77089e61e4fc68c02fe1642510c46: Time spent starting tablet: real 0.015s user 0.014s sys 0.000s
I20260809 12:15:04.047905 21044 heartbeater.cc:503] Master 127.0.222.62:33101 was elected leader, sending a full tablet report...
I20260809 12:15:04.048487 21063 ts_tablet_manager.cc:1428] T 9882fab77be443f6b78f213d71cc9860 P 663b94b3d1d0498dbfb7a86eb2dc5794: Time spent starting tablet: real 0.018s user 0.012s sys 0.007s
W20260809 12:15:04.050833 20751 tablet.cc:2369] T 9882fab77be443f6b78f213d71cc9860 P 11d77089e61e4fc68c02fe1642510c46: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260809 12:15:04.059500 21071 raft_consensus.cc:482] T 9882fab77be443f6b78f213d71cc9860 P 11d77089e61e4fc68c02fe1642510c46 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260809 12:15:04.059731 21071 raft_consensus.cc:504] T 9882fab77be443f6b78f213d71cc9860 P 11d77089e61e4fc68c02fe1642510c46 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0d0b0cb4a03a4e33b9f37441b0d4a3ef" member_type: VOTER last_known_addr { host: "127.0.222.2" port: 35123 } } peers { permanent_uuid: "663b94b3d1d0498dbfb7a86eb2dc5794" member_type: VOTER last_known_addr { host: "127.0.222.3" port: 39667 } } peers { permanent_uuid: "11d77089e61e4fc68c02fe1642510c46" member_type: VOTER last_known_addr { host: "127.0.222.1" port: 43611 } }
I20260809 12:15:04.061287 21071 leader_election.cc:288] T 9882fab77be443f6b78f213d71cc9860 P 11d77089e61e4fc68c02fe1642510c46 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 0d0b0cb4a03a4e33b9f37441b0d4a3ef (127.0.222.2:35123), 663b94b3d1d0498dbfb7a86eb2dc5794 (127.0.222.3:39667)
I20260809 12:15:04.069396 21059 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 12:15:04.061667 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.222.1:42197 (local address 127.0.222.2:35123)
0809 12:15:04.061933 (+ 266us) server_negotiation.cc:206] Beginning negotiation
0809 12:15:04.061943 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0809 12:15:04.062749 (+ 806us) server_negotiation.cc:407] Connection header received
0809 12:15:04.062945 (+ 196us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 12:15:04.062999 (+ 54us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 12:15:04.063125 (+ 126us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 12:15:04.063270 (+ 145us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0809 12:15:04.064070 (+ 800us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 12:15:04.064960 (+ 890us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 12:15:04.066949 (+ 1989us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 12:15:04.068224 (+ 1275us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 12:15:04.068361 (+ 137us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 12:15:04.068451 (+ 90us) server_negotiation.cc:1035] Waiting for connection context
0809 12:15:04.068988 (+ 537us) server_negotiation.cc:299] Negotiation successful
0809 12:15:04.069122 (+ 134us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":65}
I20260809 12:15:04.069383 21060 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 12:15:04.061760 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.222.1:46043 (local address 127.0.222.3:39667)
0809 12:15:04.061984 (+ 224us) server_negotiation.cc:206] Beginning negotiation
0809 12:15:04.061990 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0809 12:15:04.062802 (+ 812us) server_negotiation.cc:407] Connection header received
0809 12:15:04.063106 (+ 304us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 12:15:04.063111 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 12:15:04.063184 (+ 73us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 12:15:04.063248 (+ 64us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0809 12:15:04.064229 (+ 981us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 12:15:04.065047 (+ 818us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 12:15:04.066879 (+ 1832us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 12:15:04.068143 (+ 1264us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 12:15:04.068287 (+ 144us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 12:15:04.068403 (+ 116us) server_negotiation.cc:1035] Waiting for connection context
0809 12:15:04.068989 (+ 586us) server_negotiation.cc:299] Negotiation successful
0809 12:15:04.069123 (+ 134us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":65}
I20260809 12:15:04.069697 21072 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 12:15:04.061744 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.222.3:39667 (local address 127.0.222.1:46043)
0809 12:15:04.062662 (+ 918us) negotiation.cc:107] Waiting for socket to connect
0809 12:15:04.062716 (+ 54us) client_negotiation.cc:174] Beginning negotiation
0809 12:15:04.062913 (+ 197us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 12:15:04.063472 (+ 559us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 12:15:04.063478 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 12:15:04.063556 (+ 78us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0809 12:15:04.064069 (+ 513us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 12:15:04.064081 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 12:15:04.065190 (+ 1109us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 12:15:04.065196 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 12:15:04.066722 (+ 1526us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 12:15:04.066735 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 12:15:04.068427 (+ 1692us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 12:15:04.068435 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 12:15:04.068747 (+ 312us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 12:15:04.068848 (+ 101us) client_negotiation.cc:769] Sending connection context
0809 12:15:04.068949 (+ 101us) client_negotiation.cc:240] Negotiation successful
0809 12:15:04.069169 (+ 220us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":800,"mutex_wait_us":214,"spinlock_wait_cycles":7680,"thread_start_us":441,"threads_started":1}
I20260809 12:15:04.069895 21073 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 12:15:04.061659 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.222.2:35123 (local address 127.0.222.1:42197)
0809 12:15:04.062560 (+ 901us) negotiation.cc:107] Waiting for socket to connect
0809 12:15:04.062576 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0809 12:15:04.062770 (+ 194us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 12:15:04.063279 (+ 509us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 12:15:04.063286 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 12:15:04.063320 (+ 34us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0809 12:15:04.063910 (+ 590us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 12:15:04.063921 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 12:15:04.065094 (+ 1173us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 12:15:04.065100 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 12:15:04.066728 (+ 1628us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 12:15:04.066742 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 12:15:04.068445 (+ 1703us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 12:15:04.068450 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 12:15:04.068745 (+ 295us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 12:15:04.068852 (+ 107us) client_negotiation.cc:769] Sending connection context
0809 12:15:04.068949 (+ 97us) client_negotiation.cc:240] Negotiation successful
0809 12:15:04.069169 (+ 220us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":770,"mutex_wait_us":168,"spinlock_wait_cycles":24448,"thread_start_us":565,"threads_started":1}
I20260809 12:15:04.070761 20995 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9882fab77be443f6b78f213d71cc9860" candidate_uuid: "11d77089e61e4fc68c02fe1642510c46" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "663b94b3d1d0498dbfb7a86eb2dc5794" is_pre_election: true
I20260809 12:15:04.070883 20848 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9882fab77be443f6b78f213d71cc9860" candidate_uuid: "11d77089e61e4fc68c02fe1642510c46" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0d0b0cb4a03a4e33b9f37441b0d4a3ef" is_pre_election: true
I20260809 12:15:04.071360 20995 raft_consensus.cc:2454] T 9882fab77be443f6b78f213d71cc9860 P 663b94b3d1d0498dbfb7a86eb2dc5794 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 11d77089e61e4fc68c02fe1642510c46 in term 0.
I20260809 12:15:04.071439 20848 raft_consensus.cc:2454] T 9882fab77be443f6b78f213d71cc9860 P 0d0b0cb4a03a4e33b9f37441b0d4a3ef [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 11d77089e61e4fc68c02fe1642510c46 in term 0.
I20260809 12:15:04.072153 20629 leader_election.cc:302] T 9882fab77be443f6b78f213d71cc9860 P 11d77089e61e4fc68c02fe1642510c46 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 3 yes votes; 0 no votes. yes voters: 0d0b0cb4a03a4e33b9f37441b0d4a3ef, 11d77089e61e4fc68c02fe1642510c46, 663b94b3d1d0498dbfb7a86eb2dc5794; no voters:
I20260809 12:15:04.072576 21071 raft_consensus.cc:2781] T 9882fab77be443f6b78f213d71cc9860 P 11d77089e61e4fc68c02fe1642510c46 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260809 12:15:04.072729 21071 raft_consensus.cc:482] T 9882fab77be443f6b78f213d71cc9860 P 11d77089e61e4fc68c02fe1642510c46 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260809 12:15:04.072851 21071 raft_consensus.cc:3037] T 9882fab77be443f6b78f213d71cc9860 P 11d77089e61e4fc68c02fe1642510c46 [term 0 FOLLOWER]: Advancing to term 1
I20260809 12:15:04.075052 21071 raft_consensus.cc:504] T 9882fab77be443f6b78f213d71cc9860 P 11d77089e61e4fc68c02fe1642510c46 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0d0b0cb4a03a4e33b9f37441b0d4a3ef" member_type: VOTER last_known_addr { host: "127.0.222.2" port: 35123 } } peers { permanent_uuid: "663b94b3d1d0498dbfb7a86eb2dc5794" member_type: VOTER last_known_addr { host: "127.0.222.3" port: 39667 } } peers { permanent_uuid: "11d77089e61e4fc68c02fe1642510c46" member_type: VOTER last_known_addr { host: "127.0.222.1" port: 43611 } }
I20260809 12:15:04.075659 21071 leader_election.cc:288] T 9882fab77be443f6b78f213d71cc9860 P 11d77089e61e4fc68c02fe1642510c46 [CANDIDATE]: Term 1 election: Requested vote from peers 0d0b0cb4a03a4e33b9f37441b0d4a3ef (127.0.222.2:35123), 663b94b3d1d0498dbfb7a86eb2dc5794 (127.0.222.3:39667)
I20260809 12:15:04.076118 20848 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9882fab77be443f6b78f213d71cc9860" candidate_uuid: "11d77089e61e4fc68c02fe1642510c46" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0d0b0cb4a03a4e33b9f37441b0d4a3ef"
I20260809 12:15:04.076138 20995 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9882fab77be443f6b78f213d71cc9860" candidate_uuid: "11d77089e61e4fc68c02fe1642510c46" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "663b94b3d1d0498dbfb7a86eb2dc5794"
I20260809 12:15:04.076328 20848 raft_consensus.cc:3037] T 9882fab77be443f6b78f213d71cc9860 P 0d0b0cb4a03a4e33b9f37441b0d4a3ef [term 0 FOLLOWER]: Advancing to term 1
I20260809 12:15:04.076329 20995 raft_consensus.cc:3037] T 9882fab77be443f6b78f213d71cc9860 P 663b94b3d1d0498dbfb7a86eb2dc5794 [term 0 FOLLOWER]: Advancing to term 1
I20260809 12:15:04.078637 20995 raft_consensus.cc:2454] T 9882fab77be443f6b78f213d71cc9860 P 663b94b3d1d0498dbfb7a86eb2dc5794 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 11d77089e61e4fc68c02fe1642510c46 in term 1.
I20260809 12:15:04.079092 20629 leader_election.cc:302] T 9882fab77be443f6b78f213d71cc9860 P 11d77089e61e4fc68c02fe1642510c46 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 11d77089e61e4fc68c02fe1642510c46, 663b94b3d1d0498dbfb7a86eb2dc5794; no voters:
I20260809 12:15:04.079368 21071 raft_consensus.cc:2781] T 9882fab77be443f6b78f213d71cc9860 P 11d77089e61e4fc68c02fe1642510c46 [term 1 FOLLOWER]: Leader election won for term 1
I20260809 12:15:04.079412 20848 raft_consensus.cc:2454] T 9882fab77be443f6b78f213d71cc9860 P 0d0b0cb4a03a4e33b9f37441b0d4a3ef [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 11d77089e61e4fc68c02fe1642510c46 in term 1.
I20260809 12:15:04.079954 21071 raft_consensus.cc:686] T 9882fab77be443f6b78f213d71cc9860 P 11d77089e61e4fc68c02fe1642510c46 [term 1 LEADER]: Becoming Leader. State: Replica: 11d77089e61e4fc68c02fe1642510c46, State: Running, Role: LEADER
I20260809 12:15:04.080433 21071 consensus_queue.cc:237] T 9882fab77be443f6b78f213d71cc9860 P 11d77089e61e4fc68c02fe1642510c46 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0d0b0cb4a03a4e33b9f37441b0d4a3ef" member_type: VOTER last_known_addr { host: "127.0.222.2" port: 35123 } } peers { permanent_uuid: "663b94b3d1d0498dbfb7a86eb2dc5794" member_type: VOTER last_known_addr { host: "127.0.222.3" port: 39667 } } peers { permanent_uuid: "11d77089e61e4fc68c02fe1642510c46" member_type: VOTER last_known_addr { host: "127.0.222.1" port: 43611 } }
I20260809 12:15:04.085325 20541 catalog_manager.cc:5495] T 9882fab77be443f6b78f213d71cc9860 P 11d77089e61e4fc68c02fe1642510c46 reported cstate change: term changed from 0 to 1, leader changed from <none> to 11d77089e61e4fc68c02fe1642510c46 (127.0.222.1). New cstate: current_term: 1 leader_uuid: "11d77089e61e4fc68c02fe1642510c46" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0d0b0cb4a03a4e33b9f37441b0d4a3ef" member_type: VOTER last_known_addr { host: "127.0.222.2" port: 35123 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "663b94b3d1d0498dbfb7a86eb2dc5794" member_type: VOTER last_known_addr { host: "127.0.222.3" port: 39667 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "11d77089e61e4fc68c02fe1642510c46" member_type: VOTER last_known_addr { host: "127.0.222.1" port: 43611 } health_report { overall_health: HEALTHY } } }
W20260809 12:15:04.111702 20901 tablet.cc:2369] T 9882fab77be443f6b78f213d71cc9860 P 0d0b0cb4a03a4e33b9f37441b0d4a3ef: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260809 12:15:04.124444 21061 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 12:15:04.118657 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59498 (local address 127.0.222.1:43611)
0809 12:15:04.118901 (+ 244us) server_negotiation.cc:206] Beginning negotiation
0809 12:15:04.118907 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0809 12:15:04.119056 (+ 149us) server_negotiation.cc:407] Connection header received
0809 12:15:04.119305 (+ 249us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 12:15:04.119309 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 12:15:04.119393 (+ 84us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 12:15:04.119555 (+ 162us) server_negotiation.cc:226] Negotiated authn=TOKEN
0809 12:15:04.119965 (+ 410us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 12:15:04.120688 (+ 723us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 12:15:04.122101 (+ 1413us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 12:15:04.122514 (+ 413us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 12:15:04.122665 (+ 151us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 12:15:04.123354 (+ 689us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0809 12:15:04.123845 (+ 491us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0809 12:15:04.124037 (+ 192us) server_negotiation.cc:1035] Waiting for connection context
0809 12:15:04.124155 (+ 118us) server_negotiation.cc:299] Negotiation successful
0809 12:15:04.124202 (+ 47us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":80}
I20260809 12:15:04.132964 20848 raft_consensus.cc:1261] T 9882fab77be443f6b78f213d71cc9860 P 0d0b0cb4a03a4e33b9f37441b0d4a3ef [term 1 FOLLOWER]: Refusing update from remote peer 11d77089e61e4fc68c02fe1642510c46: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260809 12:15:04.132964 20995 raft_consensus.cc:1261] T 9882fab77be443f6b78f213d71cc9860 P 663b94b3d1d0498dbfb7a86eb2dc5794 [term 1 FOLLOWER]: Refusing update from remote peer 11d77089e61e4fc68c02fe1642510c46: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260809 12:15:04.134081 21074 consensus_queue.cc:1035] T 9882fab77be443f6b78f213d71cc9860 P 11d77089e61e4fc68c02fe1642510c46 [LEADER]: Connected to new peer: Peer: permanent_uuid: "0d0b0cb4a03a4e33b9f37441b0d4a3ef" member_type: VOTER last_known_addr { host: "127.0.222.2" port: 35123 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260809 12:15:04.134481 21071 consensus_queue.cc:1035] T 9882fab77be443f6b78f213d71cc9860 P 11d77089e61e4fc68c02fe1642510c46 [LEADER]: Connected to new peer: Peer: permanent_uuid: "663b94b3d1d0498dbfb7a86eb2dc5794" member_type: VOTER last_known_addr { host: "127.0.222.3" port: 39667 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260809 12:15:04.146698 21077 mvcc.cc:205] Tried to move back new op lower bound from 7316593476117020672 to 7316593475917504512. Current Snapshot: MvccSnapshot[applied={T|T < 7316593476117020672}]
I20260809 12:15:04.150089 21079 mvcc.cc:205] Tried to move back new op lower bound from 7316593476117020672 to 7316593475917504512. Current Snapshot: MvccSnapshot[applied={T|T < 7316593476117020672}]
I20260809 12:15:04.160600 20541 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:53474:
table { table_name: "test-table" } modify_external_catalogs: true
I20260809 12:15:04.160902 20541 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:53474:
table { table_name: "test-table" } modify_external_catalogs: true
I20260809 12:15:04.165422 20541 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 88d10fc7ebd243ce8011d352fe6d15f0: Sending DeleteTablet for 3 replicas of tablet 9882fab77be443f6b78f213d71cc9860
I20260809 12:15:04.166923 20973 tablet_service.cc:1514] Processing DeleteTablet for tablet 9882fab77be443f6b78f213d71cc9860 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-09 12:15:04 UTC) from {username='kudu', principal='kudu/127.0.222.62@KRBTEST.COM'} at 127.0.0.1:59440
I20260809 12:15:04.167651 20826 tablet_service.cc:1514] Processing DeleteTablet for tablet 9882fab77be443f6b78f213d71cc9860 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-09 12:15:04 UTC) from {username='kudu', principal='kudu/127.0.222.62@KRBTEST.COM'} at 127.0.0.1:51640
I20260809 12:15:04.168469 21089 tablet_replica.cc:330] T 9882fab77be443f6b78f213d71cc9860 P 0d0b0cb4a03a4e33b9f37441b0d4a3ef: stopping tablet replica
I20260809 12:15:04.169006 20681 tablet_service.cc:1514] Processing DeleteTablet for tablet 9882fab77be443f6b78f213d71cc9860 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-09 12:15:04 UTC) from {username='kudu', principal='kudu/127.0.222.62@KRBTEST.COM'} at 127.0.0.1:59496
I20260809 12:15:04.169957 21090 tablet_replica.cc:330] T 9882fab77be443f6b78f213d71cc9860 P 11d77089e61e4fc68c02fe1642510c46: stopping tablet replica
I20260809 12:15:04.170403 21090 raft_consensus.cc:2229] T 9882fab77be443f6b78f213d71cc9860 P 11d77089e61e4fc68c02fe1642510c46 [term 1 LEADER]: Raft consensus shutting down.
I20260809 12:15:04.170584 888 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskgH5Pjm/build/asan/bin/kudu pid 20611
I20260809 12:15:04.170598 21089 raft_consensus.cc:2229] T 9882fab77be443f6b78f213d71cc9860 P 0d0b0cb4a03a4e33b9f37441b0d4a3ef [term 1 FOLLOWER]: Raft consensus shutting down.
I20260809 12:15:04.171351 21090 raft_consensus.cc:2258] T 9882fab77be443f6b78f213d71cc9860 P 11d77089e61e4fc68c02fe1642510c46 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260809 12:15:04.171653 21089 raft_consensus.cc:2258] T 9882fab77be443f6b78f213d71cc9860 P 0d0b0cb4a03a4e33b9f37441b0d4a3ef [term 1 FOLLOWER]: Raft consensus is shut down!
I20260809 12:15:04.174055 21089 ts_tablet_manager.cc:1905] T 9882fab77be443f6b78f213d71cc9860 P 0d0b0cb4a03a4e33b9f37441b0d4a3ef: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260809 12:15:04.178565 21088 tablet_replica.cc:330] T 9882fab77be443f6b78f213d71cc9860 P 663b94b3d1d0498dbfb7a86eb2dc5794: stopping tablet replica
I20260809 12:15:04.179051 21088 raft_consensus.cc:2229] T 9882fab77be443f6b78f213d71cc9860 P 663b94b3d1d0498dbfb7a86eb2dc5794 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260809 12:15:04.179663 21088 raft_consensus.cc:2258] T 9882fab77be443f6b78f213d71cc9860 P 663b94b3d1d0498dbfb7a86eb2dc5794 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260809 12:15:04.181775 21089 ts_tablet_manager.cc:1918] T 9882fab77be443f6b78f213d71cc9860 P 0d0b0cb4a03a4e33b9f37441b0d4a3ef: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260809 12:15:04.181957 21089 log.cc:1196] T 9882fab77be443f6b78f213d71cc9860 P 0d0b0cb4a03a4e33b9f37441b0d4a3ef: Deleting WAL directory at /tmp/dist-test-taskgH5Pjm/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786277382073938-888-0/minicluster-data/ts-1/wal/wals/9882fab77be443f6b78f213d71cc9860
I20260809 12:15:04.182577 21089 ts_tablet_manager.cc:1939] T 9882fab77be443f6b78f213d71cc9860 P 0d0b0cb4a03a4e33b9f37441b0d4a3ef: Deleting consensus metadata
I20260809 12:15:04.184234 21088 ts_tablet_manager.cc:1905] T 9882fab77be443f6b78f213d71cc9860 P 663b94b3d1d0498dbfb7a86eb2dc5794: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260809 12:15:04.184513 20525 catalog_manager.cc:4841] TS 0d0b0cb4a03a4e33b9f37441b0d4a3ef (127.0.222.2:35123): tablet 9882fab77be443f6b78f213d71cc9860 (table test-table [id=9f70105361c842e1975726244d94a22e]) successfully deleted
I20260809 12:15:04.185694 21090 ts_tablet_manager.cc:1905] T 9882fab77be443f6b78f213d71cc9860 P 11d77089e61e4fc68c02fe1642510c46: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260809 12:15:04.191419 21088 ts_tablet_manager.cc:1918] T 9882fab77be443f6b78f213d71cc9860 P 663b94b3d1d0498dbfb7a86eb2dc5794: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260809 12:15:04.191602 21088 log.cc:1196] T 9882fab77be443f6b78f213d71cc9860 P 663b94b3d1d0498dbfb7a86eb2dc5794: Deleting WAL directory at /tmp/dist-test-taskgH5Pjm/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786277382073938-888-0/minicluster-data/ts-2/wal/wals/9882fab77be443f6b78f213d71cc9860
I20260809 12:15:04.192135 21088 ts_tablet_manager.cc:1939] T 9882fab77be443f6b78f213d71cc9860 P 663b94b3d1d0498dbfb7a86eb2dc5794: Deleting consensus metadata
I20260809 12:15:04.193446 21090 ts_tablet_manager.cc:1918] T 9882fab77be443f6b78f213d71cc9860 P 11d77089e61e4fc68c02fe1642510c46: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260809 12:15:04.193650 21090 log.cc:1196] T 9882fab77be443f6b78f213d71cc9860 P 11d77089e61e4fc68c02fe1642510c46: Deleting WAL directory at /tmp/dist-test-taskgH5Pjm/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786277382073938-888-0/minicluster-data/ts-0/wal/wals/9882fab77be443f6b78f213d71cc9860
I20260809 12:15:04.194252 21090 ts_tablet_manager.cc:1939] T 9882fab77be443f6b78f213d71cc9860 P 11d77089e61e4fc68c02fe1642510c46: Deleting consensus metadata
I20260809 12:15:04.194525 20526 catalog_manager.cc:4841] TS 663b94b3d1d0498dbfb7a86eb2dc5794 (127.0.222.3:39667): tablet 9882fab77be443f6b78f213d71cc9860 (table test-table [id=9f70105361c842e1975726244d94a22e]) successfully deleted
I20260809 12:15:04.195909 20526 catalog_manager.cc:4841] TS 11d77089e61e4fc68c02fe1642510c46 (127.0.222.1:43611): tablet 9882fab77be443f6b78f213d71cc9860 (table test-table [id=9f70105361c842e1975726244d94a22e]) successfully deleted
Aug 09 12:15:04 dist-test-slave-2w3w krb5kdc[20495](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786277701, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.222.1@KRBTEST.COM
I20260809 12:15:04.205416 21061 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 12:15:04.172630 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59510 (local address 127.0.222.1:43611)
0809 12:15:04.172820 (+ 190us) server_negotiation.cc:206] Beginning negotiation
0809 12:15:04.172826 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0809 12:15:04.175530 (+ 2704us) server_negotiation.cc:407] Connection header received
0809 12:15:04.175800 (+ 270us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 12:15:04.175805 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 12:15:04.175882 (+ 77us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 12:15:04.175955 (+ 73us) server_negotiation.cc:226] Negotiated authn=SASL
0809 12:15:04.178367 (+ 2412us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 12:15:04.179642 (+ 1275us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 12:15:04.191165 (+ 11523us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 12:15:04.192027 (+ 862us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 12:15:04.192142 (+ 115us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 12:15:04.198131 (+ 5989us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 12:15:04.198152 (+ 21us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 12:15:04.198173 (+ 21us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0809 12:15:04.198222 (+ 49us) server_negotiation.cc:922] calling sasl_server_start()
0809 12:15:04.202254 (+ 4032us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 12:15:04.203071 (+ 817us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 12:15:04.203083 (+ 12us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 12:15:04.203089 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0809 12:15:04.203274 (+ 185us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 12:15:04.203662 (+ 388us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 12:15:04.203667 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 12:15:04.203671 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0809 12:15:04.204169 (+ 498us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 12:15:04.204478 (+ 309us) server_negotiation.cc:1035] Waiting for connection context
0809 12:15:04.204769 (+ 291us) server_negotiation.cc:299] Negotiation successful
0809 12:15:04.205025 (+ 256us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":62}
I20260809 12:15:04.206156 20745 generic_service.cc:196] Checking for leaks (request via RPC)
I20260809 12:15:04.309126 888 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskgH5Pjm/build/asan/bin/kudu with pid 20611
I20260809 12:15:04.331593 888 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskgH5Pjm/build/asan/bin/kudu pid 20758
Aug 09 12:15:04 dist-test-slave-2w3w krb5kdc[20495](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786277701, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.222.2@KRBTEST.COM
I20260809 12:15:04.347283 21059 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 12:15:04.332468 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51644 (local address 127.0.222.2:35123)
0809 12:15:04.332652 (+ 184us) server_negotiation.cc:206] Beginning negotiation
0809 12:15:04.332657 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0809 12:15:04.332677 (+ 20us) server_negotiation.cc:407] Connection header received
0809 12:15:04.332842 (+ 165us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 12:15:04.332846 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 12:15:04.332910 (+ 64us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 12:15:04.333011 (+ 101us) server_negotiation.cc:226] Negotiated authn=SASL
0809 12:15:04.334314 (+ 1303us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 12:15:04.335146 (+ 832us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 12:15:04.336452 (+ 1306us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 12:15:04.336829 (+ 377us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 12:15:04.336919 (+ 90us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 12:15:04.341378 (+ 4459us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 12:15:04.341394 (+ 16us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 12:15:04.341411 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0809 12:15:04.341447 (+ 36us) server_negotiation.cc:922] calling sasl_server_start()
0809 12:15:04.344203 (+ 2756us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 12:15:04.344880 (+ 677us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 12:15:04.344887 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 12:15:04.344890 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0809 12:15:04.345019 (+ 129us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 12:15:04.345412 (+ 393us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 12:15:04.345416 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 12:15:04.345417 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0809 12:15:04.345926 (+ 509us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 12:15:04.346183 (+ 257us) server_negotiation.cc:1035] Waiting for connection context
0809 12:15:04.346523 (+ 340us) server_negotiation.cc:299] Negotiation successful
0809 12:15:04.346703 (+ 180us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":68}
I20260809 12:15:04.347986 20894 generic_service.cc:196] Checking for leaks (request via RPC)
I20260809 12:15:04.439090 888 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskgH5Pjm/build/asan/bin/kudu with pid 20758
I20260809 12:15:04.459149 888 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskgH5Pjm/build/asan/bin/kudu pid 20907
Aug 09 12:15:04 dist-test-slave-2w3w krb5kdc[20495](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786277701, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.222.3@KRBTEST.COM
I20260809 12:15:04.475456 21060 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 12:15:04.459922 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59452 (local address 127.0.222.3:39667)
0809 12:15:04.460147 (+ 225us) server_negotiation.cc:206] Beginning negotiation
0809 12:15:04.460153 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0809 12:15:04.460227 (+ 74us) server_negotiation.cc:407] Connection header received
0809 12:15:04.460455 (+ 228us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 12:15:04.460460 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 12:15:04.460539 (+ 79us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 12:15:04.460663 (+ 124us) server_negotiation.cc:226] Negotiated authn=SASL
0809 12:15:04.462036 (+ 1373us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 12:15:04.462779 (+ 743us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 12:15:04.464222 (+ 1443us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 12:15:04.464635 (+ 413us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 12:15:04.464779 (+ 144us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 12:15:04.469383 (+ 4604us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 12:15:04.469400 (+ 17us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 12:15:04.469417 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0809 12:15:04.469452 (+ 35us) server_negotiation.cc:922] calling sasl_server_start()
0809 12:15:04.472435 (+ 2983us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 12:15:04.473186 (+ 751us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 12:15:04.473203 (+ 17us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 12:15:04.473205 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0809 12:15:04.473344 (+ 139us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 12:15:04.473758 (+ 414us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 12:15:04.473764 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 12:15:04.473767 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0809 12:15:04.474135 (+ 368us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 12:15:04.474452 (+ 317us) server_negotiation.cc:1035] Waiting for connection context
0809 12:15:04.474788 (+ 336us) server_negotiation.cc:299] Negotiation successful
0809 12:15:04.475061 (+ 273us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":82}
I20260809 12:15:04.476210 21039 generic_service.cc:196] Checking for leaks (request via RPC)
I20260809 12:15:04.566119 888 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskgH5Pjm/build/asan/bin/kudu with pid 20907
I20260809 12:15:04.585600 888 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskgH5Pjm/build/asan/bin/kudu pid 20508
I20260809 12:15:04.586817 20578 generic_service.cc:196] Checking for leaks (request via RPC)
I20260809 12:15:04.700309 888 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskgH5Pjm/build/asan/bin/kudu with pid 20508
2026-08-09T12:15:04Z chronyd exiting
[ OK ] EncryptionPolicyTest.TestEncryptionPolicy/2 (4508 ms)
[ RUN ] EncryptionPolicyTest.TestEncryptionPolicy/3
2026-08-09T12:15:04Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-09T12:15:04Z Disabled control of system clock
I20260809 12:15:04.796078 888 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskgH5Pjm/build/asan/bin/kudu
/tmp/dist-test-taskgH5Pjm/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskgH5Pjm/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786277382073938-888-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskgH5Pjm/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786277382073938-888-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskgH5Pjm/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786277382073938-888-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskgH5Pjm/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786277382073938-888-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.222.62:40975
--webserver_interface=127.0.222.62
--webserver_port=0
--builtin_ntp_servers=127.0.222.20:39351
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.222.62:40975
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20260809 12:15:05.154433 21100 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260809 12:15:05.154778 21100 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260809 12:15:05.154886 21100 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260809 12:15:05.163201 21100 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260809 12:15:05.163327 21100 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260809 12:15:05.163372 21100 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260809 12:15:05.163431 21100 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260809 12:15:05.163473 21100 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260809 12:15:05.174073 21100 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.222.20:39351
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskgH5Pjm/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786277382073938-888-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskgH5Pjm/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786277382073938-888-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.222.62:40975
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.222.62:40975
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskgH5Pjm/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786277382073938-888-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.222.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskgH5Pjm/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786277382073938-888-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 09 Aug 2026 11:06:41 UTC on ccycloud.quasar-bwoogy.root.comops.site
build id 4575991
ASAN enabled
I20260809 12:15:05.175519 21100 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260809 12:15:05.177558 21100 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260809 12:15:05.190033 21105 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 12:15:05.190048 21106 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 12:15:05.190281 21108 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260809 12:15:05.191363 21100 server_base.cc:1034] running on GCE node
I20260809 12:15:05.192369 21100 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260809 12:15:05.193977 21100 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260809 12:15:05.195195 21100 hybrid_clock.cc:648] HybridClock initialized: now 1786277705195131 us; error 62 us; skew 500 ppm
I20260809 12:15:05.195616 21100 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260809 12:15:05.198323 21100 webserver.cc:460] Webserver started at http://127.0.222.62:42209/ using document root <none> and password file <none>
I20260809 12:15:05.199179 21100 fs_manager.cc:359] Metadata directory not provided
I20260809 12:15:05.199316 21100 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260809 12:15:05.199699 21100 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260809 12:15:05.203125 21100 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskgH5Pjm/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786277382073938-888-0/minicluster-data/master-0/data/instance:
uuid: "17228f8b9e32445fa92080e7b2d6d9a7"
format_stamp: "Formatted at 2026-08-09 12:15:05 on dist-test-slave-2w3w"
server_key: "ac074026388648710628bb1aed88e826"
server_key_iv: "8768c061c885b6d334be658f37c438ab"
server_key_version: "encryptionkey@0"
I20260809 12:15:05.204054 21100 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskgH5Pjm/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786277382073938-888-0/minicluster-data/master-0/wal/instance:
uuid: "17228f8b9e32445fa92080e7b2d6d9a7"
format_stamp: "Formatted at 2026-08-09 12:15:05 on dist-test-slave-2w3w"
server_key: "ac074026388648710628bb1aed88e826"
server_key_iv: "8768c061c885b6d334be658f37c438ab"
server_key_version: "encryptionkey@0"
I20260809 12:15:05.210052 21100 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.001s sys 0.005s
I20260809 12:15:05.213956 21114 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260809 12:15:05.215571 21100 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.002s
I20260809 12:15:05.215790 21100 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskgH5Pjm/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786277382073938-888-0/minicluster-data/master-0/data,/tmp/dist-test-taskgH5Pjm/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786277382073938-888-0/minicluster-data/master-0/wal
uuid: "17228f8b9e32445fa92080e7b2d6d9a7"
format_stamp: "Formatted at 2026-08-09 12:15:05 on dist-test-slave-2w3w"
server_key: "ac074026388648710628bb1aed88e826"
server_key_iv: "8768c061c885b6d334be658f37c438ab"
server_key_version: "encryptionkey@0"
I20260809 12:15:05.216051 21100 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskgH5Pjm/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786277382073938-888-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskgH5Pjm/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786277382073938-888-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskgH5Pjm/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786277382073938-888-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260809 12:15:05.234589 21100 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260809 12:15:05.235682 21100 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260809 12:15:05.236039 21100 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260809 12:15:05.256747 21100 rpc_server.cc:307] RPC server started. Bound to: 127.0.222.62:40975
I20260809 12:15:05.256767 21167 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.222.62:40975 every 8 connection(s)
I20260809 12:15:05.258327 21100 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskgH5Pjm/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786277382073938-888-0/minicluster-data/master-0/data/info.pb
I20260809 12:15:05.258735 888 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskgH5Pjm/build/asan/bin/kudu as pid 21100
I20260809 12:15:05.258985 888 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskgH5Pjm/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786277382073938-888-0/minicluster-data/master-0/wal/instance
I20260809 12:15:05.259750 888 external_mini_cluster.cc:1428] Setting key 862d6a0c12ac625b2c029130c7a2c20c
I20260809 12:15:05.262948 21168 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260809 12:15:05.274757 21173 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 12:15:05.261913 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42460 (local address 127.0.222.62:40975)
0809 12:15:05.263145 (+ 1232us) server_negotiation.cc:206] Beginning negotiation
0809 12:15:05.263160 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0809 12:15:05.263219 (+ 59us) server_negotiation.cc:407] Connection header received
0809 12:15:05.264986 (+ 1767us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 12:15:05.265007 (+ 21us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 12:15:05.265543 (+ 536us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 12:15:05.265953 (+ 410us) server_negotiation.cc:226] Negotiated authn=SASL
0809 12:15:05.266592 (+ 639us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 12:15:05.268277 (+ 1685us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 12:15:05.270330 (+ 2053us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 12:15:05.271027 (+ 697us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 12:15:05.271171 (+ 144us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 12:15:05.272433 (+ 1262us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 12:15:05.272441 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 12:15:05.272454 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0809 12:15:05.273002 (+ 548us) server_negotiation.cc:922] calling sasl_server_start()
0809 12:15:05.273076 (+ 74us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0809 12:15:05.273097 (+ 21us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 12:15:05.273189 (+ 92us) server_negotiation.cc:1035] Waiting for connection context
0809 12:15:05.273448 (+ 259us) server_negotiation.cc:299] Negotiation successful
0809 12:15:05.273814 (+ 366us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":683,"thread_start_us":450,"threads_started":1}
I20260809 12:15:05.276152 21168 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 17228f8b9e32445fa92080e7b2d6d9a7: Bootstrap starting.
I20260809 12:15:05.279479 21168 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 17228f8b9e32445fa92080e7b2d6d9a7: Neither blocks nor log segments found. Creating new log.
I20260809 12:15:05.280463 21168 log.cc:824] T 00000000000000000000000000000000 P 17228f8b9e32445fa92080e7b2d6d9a7: Log is configured to *not* fsync() on all Append() calls
I20260809 12:15:05.283573 21168 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 17228f8b9e32445fa92080e7b2d6d9a7: No bootstrap required, opened a new log
I20260809 12:15:05.291445 21168 raft_consensus.cc:348] T 00000000000000000000000000000000 P 17228f8b9e32445fa92080e7b2d6d9a7 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "17228f8b9e32445fa92080e7b2d6d9a7" member_type: VOTER last_known_addr { host: "127.0.222.62" port: 40975 } }
I20260809 12:15:05.291800 21168 raft_consensus.cc:374] T 00000000000000000000000000000000 P 17228f8b9e32445fa92080e7b2d6d9a7 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260809 12:15:05.291878 21168 raft_consensus.cc:729] T 00000000000000000000000000000000 P 17228f8b9e32445fa92080e7b2d6d9a7 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 17228f8b9e32445fa92080e7b2d6d9a7, State: Initialized, Role: FOLLOWER
I20260809 12:15:05.292505 21168 consensus_queue.cc:260] T 00000000000000000000000000000000 P 17228f8b9e32445fa92080e7b2d6d9a7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "17228f8b9e32445fa92080e7b2d6d9a7" member_type: VOTER last_known_addr { host: "127.0.222.62" port: 40975 } }
I20260809 12:15:05.292723 21168 raft_consensus.cc:388] T 00000000000000000000000000000000 P 17228f8b9e32445fa92080e7b2d6d9a7 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260809 12:15:05.292825 21168 raft_consensus.cc:482] T 00000000000000000000000000000000 P 17228f8b9e32445fa92080e7b2d6d9a7 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260809 12:15:05.292963 21168 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 17228f8b9e32445fa92080e7b2d6d9a7 [term 0 FOLLOWER]: Advancing to term 1
I20260809 12:15:05.295184 21168 raft_consensus.cc:504] T 00000000000000000000000000000000 P 17228f8b9e32445fa92080e7b2d6d9a7 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "17228f8b9e32445fa92080e7b2d6d9a7" member_type: VOTER last_known_addr { host: "127.0.222.62" port: 40975 } }
I20260809 12:15:05.295849 21168 leader_election.cc:302] T 00000000000000000000000000000000 P 17228f8b9e32445fa92080e7b2d6d9a7 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 17228f8b9e32445fa92080e7b2d6d9a7; no voters:
I20260809 12:15:05.296713 21168 leader_election.cc:288] T 00000000000000000000000000000000 P 17228f8b9e32445fa92080e7b2d6d9a7 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260809 12:15:05.296837 21175 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 17228f8b9e32445fa92080e7b2d6d9a7 [term 1 FOLLOWER]: Leader election won for term 1
I20260809 12:15:05.297564 21175 raft_consensus.cc:686] T 00000000000000000000000000000000 P 17228f8b9e32445fa92080e7b2d6d9a7 [term 1 LEADER]: Becoming Leader. State: Replica: 17228f8b9e32445fa92080e7b2d6d9a7, State: Running, Role: LEADER
I20260809 12:15:05.298295 21175 consensus_queue.cc:237] T 00000000000000000000000000000000 P 17228f8b9e32445fa92080e7b2d6d9a7 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "17228f8b9e32445fa92080e7b2d6d9a7" member_type: VOTER last_known_addr { host: "127.0.222.62" port: 40975 } }
I20260809 12:15:05.300858 21168 sys_catalog.cc:564] T 00000000000000000000000000000000 P 17228f8b9e32445fa92080e7b2d6d9a7 [sys.catalog]: configured and running, proceeding with master startup.
I20260809 12:15:05.301616 21177 sys_catalog.cc:455] T 00000000000000000000000000000000 P 17228f8b9e32445fa92080e7b2d6d9a7 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "17228f8b9e32445fa92080e7b2d6d9a7" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "17228f8b9e32445fa92080e7b2d6d9a7" member_type: VOTER last_known_addr { host: "127.0.222.62" port: 40975 } } }
I20260809 12:15:05.301663 21176 sys_catalog.cc:455] T 00000000000000000000000000000000 P 17228f8b9e32445fa92080e7b2d6d9a7 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 17228f8b9e32445fa92080e7b2d6d9a7. Latest consensus state: current_term: 1 leader_uuid: "17228f8b9e32445fa92080e7b2d6d9a7" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "17228f8b9e32445fa92080e7b2d6d9a7" member_type: VOTER last_known_addr { host: "127.0.222.62" port: 40975 } } }
I20260809 12:15:05.302067 21177 sys_catalog.cc:458] T 00000000000000000000000000000000 P 17228f8b9e32445fa92080e7b2d6d9a7 [sys.catalog]: This master's current role is: LEADER
I20260809 12:15:05.302848 21176 sys_catalog.cc:458] T 00000000000000000000000000000000 P 17228f8b9e32445fa92080e7b2d6d9a7 [sys.catalog]: This master's current role is: LEADER
I20260809 12:15:05.307364 21180 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260809 12:15:05.312996 21180 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260809 12:15:05.322549 21180 catalog_manager.cc:1347] Generated new cluster ID: f9fa5cc5da7240c2bc8ef6f12aff700e
I20260809 12:15:05.322680 21180 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260809 12:15:05.353413 21180 catalog_manager.cc:1370] Generated new certificate authority record
I20260809 12:15:05.354779 21180 catalog_manager.cc:1504] Loading token signing keys...
I20260809 12:15:05.361487 21180 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 17228f8b9e32445fa92080e7b2d6d9a7: Generated new TSK 0
I20260809 12:15:05.362367 21180 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260809 12:15:05.378351 888 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskgH5Pjm/build/asan/bin/kudu
/tmp/dist-test-taskgH5Pjm/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskgH5Pjm/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786277382073938-888-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskgH5Pjm/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786277382073938-888-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskgH5Pjm/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786277382073938-888-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskgH5Pjm/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786277382073938-888-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.222.1:0
--local_ip_for_outbound_sockets=127.0.222.1
--webserver_interface=127.0.222.1
--webserver_port=0
--tserver_master_addrs=127.0.222.62:40975
--builtin_ntp_servers=127.0.222.20:39351
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20260809 12:15:05.734151 21194 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260809 12:15:05.734440 21194 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260809 12:15:05.734530 21194 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260809 12:15:05.741767 21194 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260809 12:15:05.741873 21194 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260809 12:15:05.741981 21194 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.222.1
I20260809 12:15:05.750974 21194 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.222.20:39351
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskgH5Pjm/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786277382073938-888-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskgH5Pjm/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786277382073938-888-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.222.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskgH5Pjm/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786277382073938-888-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.222.1
--webserver_port=0
--tserver_master_addrs=127.0.222.62:40975
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.222.1
--log_dir=/tmp/dist-test-taskgH5Pjm/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786277382073938-888-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 09 Aug 2026 11:06:41 UTC on ccycloud.quasar-bwoogy.root.comops.site
build id 4575991
ASAN enabled
I20260809 12:15:05.752197 21194 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260809 12:15:05.753923 21194 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260809 12:15:05.767102 21200 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 12:15:05.767148 21203 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 12:15:05.767194 21201 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260809 12:15:05.768096 21194 server_base.cc:1034] running on GCE node
I20260809 12:15:05.768644 21194 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260809 12:15:05.769508 21194 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260809 12:15:05.770709 21194 hybrid_clock.cc:648] HybridClock initialized: now 1786277705770674 us; error 31 us; skew 500 ppm
I20260809 12:15:05.771099 21194 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260809 12:15:05.773296 21194 webserver.cc:460] Webserver started at http://127.0.222.1:37361/ using document root <none> and password file <none>
I20260809 12:15:05.773983 21194 fs_manager.cc:359] Metadata directory not provided
I20260809 12:15:05.774071 21194 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260809 12:15:05.774428 21194 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260809 12:15:05.777381 21194 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskgH5Pjm/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786277382073938-888-0/minicluster-data/ts-0/data/instance:
uuid: "574166c798ce434c837b3cc142394423"
format_stamp: "Formatted at 2026-08-09 12:15:05 on dist-test-slave-2w3w"
server_key: "bcf12a6f92731159dd059f15b7b66e2c"
server_key_iv: "5ccec754e8364b77c630c79abfe0ff69"
server_key_version: "encryptionkey@0"
I20260809 12:15:05.778213 21194 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskgH5Pjm/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786277382073938-888-0/minicluster-data/ts-0/wal/instance:
uuid: "574166c798ce434c837b3cc142394423"
format_stamp: "Formatted at 2026-08-09 12:15:05 on dist-test-slave-2w3w"
server_key: "bcf12a6f92731159dd059f15b7b66e2c"
server_key_iv: "5ccec754e8364b77c630c79abfe0ff69"
server_key_version: "encryptionkey@0"
I20260809 12:15:05.783447 21194 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.002s sys 0.005s
I20260809 12:15:05.787074 21209 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260809 12:15:05.788380 21194 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260809 12:15:05.788558 21194 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskgH5Pjm/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786277382073938-888-0/minicluster-data/ts-0/data,/tmp/dist-test-taskgH5Pjm/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786277382073938-888-0/minicluster-data/ts-0/wal
uuid: "574166c798ce434c837b3cc142394423"
format_stamp: "Formatted at 2026-08-09 12:15:05 on dist-test-slave-2w3w"
server_key: "bcf12a6f92731159dd059f15b7b66e2c"
server_key_iv: "5ccec754e8364b77c630c79abfe0ff69"
server_key_version: "encryptionkey@0"
I20260809 12:15:05.788779 21194 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskgH5Pjm/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786277382073938-888-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskgH5Pjm/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786277382073938-888-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskgH5Pjm/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786277382073938-888-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260809 12:15:05.816707 21194 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260809 12:15:05.817615 21194 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260809 12:15:05.817917 21194 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260809 12:15:05.819034 21194 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260809 12:15:05.820698 21194 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260809 12:15:05.820833 21194 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260809 12:15:05.820982 21194 ts_tablet_manager.cc:610] Registered 0 tablets
I20260809 12:15:05.821074 21194 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260809 12:15:05.857936 21194 rpc_server.cc:307] RPC server started. Bound to: 127.0.222.1:44697
I20260809 12:15:05.857996 21333 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.222.1:44697 every 8 connection(s)
I20260809 12:15:05.859411 21194 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskgH5Pjm/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786277382073938-888-0/minicluster-data/ts-0/data/info.pb
I20260809 12:15:05.867412 888 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskgH5Pjm/build/asan/bin/kudu as pid 21194
I20260809 12:15:05.867635 888 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskgH5Pjm/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786277382073938-888-0/minicluster-data/ts-0/wal/instance
I20260809 12:15:05.868223 888 external_mini_cluster.cc:1428] Setting key 96db0045b8593b73f72fb53f9d9c4406
I20260809 12:15:05.871404 888 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskgH5Pjm/build/asan/bin/kudu
/tmp/dist-test-taskgH5Pjm/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskgH5Pjm/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786277382073938-888-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskgH5Pjm/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786277382073938-888-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskgH5Pjm/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786277382073938-888-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskgH5Pjm/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786277382073938-888-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.222.2:0
--local_ip_for_outbound_sockets=127.0.222.2
--webserver_interface=127.0.222.2
--webserver_port=0
--tserver_master_addrs=127.0.222.62:40975
--builtin_ntp_servers=127.0.222.20:39351
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
I20260809 12:15:05.874048 21336 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 12:15:05.862238 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.222.1:47965 (local address 127.0.222.62:40975)
0809 12:15:05.862707 (+ 469us) server_negotiation.cc:206] Beginning negotiation
0809 12:15:05.862712 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0809 12:15:05.864038 (+ 1326us) server_negotiation.cc:407] Connection header received
0809 12:15:05.866091 (+ 2053us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 12:15:05.866097 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 12:15:05.866207 (+ 110us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 12:15:05.866350 (+ 143us) server_negotiation.cc:226] Negotiated authn=SASL
0809 12:15:05.867958 (+ 1608us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 12:15:05.868749 (+ 791us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 12:15:05.871058 (+ 2309us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 12:15:05.871471 (+ 413us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 12:15:05.871654 (+ 183us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 12:15:05.873141 (+ 1487us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 12:15:05.873146 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 12:15:05.873149 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0809 12:15:05.873166 (+ 17us) server_negotiation.cc:922] calling sasl_server_start()
0809 12:15:05.873207 (+ 41us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0809 12:15:05.873219 (+ 12us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 12:15:05.873330 (+ 111us) server_negotiation.cc:1035] Waiting for connection context
0809 12:15:05.873588 (+ 258us) server_negotiation.cc:299] Negotiation successful
0809 12:15:05.873691 (+ 103us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":350,"thread_start_us":237,"threads_started":1}
I20260809 12:15:05.875098 21337 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 12:15:05.862839 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.222.62:40975 (local address 127.0.222.1:47965)
0809 12:15:05.863867 (+ 1028us) negotiation.cc:107] Waiting for socket to connect
0809 12:15:05.863913 (+ 46us) client_negotiation.cc:174] Beginning negotiation
0809 12:15:05.865795 (+ 1882us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 12:15:05.866607 (+ 812us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 12:15:05.866616 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 12:15:05.866751 (+ 135us) client_negotiation.cc:189] Negotiated authn=SASL
0809 12:15:05.867716 (+ 965us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 12:15:05.867747 (+ 31us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 12:15:05.868945 (+ 1198us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 12:15:05.868949 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 12:15:05.870860 (+ 1911us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 12:15:05.870876 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 12:15:05.871812 (+ 936us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 12:15:05.871833 (+ 21us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 12:15:05.872041 (+ 208us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 12:15:05.872813 (+ 772us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0809 12:15:05.872816 (+ 3us) client_negotiation.cc:656] calling sasl_client_start()
0809 12:15:05.872860 (+ 44us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0809 12:15:05.872936 (+ 76us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0809 12:15:05.873333 (+ 397us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0809 12:15:05.873348 (+ 15us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0809 12:15:05.873357 (+ 9us) client_negotiation.cc:769] Sending connection context
0809 12:15:05.873565 (+ 208us) client_negotiation.cc:240] Negotiation successful
0809 12:15:05.873952 (+ 387us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":569,"thread_start_us":308,"threads_started":1}
I20260809 12:15:05.876720 21334 heartbeater.cc:347] Connected to a master server at 127.0.222.62:40975
I20260809 12:15:05.877068 21334 heartbeater.cc:464] Registering TS with master...
I20260809 12:15:05.878065 21334 heartbeater.cc:511] Master 127.0.222.62:40975 requested a full tablet report, sending...
I20260809 12:15:05.880622 21132 ts_manager.cc:194] Registered new tserver with Master: 574166c798ce434c837b3cc142394423 (127.0.222.1:44697)
I20260809 12:15:05.883039 21132 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.222.1:47965
W20260809 12:15:06.405884 21338 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260809 12:15:06.406448 21338 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260809 12:15:06.406670 21338 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260809 12:15:06.417701 21338 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260809 12:15:06.417924 21338 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260809 12:15:06.418190 21338 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.222.2
I20260809 12:15:06.432060 21338 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.222.20:39351
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskgH5Pjm/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786277382073938-888-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskgH5Pjm/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786277382073938-888-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.222.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskgH5Pjm/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786277382073938-888-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.222.2
--webserver_port=0
--tserver_master_addrs=127.0.222.62:40975
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.222.2
--log_dir=/tmp/dist-test-taskgH5Pjm/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786277382073938-888-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 09 Aug 2026 11:06:41 UTC on ccycloud.quasar-bwoogy.root.comops.site
build id 4575991
ASAN enabled
I20260809 12:15:06.434324 21338 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260809 12:15:06.437333 21338 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260809 12:15:06.456979 21344 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 12:15:06.457262 21343 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 12:15:06.457229 21346 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260809 12:15:06.458588 21338 server_base.cc:1034] running on GCE node
I20260809 12:15:06.459573 21338 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260809 12:15:06.461175 21338 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260809 12:15:06.462450 21338 hybrid_clock.cc:648] HybridClock initialized: now 1786277706462370 us; error 55 us; skew 500 ppm
I20260809 12:15:06.463093 21338 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260809 12:15:06.466763 21338 webserver.cc:460] Webserver started at http://127.0.222.2:40303/ using document root <none> and password file <none>
I20260809 12:15:06.467872 21338 fs_manager.cc:359] Metadata directory not provided
I20260809 12:15:06.468014 21338 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260809 12:15:06.468555 21338 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260809 12:15:06.473884 21338 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskgH5Pjm/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786277382073938-888-0/minicluster-data/ts-1/data/instance:
uuid: "6e6dcda253eb40db9054aba46933006d"
format_stamp: "Formatted at 2026-08-09 12:15:06 on dist-test-slave-2w3w"
server_key: "08015fc1cbcebb604e9b1d7d3f56a705"
server_key_iv: "855266467bc71fb3250c4d8c20572b79"
server_key_version: "encryptionkey@0"
I20260809 12:15:06.475167 21338 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskgH5Pjm/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786277382073938-888-0/minicluster-data/ts-1/wal/instance:
uuid: "6e6dcda253eb40db9054aba46933006d"
format_stamp: "Formatted at 2026-08-09 12:15:06 on dist-test-slave-2w3w"
server_key: "08015fc1cbcebb604e9b1d7d3f56a705"
server_key_iv: "855266467bc71fb3250c4d8c20572b79"
server_key_version: "encryptionkey@0"
I20260809 12:15:06.485473 21338 fs_manager.cc:691] Time spent creating directory manager: real 0.010s user 0.004s sys 0.004s
I20260809 12:15:06.492161 21352 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260809 12:15:06.494573 21338 fs_manager.cc:725] Time spent opening block manager: real 0.005s user 0.004s sys 0.001s
I20260809 12:15:06.494777 21338 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskgH5Pjm/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786277382073938-888-0/minicluster-data/ts-1/data,/tmp/dist-test-taskgH5Pjm/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786277382073938-888-0/minicluster-data/ts-1/wal
uuid: "6e6dcda253eb40db9054aba46933006d"
format_stamp: "Formatted at 2026-08-09 12:15:06 on dist-test-slave-2w3w"
server_key: "08015fc1cbcebb604e9b1d7d3f56a705"
server_key_iv: "855266467bc71fb3250c4d8c20572b79"
server_key_version: "encryptionkey@0"
I20260809 12:15:06.495187 21338 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskgH5Pjm/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786277382073938-888-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskgH5Pjm/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786277382073938-888-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskgH5Pjm/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786277382073938-888-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260809 12:15:06.515291 21338 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260809 12:15:06.516681 21338 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260809 12:15:06.517136 21338 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260809 12:15:06.518842 21338 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260809 12:15:06.521960 21338 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260809 12:15:06.522194 21338 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260809 12:15:06.522332 21338 ts_tablet_manager.cc:610] Registered 0 tablets
I20260809 12:15:06.522428 21338 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260809 12:15:06.578329 21338 rpc_server.cc:307] RPC server started. Bound to: 127.0.222.2:44221
I20260809 12:15:06.578392 21474 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.222.2:44221 every 8 connection(s)
I20260809 12:15:06.580582 21338 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskgH5Pjm/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786277382073938-888-0/minicluster-data/ts-1/data/info.pb
I20260809 12:15:06.583272 888 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskgH5Pjm/build/asan/bin/kudu as pid 21338
I20260809 12:15:06.583585 888 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskgH5Pjm/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786277382073938-888-0/minicluster-data/ts-1/wal/instance
I20260809 12:15:06.584408 888 external_mini_cluster.cc:1428] Setting key 222b75ebe1e4914a64b13757157c8d2f
I20260809 12:15:06.589551 888 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskgH5Pjm/build/asan/bin/kudu
/tmp/dist-test-taskgH5Pjm/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskgH5Pjm/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786277382073938-888-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskgH5Pjm/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786277382073938-888-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskgH5Pjm/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786277382073938-888-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskgH5Pjm/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786277382073938-888-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.222.3:0
--local_ip_for_outbound_sockets=127.0.222.3
--webserver_interface=127.0.222.3
--webserver_port=0
--tserver_master_addrs=127.0.222.62:40975
--builtin_ntp_servers=127.0.222.20:39351
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
I20260809 12:15:06.604069 21478 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 12:15:06.584803 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.222.2:44027 (local address 127.0.222.62:40975)
0809 12:15:06.585647 (+ 844us) server_negotiation.cc:206] Beginning negotiation
0809 12:15:06.585655 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0809 12:15:06.588235 (+ 2580us) server_negotiation.cc:407] Connection header received
0809 12:15:06.591597 (+ 3362us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 12:15:06.591618 (+ 21us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 12:15:06.591794 (+ 176us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 12:15:06.592025 (+ 231us) server_negotiation.cc:226] Negotiated authn=SASL
0809 12:15:06.594248 (+ 2223us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 12:15:06.596174 (+ 1926us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 12:15:06.598635 (+ 2461us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 12:15:06.600013 (+ 1378us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 12:15:06.600318 (+ 305us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 12:15:06.602693 (+ 2375us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 12:15:06.602700 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 12:15:06.602703 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0809 12:15:06.602736 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0809 12:15:06.602798 (+ 62us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0809 12:15:06.602828 (+ 30us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 12:15:06.603053 (+ 225us) server_negotiation.cc:1035] Waiting for connection context
0809 12:15:06.603472 (+ 419us) server_negotiation.cc:299] Negotiation successful
0809 12:15:06.603595 (+ 123us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":595,"thread_start_us":395,"threads_started":1}
I20260809 12:15:06.606897 21479 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 12:15:06.585937 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.222.62:40975 (local address 127.0.222.2:44027)
0809 12:15:06.587965 (+ 2028us) negotiation.cc:107] Waiting for socket to connect
0809 12:15:06.588057 (+ 92us) client_negotiation.cc:174] Beginning negotiation
0809 12:15:06.591171 (+ 3114us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 12:15:06.592547 (+ 1376us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 12:15:06.592559 (+ 12us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 12:15:06.592716 (+ 157us) client_negotiation.cc:189] Negotiated authn=SASL
0809 12:15:06.593976 (+ 1260us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 12:15:06.594013 (+ 37us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 12:15:06.596395 (+ 2382us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 12:15:06.596400 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 12:15:06.598419 (+ 2019us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 12:15:06.598438 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 12:15:06.600371 (+ 1933us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 12:15:06.600383 (+ 12us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 12:15:06.600833 (+ 450us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 12:15:06.602019 (+ 1186us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0809 12:15:06.602024 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0809 12:15:06.602097 (+ 73us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0809 12:15:06.602369 (+ 272us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0809 12:15:06.603059 (+ 690us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0809 12:15:06.603085 (+ 26us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0809 12:15:06.603111 (+ 26us) client_negotiation.cc:769] Sending connection context
0809 12:15:06.603472 (+ 361us) client_negotiation.cc:240] Negotiation successful
0809 12:15:06.604472 (+ 1000us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":990,"thread_start_us":601,"threads_started":1}
I20260809 12:15:06.609041 21476 heartbeater.cc:347] Connected to a master server at 127.0.222.62:40975
I20260809 12:15:06.609555 21476 heartbeater.cc:464] Registering TS with master...
I20260809 12:15:06.611066 21476 heartbeater.cc:511] Master 127.0.222.62:40975 requested a full tablet report, sending...
I20260809 12:15:06.613523 21132 ts_manager.cc:194] Registered new tserver with Master: 6e6dcda253eb40db9054aba46933006d (127.0.222.2:44221)
I20260809 12:15:06.615618 21132 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.222.2:44027
I20260809 12:15:06.887568 21334 heartbeater.cc:503] Master 127.0.222.62:40975 was elected leader, sending a full tablet report...
W20260809 12:15:07.277827 21480 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260809 12:15:07.278368 21480 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260809 12:15:07.278713 21480 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260809 12:15:07.292826 21480 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260809 12:15:07.292992 21480 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260809 12:15:07.293375 21480 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.222.3
I20260809 12:15:07.312090 21480 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.222.20:39351
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskgH5Pjm/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786277382073938-888-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskgH5Pjm/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786277382073938-888-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.222.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskgH5Pjm/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786277382073938-888-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.222.3
--webserver_port=0
--tserver_master_addrs=127.0.222.62:40975
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.222.3
--log_dir=/tmp/dist-test-taskgH5Pjm/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786277382073938-888-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 09 Aug 2026 11:06:41 UTC on ccycloud.quasar-bwoogy.root.comops.site
build id 4575991
ASAN enabled
I20260809 12:15:07.313951 21480 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260809 12:15:07.315850 21480 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260809 12:15:07.329659 21486 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 12:15:07.329718 21485 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 12:15:07.329762 21488 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260809 12:15:07.332639 21480 server_base.cc:1034] running on GCE node
I20260809 12:15:07.333344 21480 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260809 12:15:07.334502 21480 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260809 12:15:07.335753 21480 hybrid_clock.cc:648] HybridClock initialized: now 1786277707335705 us; error 35 us; skew 500 ppm
I20260809 12:15:07.336120 21480 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260809 12:15:07.338590 21480 webserver.cc:460] Webserver started at http://127.0.222.3:40673/ using document root <none> and password file <none>
I20260809 12:15:07.339385 21480 fs_manager.cc:359] Metadata directory not provided
I20260809 12:15:07.339509 21480 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260809 12:15:07.339874 21480 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260809 12:15:07.342855 21480 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskgH5Pjm/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786277382073938-888-0/minicluster-data/ts-2/data/instance:
uuid: "014d4c8580a640a18402dd826220c63d"
format_stamp: "Formatted at 2026-08-09 12:15:07 on dist-test-slave-2w3w"
server_key: "b2f5aec8bcb6916d4433adf7cb67b5bf"
server_key_iv: "cf102298d8f8817649c1820325cb3898"
server_key_version: "encryptionkey@0"
I20260809 12:15:07.343782 21480 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskgH5Pjm/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786277382073938-888-0/minicluster-data/ts-2/wal/instance:
uuid: "014d4c8580a640a18402dd826220c63d"
format_stamp: "Formatted at 2026-08-09 12:15:07 on dist-test-slave-2w3w"
server_key: "b2f5aec8bcb6916d4433adf7cb67b5bf"
server_key_iv: "cf102298d8f8817649c1820325cb3898"
server_key_version: "encryptionkey@0"
I20260809 12:15:07.348992 21480 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.000s
I20260809 12:15:07.352474 21494 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260809 12:15:07.353971 21480 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.000s
I20260809 12:15:07.354197 21480 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskgH5Pjm/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786277382073938-888-0/minicluster-data/ts-2/data,/tmp/dist-test-taskgH5Pjm/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786277382073938-888-0/minicluster-data/ts-2/wal
uuid: "014d4c8580a640a18402dd826220c63d"
format_stamp: "Formatted at 2026-08-09 12:15:07 on dist-test-slave-2w3w"
server_key: "b2f5aec8bcb6916d4433adf7cb67b5bf"
server_key_iv: "cf102298d8f8817649c1820325cb3898"
server_key_version: "encryptionkey@0"
I20260809 12:15:07.354467 21480 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskgH5Pjm/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786277382073938-888-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskgH5Pjm/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786277382073938-888-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskgH5Pjm/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786277382073938-888-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260809 12:15:07.378027 21480 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260809 12:15:07.379076 21480 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260809 12:15:07.379400 21480 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260809 12:15:07.380543 21480 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260809 12:15:07.382167 21480 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260809 12:15:07.382323 21480 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260809 12:15:07.382476 21480 ts_tablet_manager.cc:610] Registered 0 tablets
I20260809 12:15:07.382579 21480 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260809 12:15:07.418470 21480 rpc_server.cc:307] RPC server started. Bound to: 127.0.222.3:33877
I20260809 12:15:07.418565 21618 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.222.3:33877 every 8 connection(s)
I20260809 12:15:07.419932 21480 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskgH5Pjm/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786277382073938-888-0/minicluster-data/ts-2/data/info.pb
I20260809 12:15:07.430168 888 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskgH5Pjm/build/asan/bin/kudu as pid 21480
I20260809 12:15:07.430357 888 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskgH5Pjm/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786277382073938-888-0/minicluster-data/ts-2/wal/instance
I20260809 12:15:07.430845 888 external_mini_cluster.cc:1428] Setting key 98df84e2969cbb476e1987dde14d9f95
I20260809 12:15:07.433147 21622 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 12:15:07.422615 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.222.3:51883 (local address 127.0.222.62:40975)
0809 12:15:07.423176 (+ 561us) server_negotiation.cc:206] Beginning negotiation
0809 12:15:07.423181 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0809 12:15:07.424326 (+ 1145us) server_negotiation.cc:407] Connection header received
0809 12:15:07.426141 (+ 1815us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 12:15:07.426148 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 12:15:07.426247 (+ 99us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 12:15:07.426369 (+ 122us) server_negotiation.cc:226] Negotiated authn=SASL
0809 12:15:07.427494 (+ 1125us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 12:15:07.428284 (+ 790us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 12:15:07.429918 (+ 1634us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 12:15:07.430384 (+ 466us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 12:15:07.430534 (+ 150us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 12:15:07.432095 (+ 1561us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 12:15:07.432100 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 12:15:07.432104 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0809 12:15:07.432122 (+ 18us) server_negotiation.cc:922] calling sasl_server_start()
0809 12:15:07.432197 (+ 75us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0809 12:15:07.432218 (+ 21us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 12:15:07.432322 (+ 104us) server_negotiation.cc:1035] Waiting for connection context
0809 12:15:07.432604 (+ 282us) server_negotiation.cc:299] Negotiation successful
0809 12:15:07.432709 (+ 105us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":448,"thread_start_us":265,"threads_started":1}
I20260809 12:15:07.434135 21623 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 12:15:07.423265 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.222.62:40975 (local address 127.0.222.3:51883)
0809 12:15:07.424167 (+ 902us) negotiation.cc:107] Waiting for socket to connect
0809 12:15:07.424202 (+ 35us) client_negotiation.cc:174] Beginning negotiation
0809 12:15:07.425870 (+ 1668us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 12:15:07.426600 (+ 730us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 12:15:07.426608 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 12:15:07.426691 (+ 83us) client_negotiation.cc:189] Negotiated authn=SASL
0809 12:15:07.427318 (+ 627us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 12:15:07.427336 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 12:15:07.428455 (+ 1119us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 12:15:07.428459 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 12:15:07.429744 (+ 1285us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 12:15:07.429756 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 12:15:07.430622 (+ 866us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 12:15:07.430631 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 12:15:07.430870 (+ 239us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 12:15:07.431687 (+ 817us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0809 12:15:07.431691 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0809 12:15:07.431755 (+ 64us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0809 12:15:07.431846 (+ 91us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0809 12:15:07.432333 (+ 487us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0809 12:15:07.432344 (+ 11us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0809 12:15:07.432353 (+ 9us) client_negotiation.cc:769] Sending connection context
0809 12:15:07.432573 (+ 220us) client_negotiation.cc:240] Negotiation successful
0809 12:15:07.433026 (+ 453us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":474,"thread_start_us":294,"threads_started":1}
I20260809 12:15:07.435580 21619 heartbeater.cc:347] Connected to a master server at 127.0.222.62:40975
I20260809 12:15:07.435927 21619 heartbeater.cc:464] Registering TS with master...
I20260809 12:15:07.436726 21619 heartbeater.cc:511] Master 127.0.222.62:40975 requested a full tablet report, sending...
I20260809 12:15:07.438120 21132 ts_manager.cc:194] Registered new tserver with Master: 014d4c8580a640a18402dd826220c63d (127.0.222.3:33877)
I20260809 12:15:07.438923 21132 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.222.3:51883
I20260809 12:15:07.446671 888 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260809 12:15:07.457042 21622 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 12:15:07.451029 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42466 (local address 127.0.222.62:40975)
0809 12:15:07.451258 (+ 229us) server_negotiation.cc:206] Beginning negotiation
0809 12:15:07.451268 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0809 12:15:07.451635 (+ 367us) server_negotiation.cc:407] Connection header received
0809 12:15:07.451748 (+ 113us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 12:15:07.451753 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 12:15:07.451828 (+ 75us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 12:15:07.451921 (+ 93us) server_negotiation.cc:226] Negotiated authn=SASL
0809 12:15:07.452435 (+ 514us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 12:15:07.453172 (+ 737us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 12:15:07.454523 (+ 1351us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 12:15:07.455056 (+ 533us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 12:15:07.455166 (+ 110us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 12:15:07.456278 (+ 1112us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 12:15:07.456282 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 12:15:07.456285 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0809 12:15:07.456303 (+ 18us) server_negotiation.cc:922] calling sasl_server_start()
0809 12:15:07.456333 (+ 30us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0809 12:15:07.456345 (+ 12us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 12:15:07.456460 (+ 115us) server_negotiation.cc:1035] Waiting for connection context
0809 12:15:07.456630 (+ 170us) server_negotiation.cc:299] Negotiation successful
0809 12:15:07.456710 (+ 80us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":61}
I20260809 12:15:07.461256 21132 catalog_manager.cc:1908] Servicing CreateTable request from {username='slave'} at 127.0.0.1:42466:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260809 12:15:07.468307 21132 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260809 12:15:07.494691 21636 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 12:15:07.482023 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38090 (local address 127.0.222.2:44221)
0809 12:15:07.483206 (+ 1183us) server_negotiation.cc:206] Beginning negotiation
0809 12:15:07.483220 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0809 12:15:07.484114 (+ 894us) server_negotiation.cc:407] Connection header received
0809 12:15:07.485061 (+ 947us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 12:15:07.485070 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 12:15:07.485347 (+ 277us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 12:15:07.485537 (+ 190us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0809 12:15:07.487094 (+ 1557us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 12:15:07.488583 (+ 1489us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 12:15:07.491778 (+ 3195us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 12:15:07.493175 (+ 1397us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 12:15:07.493319 (+ 144us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 12:15:07.493566 (+ 247us) server_negotiation.cc:1035] Waiting for connection context
0809 12:15:07.494162 (+ 596us) server_negotiation.cc:299] Negotiation successful
0809 12:15:07.494342 (+ 180us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":955,"thread_start_us":927,"threads_started":1}
I20260809 12:15:07.495077 21639 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 12:15:07.485944 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.222.1:44697 (local address 127.0.0.1:35782)
0809 12:15:07.487091 (+ 1147us) negotiation.cc:107] Waiting for socket to connect
0809 12:15:07.487107 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0809 12:15:07.487237 (+ 130us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 12:15:07.487837 (+ 600us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 12:15:07.487842 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 12:15:07.487867 (+ 25us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0809 12:15:07.488175 (+ 308us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 12:15:07.488186 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 12:15:07.490067 (+ 1881us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 12:15:07.490073 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 12:15:07.492093 (+ 2020us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 12:15:07.492105 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 12:15:07.494274 (+ 2169us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 12:15:07.494279 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 12:15:07.494402 (+ 123us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 12:15:07.494423 (+ 21us) client_negotiation.cc:769] Sending connection context
0809 12:15:07.494534 (+ 111us) client_negotiation.cc:240] Negotiation successful
0809 12:15:07.494667 (+ 133us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":995,"thread_start_us":827,"threads_started":1}
I20260809 12:15:07.495177 21638 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 12:15:07.485111 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35782 (local address 127.0.222.1:44697)
0809 12:15:07.486369 (+ 1258us) server_negotiation.cc:206] Beginning negotiation
0809 12:15:07.486379 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0809 12:15:07.487223 (+ 844us) server_negotiation.cc:407] Connection header received
0809 12:15:07.487377 (+ 154us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 12:15:07.487384 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 12:15:07.487636 (+ 252us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 12:15:07.487827 (+ 191us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0809 12:15:07.488364 (+ 537us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 12:15:07.489910 (+ 1546us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 12:15:07.492271 (+ 2361us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 12:15:07.493722 (+ 1451us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 12:15:07.493860 (+ 138us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 12:15:07.494156 (+ 296us) server_negotiation.cc:1035] Waiting for connection context
0809 12:15:07.494581 (+ 425us) server_negotiation.cc:299] Negotiation successful
0809 12:15:07.494769 (+ 188us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1073,"thread_start_us":1000,"threads_started":1}
I20260809 12:15:07.495450 21637 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 12:15:07.482614 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53374 (local address 127.0.222.3:33877)
0809 12:15:07.484706 (+ 2092us) server_negotiation.cc:206] Beginning negotiation
0809 12:15:07.484713 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0809 12:15:07.484749 (+ 36us) server_negotiation.cc:407] Connection header received
0809 12:15:07.485178 (+ 429us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 12:15:07.485188 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 12:15:07.485496 (+ 308us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 12:15:07.485719 (+ 223us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0809 12:15:07.487103 (+ 1384us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 12:15:07.488710 (+ 1607us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 12:15:07.491356 (+ 2646us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 12:15:07.492879 (+ 1523us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 12:15:07.493027 (+ 148us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 12:15:07.493326 (+ 299us) server_negotiation.cc:1035] Waiting for connection context
0809 12:15:07.494845 (+ 1519us) server_negotiation.cc:299] Negotiation successful
0809 12:15:07.495041 (+ 196us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1931,"thread_start_us":1769,"threads_started":1}
I20260809 12:15:07.494705 21634 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 12:15:07.481381 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.222.3:33877 (local address 127.0.0.1:53374)
0809 12:15:07.484584 (+ 3203us) negotiation.cc:107] Waiting for socket to connect
0809 12:15:07.484602 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0809 12:15:07.484941 (+ 339us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 12:15:07.486016 (+ 1075us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 12:15:07.486026 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 12:15:07.486070 (+ 44us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0809 12:15:07.486873 (+ 803us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 12:15:07.486886 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 12:15:07.488901 (+ 2015us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 12:15:07.488907 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 12:15:07.491112 (+ 2205us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 12:15:07.491129 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 12:15:07.493460 (+ 2331us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 12:15:07.493466 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 12:15:07.493987 (+ 521us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 12:15:07.494017 (+ 30us) client_negotiation.cc:769] Sending connection context
0809 12:15:07.494088 (+ 71us) client_negotiation.cc:240] Negotiation successful
0809 12:15:07.494303 (+ 215us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":3072,"mutex_wait_us":84,"spinlock_wait_cycles":768,"thread_start_us":2038,"threads_started":1}
I20260809 12:15:07.496196 21635 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 12:15:07.481031 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.222.2:44221 (local address 127.0.0.1:38090)
0809 12:15:07.483924 (+ 2893us) negotiation.cc:107] Waiting for socket to connect
0809 12:15:07.483990 (+ 66us) client_negotiation.cc:174] Beginning negotiation
0809 12:15:07.484748 (+ 758us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 12:15:07.485905 (+ 1157us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 12:15:07.485924 (+ 19us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 12:15:07.485962 (+ 38us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0809 12:15:07.486779 (+ 817us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 12:15:07.486800 (+ 21us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 12:15:07.488756 (+ 1956us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 12:15:07.488761 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 12:15:07.491573 (+ 2812us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 12:15:07.491594 (+ 21us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 12:15:07.493730 (+ 2136us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 12:15:07.493735 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 12:15:07.493877 (+ 142us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 12:15:07.493937 (+ 60us) client_negotiation.cc:769] Sending connection context
0809 12:15:07.495638 (+ 1701us) client_negotiation.cc:240] Negotiation successful
0809 12:15:07.495776 (+ 138us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":2554,"mutex_wait_us":46,"spinlock_wait_cycles":12544,"thread_start_us":3287,"threads_started":1}
I20260809 12:15:07.499205 21262 tablet_service.cc:1467] Processing CreateTablet for tablet 11323aa901c0491a82e399f6e6ab79cc (DEFAULT_TABLE table=test-table [id=2db15243448f4c29b5a0e791f8b09f1b]), partition=RANGE (key) PARTITION UNBOUNDED
I20260809 12:15:07.499650 21548 tablet_service.cc:1467] Processing CreateTablet for tablet 11323aa901c0491a82e399f6e6ab79cc (DEFAULT_TABLE table=test-table [id=2db15243448f4c29b5a0e791f8b09f1b]), partition=RANGE (key) PARTITION UNBOUNDED
I20260809 12:15:07.500713 21262 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 11323aa901c0491a82e399f6e6ab79cc. 1 dirs total, 0 dirs full, 0 dirs failed
I20260809 12:15:07.500475 21405 tablet_service.cc:1467] Processing CreateTablet for tablet 11323aa901c0491a82e399f6e6ab79cc (DEFAULT_TABLE table=test-table [id=2db15243448f4c29b5a0e791f8b09f1b]), partition=RANGE (key) PARTITION UNBOUNDED
I20260809 12:15:07.501196 21548 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 11323aa901c0491a82e399f6e6ab79cc. 1 dirs total, 0 dirs full, 0 dirs failed
I20260809 12:15:07.501758 21405 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 11323aa901c0491a82e399f6e6ab79cc. 1 dirs total, 0 dirs full, 0 dirs failed
I20260809 12:15:07.514921 21640 tablet_bootstrap.cc:492] T 11323aa901c0491a82e399f6e6ab79cc P 014d4c8580a640a18402dd826220c63d: Bootstrap starting.
I20260809 12:15:07.517083 21641 tablet_bootstrap.cc:492] T 11323aa901c0491a82e399f6e6ab79cc P 574166c798ce434c837b3cc142394423: Bootstrap starting.
I20260809 12:15:07.518401 21642 tablet_bootstrap.cc:492] T 11323aa901c0491a82e399f6e6ab79cc P 6e6dcda253eb40db9054aba46933006d: Bootstrap starting.
I20260809 12:15:07.519193 21640 tablet_bootstrap.cc:654] T 11323aa901c0491a82e399f6e6ab79cc P 014d4c8580a640a18402dd826220c63d: Neither blocks nor log segments found. Creating new log.
I20260809 12:15:07.520607 21640 log.cc:824] T 11323aa901c0491a82e399f6e6ab79cc P 014d4c8580a640a18402dd826220c63d: Log is configured to *not* fsync() on all Append() calls
I20260809 12:15:07.521327 21641 tablet_bootstrap.cc:654] T 11323aa901c0491a82e399f6e6ab79cc P 574166c798ce434c837b3cc142394423: Neither blocks nor log segments found. Creating new log.
I20260809 12:15:07.522153 21642 tablet_bootstrap.cc:654] T 11323aa901c0491a82e399f6e6ab79cc P 6e6dcda253eb40db9054aba46933006d: Neither blocks nor log segments found. Creating new log.
I20260809 12:15:07.522552 21641 log.cc:824] T 11323aa901c0491a82e399f6e6ab79cc P 574166c798ce434c837b3cc142394423: Log is configured to *not* fsync() on all Append() calls
I20260809 12:15:07.523551 21642 log.cc:824] T 11323aa901c0491a82e399f6e6ab79cc P 6e6dcda253eb40db9054aba46933006d: Log is configured to *not* fsync() on all Append() calls
I20260809 12:15:07.526645 21640 tablet_bootstrap.cc:492] T 11323aa901c0491a82e399f6e6ab79cc P 014d4c8580a640a18402dd826220c63d: No bootstrap required, opened a new log
I20260809 12:15:07.526858 21642 tablet_bootstrap.cc:492] T 11323aa901c0491a82e399f6e6ab79cc P 6e6dcda253eb40db9054aba46933006d: No bootstrap required, opened a new log
I20260809 12:15:07.526953 21640 ts_tablet_manager.cc:1397] T 11323aa901c0491a82e399f6e6ab79cc P 014d4c8580a640a18402dd826220c63d: Time spent bootstrapping tablet: real 0.012s user 0.000s sys 0.008s
I20260809 12:15:07.527136 21641 tablet_bootstrap.cc:492] T 11323aa901c0491a82e399f6e6ab79cc P 574166c798ce434c837b3cc142394423: No bootstrap required, opened a new log
I20260809 12:15:07.527151 21642 ts_tablet_manager.cc:1397] T 11323aa901c0491a82e399f6e6ab79cc P 6e6dcda253eb40db9054aba46933006d: Time spent bootstrapping tablet: real 0.009s user 0.005s sys 0.003s
I20260809 12:15:07.527406 21641 ts_tablet_manager.cc:1397] T 11323aa901c0491a82e399f6e6ab79cc P 574166c798ce434c837b3cc142394423: Time spent bootstrapping tablet: real 0.011s user 0.005s sys 0.004s
I20260809 12:15:07.534209 21640 raft_consensus.cc:348] T 11323aa901c0491a82e399f6e6ab79cc P 014d4c8580a640a18402dd826220c63d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e6dcda253eb40db9054aba46933006d" member_type: VOTER last_known_addr { host: "127.0.222.2" port: 44221 } } peers { permanent_uuid: "014d4c8580a640a18402dd826220c63d" member_type: VOTER last_known_addr { host: "127.0.222.3" port: 33877 } } peers { permanent_uuid: "574166c798ce434c837b3cc142394423" member_type: VOTER last_known_addr { host: "127.0.222.1" port: 44697 } }
I20260809 12:15:07.534627 21640 raft_consensus.cc:374] T 11323aa901c0491a82e399f6e6ab79cc P 014d4c8580a640a18402dd826220c63d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260809 12:15:07.534734 21640 raft_consensus.cc:729] T 11323aa901c0491a82e399f6e6ab79cc P 014d4c8580a640a18402dd826220c63d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 014d4c8580a640a18402dd826220c63d, State: Initialized, Role: FOLLOWER
I20260809 12:15:07.535462 21640 consensus_queue.cc:260] T 11323aa901c0491a82e399f6e6ab79cc P 014d4c8580a640a18402dd826220c63d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e6dcda253eb40db9054aba46933006d" member_type: VOTER last_known_addr { host: "127.0.222.2" port: 44221 } } peers { permanent_uuid: "014d4c8580a640a18402dd826220c63d" member_type: VOTER last_known_addr { host: "127.0.222.3" port: 33877 } } peers { permanent_uuid: "574166c798ce434c837b3cc142394423" member_type: VOTER last_known_addr { host: "127.0.222.1" port: 44697 } }
I20260809 12:15:07.536597 21619 heartbeater.cc:503] Master 127.0.222.62:40975 was elected leader, sending a full tablet report...
I20260809 12:15:07.537313 21640 ts_tablet_manager.cc:1428] T 11323aa901c0491a82e399f6e6ab79cc P 014d4c8580a640a18402dd826220c63d: Time spent starting tablet: real 0.010s user 0.006s sys 0.003s
I20260809 12:15:07.537796 21642 raft_consensus.cc:348] T 11323aa901c0491a82e399f6e6ab79cc P 6e6dcda253eb40db9054aba46933006d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e6dcda253eb40db9054aba46933006d" member_type: VOTER last_known_addr { host: "127.0.222.2" port: 44221 } } peers { permanent_uuid: "014d4c8580a640a18402dd826220c63d" member_type: VOTER last_known_addr { host: "127.0.222.3" port: 33877 } } peers { permanent_uuid: "574166c798ce434c837b3cc142394423" member_type: VOTER last_known_addr { host: "127.0.222.1" port: 44697 } }
I20260809 12:15:07.538228 21642 raft_consensus.cc:374] T 11323aa901c0491a82e399f6e6ab79cc P 6e6dcda253eb40db9054aba46933006d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260809 12:15:07.538344 21642 raft_consensus.cc:729] T 11323aa901c0491a82e399f6e6ab79cc P 6e6dcda253eb40db9054aba46933006d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6e6dcda253eb40db9054aba46933006d, State: Initialized, Role: FOLLOWER
I20260809 12:15:07.538692 21641 raft_consensus.cc:348] T 11323aa901c0491a82e399f6e6ab79cc P 574166c798ce434c837b3cc142394423 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e6dcda253eb40db9054aba46933006d" member_type: VOTER last_known_addr { host: "127.0.222.2" port: 44221 } } peers { permanent_uuid: "014d4c8580a640a18402dd826220c63d" member_type: VOTER last_known_addr { host: "127.0.222.3" port: 33877 } } peers { permanent_uuid: "574166c798ce434c837b3cc142394423" member_type: VOTER last_known_addr { host: "127.0.222.1" port: 44697 } }
I20260809 12:15:07.539139 21641 raft_consensus.cc:374] T 11323aa901c0491a82e399f6e6ab79cc P 574166c798ce434c837b3cc142394423 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260809 12:15:07.538936 21642 consensus_queue.cc:260] T 11323aa901c0491a82e399f6e6ab79cc P 6e6dcda253eb40db9054aba46933006d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e6dcda253eb40db9054aba46933006d" member_type: VOTER last_known_addr { host: "127.0.222.2" port: 44221 } } peers { permanent_uuid: "014d4c8580a640a18402dd826220c63d" member_type: VOTER last_known_addr { host: "127.0.222.3" port: 33877 } } peers { permanent_uuid: "574166c798ce434c837b3cc142394423" member_type: VOTER last_known_addr { host: "127.0.222.1" port: 44697 } }
I20260809 12:15:07.539239 21641 raft_consensus.cc:729] T 11323aa901c0491a82e399f6e6ab79cc P 574166c798ce434c837b3cc142394423 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 574166c798ce434c837b3cc142394423, State: Initialized, Role: FOLLOWER
I20260809 12:15:07.539839 21641 consensus_queue.cc:260] T 11323aa901c0491a82e399f6e6ab79cc P 574166c798ce434c837b3cc142394423 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e6dcda253eb40db9054aba46933006d" member_type: VOTER last_known_addr { host: "127.0.222.2" port: 44221 } } peers { permanent_uuid: "014d4c8580a640a18402dd826220c63d" member_type: VOTER last_known_addr { host: "127.0.222.3" port: 33877 } } peers { permanent_uuid: "574166c798ce434c837b3cc142394423" member_type: VOTER last_known_addr { host: "127.0.222.1" port: 44697 } }
I20260809 12:15:07.540768 21642 ts_tablet_manager.cc:1428] T 11323aa901c0491a82e399f6e6ab79cc P 6e6dcda253eb40db9054aba46933006d: Time spent starting tablet: real 0.014s user 0.005s sys 0.008s
I20260809 12:15:07.541044 21476 heartbeater.cc:503] Master 127.0.222.62:40975 was elected leader, sending a full tablet report...
I20260809 12:15:07.541576 21641 ts_tablet_manager.cc:1428] T 11323aa901c0491a82e399f6e6ab79cc P 574166c798ce434c837b3cc142394423: Time spent starting tablet: real 0.014s user 0.006s sys 0.007s
W20260809 12:15:07.584043 21477 tablet.cc:2369] T 11323aa901c0491a82e399f6e6ab79cc P 6e6dcda253eb40db9054aba46933006d: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260809 12:15:07.613140 21335 tablet.cc:2369] T 11323aa901c0491a82e399f6e6ab79cc P 574166c798ce434c837b3cc142394423: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260809 12:15:07.671936 21620 tablet.cc:2369] T 11323aa901c0491a82e399f6e6ab79cc P 014d4c8580a640a18402dd826220c63d: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260809 12:15:07.828147 21647 raft_consensus.cc:482] T 11323aa901c0491a82e399f6e6ab79cc P 6e6dcda253eb40db9054aba46933006d [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260809 12:15:07.828603 21647 raft_consensus.cc:504] T 11323aa901c0491a82e399f6e6ab79cc P 6e6dcda253eb40db9054aba46933006d [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e6dcda253eb40db9054aba46933006d" member_type: VOTER last_known_addr { host: "127.0.222.2" port: 44221 } } peers { permanent_uuid: "014d4c8580a640a18402dd826220c63d" member_type: VOTER last_known_addr { host: "127.0.222.3" port: 33877 } } peers { permanent_uuid: "574166c798ce434c837b3cc142394423" member_type: VOTER last_known_addr { host: "127.0.222.1" port: 44697 } }
I20260809 12:15:07.830531 21647 leader_election.cc:288] T 11323aa901c0491a82e399f6e6ab79cc P 6e6dcda253eb40db9054aba46933006d [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 014d4c8580a640a18402dd826220c63d (127.0.222.3:33877), 574166c798ce434c837b3cc142394423 (127.0.222.1:44697)
I20260809 12:15:07.833187 21648 raft_consensus.cc:482] T 11323aa901c0491a82e399f6e6ab79cc P 574166c798ce434c837b3cc142394423 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260809 12:15:07.833536 21648 raft_consensus.cc:504] T 11323aa901c0491a82e399f6e6ab79cc P 574166c798ce434c837b3cc142394423 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e6dcda253eb40db9054aba46933006d" member_type: VOTER last_known_addr { host: "127.0.222.2" port: 44221 } } peers { permanent_uuid: "014d4c8580a640a18402dd826220c63d" member_type: VOTER last_known_addr { host: "127.0.222.3" port: 33877 } } peers { permanent_uuid: "574166c798ce434c837b3cc142394423" member_type: VOTER last_known_addr { host: "127.0.222.1" port: 44697 } }
I20260809 12:15:07.835573 21648 leader_election.cc:288] T 11323aa901c0491a82e399f6e6ab79cc P 574166c798ce434c837b3cc142394423 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 6e6dcda253eb40db9054aba46933006d (127.0.222.2:44221), 014d4c8580a640a18402dd826220c63d (127.0.222.3:33877)
I20260809 12:15:07.841615 21637 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 12:15:07.831050 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.222.2:38059 (local address 127.0.222.3:33877)
0809 12:15:07.831271 (+ 221us) server_negotiation.cc:206] Beginning negotiation
0809 12:15:07.831279 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0809 12:15:07.832092 (+ 813us) server_negotiation.cc:407] Connection header received
0809 12:15:07.832311 (+ 219us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 12:15:07.832316 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 12:15:07.832406 (+ 90us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 12:15:07.832475 (+ 69us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0809 12:15:07.833370 (+ 895us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 12:15:07.834503 (+ 1133us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 12:15:07.838220 (+ 3717us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 12:15:07.840030 (+ 1810us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 12:15:07.840162 (+ 132us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 12:15:07.840325 (+ 163us) server_negotiation.cc:1035] Waiting for connection context
0809 12:15:07.841041 (+ 716us) server_negotiation.cc:299] Negotiation successful
0809 12:15:07.841236 (+ 195us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":74}
I20260809 12:15:07.843065 21649 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 12:15:07.830819 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.222.3:33877 (local address 127.0.222.2:38059)
0809 12:15:07.831933 (+ 1114us) negotiation.cc:107] Waiting for socket to connect
0809 12:15:07.831957 (+ 24us) client_negotiation.cc:174] Beginning negotiation
0809 12:15:07.832146 (+ 189us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 12:15:07.832621 (+ 475us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 12:15:07.832628 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 12:15:07.832660 (+ 32us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0809 12:15:07.833184 (+ 524us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 12:15:07.833199 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 12:15:07.834655 (+ 1456us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 12:15:07.834661 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 12:15:07.837919 (+ 3258us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 12:15:07.837941 (+ 22us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 12:15:07.840547 (+ 2606us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 12:15:07.840556 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 12:15:07.840799 (+ 243us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 12:15:07.840853 (+ 54us) client_negotiation.cc:769] Sending connection context
0809 12:15:07.842247 (+ 1394us) client_negotiation.cc:240] Negotiation successful
0809 12:15:07.842456 (+ 209us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":961,"mutex_wait_us":90,"thread_start_us":645,"threads_started":1}
I20260809 12:15:07.844532 21570 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "11323aa901c0491a82e399f6e6ab79cc" candidate_uuid: "6e6dcda253eb40db9054aba46933006d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "014d4c8580a640a18402dd826220c63d" is_pre_election: true
I20260809 12:15:07.845245 21570 raft_consensus.cc:2454] T 11323aa901c0491a82e399f6e6ab79cc P 014d4c8580a640a18402dd826220c63d [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6e6dcda253eb40db9054aba46933006d in term 0.
I20260809 12:15:07.846334 21354 leader_election.cc:302] T 11323aa901c0491a82e399f6e6ab79cc P 6e6dcda253eb40db9054aba46933006d [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 014d4c8580a640a18402dd826220c63d, 6e6dcda253eb40db9054aba46933006d; no voters:
I20260809 12:15:07.846897 21647 raft_consensus.cc:2781] T 11323aa901c0491a82e399f6e6ab79cc P 6e6dcda253eb40db9054aba46933006d [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260809 12:15:07.847088 21647 raft_consensus.cc:482] T 11323aa901c0491a82e399f6e6ab79cc P 6e6dcda253eb40db9054aba46933006d [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260809 12:15:07.847211 21647 raft_consensus.cc:3037] T 11323aa901c0491a82e399f6e6ab79cc P 6e6dcda253eb40db9054aba46933006d [term 0 FOLLOWER]: Advancing to term 1
I20260809 12:15:07.849025 21638 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 12:15:07.831314 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.222.2:53411 (local address 127.0.222.1:44697)
0809 12:15:07.831544 (+ 230us) server_negotiation.cc:206] Beginning negotiation
0809 12:15:07.831552 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0809 12:15:07.832155 (+ 603us) server_negotiation.cc:407] Connection header received
0809 12:15:07.832384 (+ 229us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 12:15:07.832388 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 12:15:07.832593 (+ 205us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 12:15:07.832692 (+ 99us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0809 12:15:07.833516 (+ 824us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 12:15:07.834602 (+ 1086us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 12:15:07.845704 (+ 11102us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 12:15:07.847354 (+ 1650us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 12:15:07.847484 (+ 130us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 12:15:07.847621 (+ 137us) server_negotiation.cc:1035] Waiting for connection context
0809 12:15:07.848607 (+ 986us) server_negotiation.cc:299] Negotiation successful
0809 12:15:07.848758 (+ 151us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":81}
I20260809 12:15:07.849852 21650 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 12:15:07.831084 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.222.1:44697 (local address 127.0.222.2:53411)
0809 12:15:07.831984 (+ 900us) negotiation.cc:107] Waiting for socket to connect
0809 12:15:07.831999 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0809 12:15:07.832184 (+ 185us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 12:15:07.832717 (+ 533us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 12:15:07.832724 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 12:15:07.832746 (+ 22us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0809 12:15:07.833265 (+ 519us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 12:15:07.833297 (+ 32us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 12:15:07.834719 (+ 1422us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 12:15:07.834724 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 12:15:07.843508 (+ 8784us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 12:15:07.843530 (+ 22us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 12:15:07.848033 (+ 4503us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 12:15:07.848040 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 12:15:07.848401 (+ 361us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 12:15:07.848440 (+ 39us) client_negotiation.cc:769] Sending connection context
0809 12:15:07.849293 (+ 853us) client_negotiation.cc:240] Negotiation successful
0809 12:15:07.849442 (+ 149us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":799,"mutex_wait_us":205,"thread_start_us":629,"threads_started":1}
I20260809 12:15:07.850981 21283 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "11323aa901c0491a82e399f6e6ab79cc" candidate_uuid: "6e6dcda253eb40db9054aba46933006d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "574166c798ce434c837b3cc142394423" is_pre_election: true
I20260809 12:15:07.851619 21283 raft_consensus.cc:2454] T 11323aa901c0491a82e399f6e6ab79cc P 574166c798ce434c837b3cc142394423 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6e6dcda253eb40db9054aba46933006d in term 0.
I20260809 12:15:07.854056 21647 raft_consensus.cc:504] T 11323aa901c0491a82e399f6e6ab79cc P 6e6dcda253eb40db9054aba46933006d [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e6dcda253eb40db9054aba46933006d" member_type: VOTER last_known_addr { host: "127.0.222.2" port: 44221 } } peers { permanent_uuid: "014d4c8580a640a18402dd826220c63d" member_type: VOTER last_known_addr { host: "127.0.222.3" port: 33877 } } peers { permanent_uuid: "574166c798ce434c837b3cc142394423" member_type: VOTER last_known_addr { host: "127.0.222.1" port: 44697 } }
I20260809 12:15:07.855355 21283 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "11323aa901c0491a82e399f6e6ab79cc" candidate_uuid: "6e6dcda253eb40db9054aba46933006d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "574166c798ce434c837b3cc142394423"
I20260809 12:15:07.855443 21570 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "11323aa901c0491a82e399f6e6ab79cc" candidate_uuid: "6e6dcda253eb40db9054aba46933006d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "014d4c8580a640a18402dd826220c63d"
I20260809 12:15:07.855564 21283 raft_consensus.cc:3037] T 11323aa901c0491a82e399f6e6ab79cc P 574166c798ce434c837b3cc142394423 [term 0 FOLLOWER]: Advancing to term 1
I20260809 12:15:07.855672 21570 raft_consensus.cc:3037] T 11323aa901c0491a82e399f6e6ab79cc P 014d4c8580a640a18402dd826220c63d [term 0 FOLLOWER]: Advancing to term 1
I20260809 12:15:07.856626 21637 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 12:15:07.835885 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.222.1:53685 (local address 127.0.222.3:33877)
0809 12:15:07.841909 (+ 6024us) server_negotiation.cc:206] Beginning negotiation
0809 12:15:07.841917 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0809 12:15:07.848729 (+ 6812us) server_negotiation.cc:407] Connection header received
0809 12:15:07.849002 (+ 273us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 12:15:07.849009 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 12:15:07.849087 (+ 78us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 12:15:07.849153 (+ 66us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0809 12:15:07.849944 (+ 791us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 12:15:07.851091 (+ 1147us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 12:15:07.854262 (+ 3171us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 12:15:07.855383 (+ 1121us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 12:15:07.855512 (+ 129us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 12:15:07.855612 (+ 100us) server_negotiation.cc:1035] Waiting for connection context
0809 12:15:07.856194 (+ 582us) server_negotiation.cc:299] Negotiation successful
0809 12:15:07.856331 (+ 137us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":5893,"thread_start_us":15981,"threads_started":1}
I20260809 12:15:07.856832 21636 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 12:15:07.836478 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.222.1:59071 (local address 127.0.222.2:44221)
0809 12:15:07.836714 (+ 236us) server_negotiation.cc:206] Beginning negotiation
0809 12:15:07.836722 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0809 12:15:07.847063 (+ 10341us) server_negotiation.cc:407] Connection header received
0809 12:15:07.848045 (+ 982us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 12:15:07.848050 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 12:15:07.848144 (+ 94us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 12:15:07.848208 (+ 64us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0809 12:15:07.850905 (+ 2697us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 12:15:07.851976 (+ 1071us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 12:15:07.854455 (+ 2479us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 12:15:07.855677 (+ 1222us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 12:15:07.855808 (+ 131us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 12:15:07.855907 (+ 99us) server_negotiation.cc:1035] Waiting for connection context
0809 12:15:07.856453 (+ 546us) server_negotiation.cc:299] Negotiation successful
0809 12:15:07.856580 (+ 127us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":5,"server-negotiator.queue_time_us":72}
I20260809 12:15:07.857409 21652 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 12:15:07.836299 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.222.3:33877 (local address 127.0.222.1:53685)
0809 12:15:07.848647 (+ 12348us) negotiation.cc:107] Waiting for socket to connect
0809 12:15:07.848663 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0809 12:15:07.848841 (+ 178us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 12:15:07.849386 (+ 545us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 12:15:07.849390 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 12:15:07.849417 (+ 27us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0809 12:15:07.849733 (+ 316us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 12:15:07.849758 (+ 25us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 12:15:07.851278 (+ 1520us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 12:15:07.851283 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 12:15:07.854032 (+ 2749us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 12:15:07.854048 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 12:15:07.855768 (+ 1720us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 12:15:07.855776 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 12:15:07.855980 (+ 204us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 12:15:07.856021 (+ 41us) client_negotiation.cc:769] Sending connection context
0809 12:15:07.856860 (+ 839us) client_negotiation.cc:240] Negotiation successful
0809 12:15:07.857002 (+ 142us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":12226,"mutex_wait_us":32,"thread_start_us":11301,"threads_started":1}
I20260809 12:15:07.857602 21653 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 12:15:07.835889 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.222.2:44221 (local address 127.0.222.1:59071)
0809 12:15:07.846949 (+ 11060us) negotiation.cc:107] Waiting for socket to connect
0809 12:15:07.846987 (+ 38us) client_negotiation.cc:174] Beginning negotiation
0809 12:15:07.847709 (+ 722us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 12:15:07.850283 (+ 2574us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 12:15:07.850288 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 12:15:07.850317 (+ 29us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0809 12:15:07.850739 (+ 422us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 12:15:07.850750 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 12:15:07.852159 (+ 1409us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 12:15:07.852164 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 12:15:07.854265 (+ 2101us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 12:15:07.854278 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 12:15:07.856053 (+ 1775us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 12:15:07.856060 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 12:15:07.856255 (+ 195us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 12:15:07.856288 (+ 33us) client_negotiation.cc:769] Sending connection context
0809 12:15:07.857051 (+ 763us) client_negotiation.cc:240] Negotiation successful
0809 12:15:07.857202 (+ 151us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":10920,"mutex_wait_us":30,"thread_start_us":14106,"threads_started":1}
I20260809 12:15:07.858084 21569 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "11323aa901c0491a82e399f6e6ab79cc" candidate_uuid: "574166c798ce434c837b3cc142394423" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "014d4c8580a640a18402dd826220c63d" is_pre_election: true
I20260809 12:15:07.858567 21427 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "11323aa901c0491a82e399f6e6ab79cc" candidate_uuid: "574166c798ce434c837b3cc142394423" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6e6dcda253eb40db9054aba46933006d" is_pre_election: true
I20260809 12:15:07.859126 21570 raft_consensus.cc:2454] T 11323aa901c0491a82e399f6e6ab79cc P 014d4c8580a640a18402dd826220c63d [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6e6dcda253eb40db9054aba46933006d in term 1.
I20260809 12:15:07.859582 21427 raft_consensus.cc:2379] T 11323aa901c0491a82e399f6e6ab79cc P 6e6dcda253eb40db9054aba46933006d [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 574166c798ce434c837b3cc142394423 in current term 1: Already voted for candidate 6e6dcda253eb40db9054aba46933006d in this term.
I20260809 12:15:07.859543 21354 leader_election.cc:302] T 11323aa901c0491a82e399f6e6ab79cc P 6e6dcda253eb40db9054aba46933006d [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 014d4c8580a640a18402dd826220c63d, 6e6dcda253eb40db9054aba46933006d; no voters:
I20260809 12:15:07.860603 21213 leader_election.cc:302] T 11323aa901c0491a82e399f6e6ab79cc P 574166c798ce434c837b3cc142394423 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 574166c798ce434c837b3cc142394423; no voters: 014d4c8580a640a18402dd826220c63d, 6e6dcda253eb40db9054aba46933006d
I20260809 12:15:07.861518 21283 raft_consensus.cc:2454] T 11323aa901c0491a82e399f6e6ab79cc P 574166c798ce434c837b3cc142394423 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6e6dcda253eb40db9054aba46933006d in term 1.
I20260809 12:15:07.861703 21647 leader_election.cc:288] T 11323aa901c0491a82e399f6e6ab79cc P 6e6dcda253eb40db9054aba46933006d [CANDIDATE]: Term 1 election: Requested vote from peers 014d4c8580a640a18402dd826220c63d (127.0.222.3:33877), 574166c798ce434c837b3cc142394423 (127.0.222.1:44697)
I20260809 12:15:07.861941 21647 raft_consensus.cc:2781] T 11323aa901c0491a82e399f6e6ab79cc P 6e6dcda253eb40db9054aba46933006d [term 1 FOLLOWER]: Leader election won for term 1
I20260809 12:15:07.862165 21648 raft_consensus.cc:2726] T 11323aa901c0491a82e399f6e6ab79cc P 574166c798ce434c837b3cc142394423 [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260809 12:15:07.862335 21647 raft_consensus.cc:686] T 11323aa901c0491a82e399f6e6ab79cc P 6e6dcda253eb40db9054aba46933006d [term 1 LEADER]: Becoming Leader. State: Replica: 6e6dcda253eb40db9054aba46933006d, State: Running, Role: LEADER
I20260809 12:15:07.862895 21647 consensus_queue.cc:237] T 11323aa901c0491a82e399f6e6ab79cc P 6e6dcda253eb40db9054aba46933006d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e6dcda253eb40db9054aba46933006d" member_type: VOTER last_known_addr { host: "127.0.222.2" port: 44221 } } peers { permanent_uuid: "014d4c8580a640a18402dd826220c63d" member_type: VOTER last_known_addr { host: "127.0.222.3" port: 33877 } } peers { permanent_uuid: "574166c798ce434c837b3cc142394423" member_type: VOTER last_known_addr { host: "127.0.222.1" port: 44697 } }
I20260809 12:15:07.868304 21132 catalog_manager.cc:5495] T 11323aa901c0491a82e399f6e6ab79cc P 6e6dcda253eb40db9054aba46933006d reported cstate change: term changed from 0 to 1, leader changed from <none> to 6e6dcda253eb40db9054aba46933006d (127.0.222.2). New cstate: current_term: 1 leader_uuid: "6e6dcda253eb40db9054aba46933006d" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e6dcda253eb40db9054aba46933006d" member_type: VOTER last_known_addr { host: "127.0.222.2" port: 44221 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "014d4c8580a640a18402dd826220c63d" member_type: VOTER last_known_addr { host: "127.0.222.3" port: 33877 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "574166c798ce434c837b3cc142394423" member_type: VOTER last_known_addr { host: "127.0.222.1" port: 44697 } health_report { overall_health: UNKNOWN } } }
I20260809 12:15:07.955021 21636 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 12:15:07.949434 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38102 (local address 127.0.222.2:44221)
0809 12:15:07.949670 (+ 236us) server_negotiation.cc:206] Beginning negotiation
0809 12:15:07.949675 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0809 12:15:07.949772 (+ 97us) server_negotiation.cc:407] Connection header received
0809 12:15:07.949986 (+ 214us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 12:15:07.949991 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 12:15:07.950071 (+ 80us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 12:15:07.950200 (+ 129us) server_negotiation.cc:226] Negotiated authn=TOKEN
0809 12:15:07.950619 (+ 419us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 12:15:07.951425 (+ 806us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 12:15:07.952799 (+ 1374us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 12:15:07.953205 (+ 406us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 12:15:07.953301 (+ 96us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 12:15:07.953825 (+ 524us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0809 12:15:07.954307 (+ 482us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0809 12:15:07.954442 (+ 135us) server_negotiation.cc:1035] Waiting for connection context
0809 12:15:07.954606 (+ 164us) server_negotiation.cc:299] Negotiation successful
0809 12:15:07.954670 (+ 64us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":107}
I20260809 12:15:07.963438 21570 raft_consensus.cc:1261] T 11323aa901c0491a82e399f6e6ab79cc P 014d4c8580a640a18402dd826220c63d [term 1 FOLLOWER]: Refusing update from remote peer 6e6dcda253eb40db9054aba46933006d: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260809 12:15:07.963501 21283 raft_consensus.cc:1261] T 11323aa901c0491a82e399f6e6ab79cc P 574166c798ce434c837b3cc142394423 [term 1 FOLLOWER]: Refusing update from remote peer 6e6dcda253eb40db9054aba46933006d: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260809 12:15:07.964365 21654 consensus_queue.cc:1035] T 11323aa901c0491a82e399f6e6ab79cc P 6e6dcda253eb40db9054aba46933006d [LEADER]: Connected to new peer: Peer: permanent_uuid: "014d4c8580a640a18402dd826220c63d" member_type: VOTER last_known_addr { host: "127.0.222.3" port: 33877 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260809 12:15:07.964718 21647 consensus_queue.cc:1035] T 11323aa901c0491a82e399f6e6ab79cc P 6e6dcda253eb40db9054aba46933006d [LEADER]: Connected to new peer: Peer: permanent_uuid: "574166c798ce434c837b3cc142394423" member_type: VOTER last_known_addr { host: "127.0.222.1" port: 44697 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260809 12:15:07.989224 21132 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:42466:
table { table_name: "test-table" } modify_external_catalogs: true
I20260809 12:15:07.989512 21132 catalog_manager.cc:2645] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:42466:
table { table_name: "test-table" } modify_external_catalogs: true
I20260809 12:15:07.993412 21132 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 17228f8b9e32445fa92080e7b2d6d9a7: Sending DeleteTablet for 3 replicas of tablet 11323aa901c0491a82e399f6e6ab79cc
I20260809 12:15:07.994722 21548 tablet_service.cc:1514] Processing DeleteTablet for tablet 11323aa901c0491a82e399f6e6ab79cc with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-09 12:15:07 UTC) from {username='slave'} at 127.0.0.1:53374
I20260809 12:15:07.994904 21262 tablet_service.cc:1514] Processing DeleteTablet for tablet 11323aa901c0491a82e399f6e6ab79cc with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-09 12:15:07 UTC) from {username='slave'} at 127.0.0.1:35782
I20260809 12:15:07.995460 21405 tablet_service.cc:1514] Processing DeleteTablet for tablet 11323aa901c0491a82e399f6e6ab79cc with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-09 12:15:07 UTC) from {username='slave'} at 127.0.0.1:38090
I20260809 12:15:08.004122 888 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskgH5Pjm/build/asan/bin/kudu pid 21194
I20260809 12:15:08.005419 21669 tablet_replica.cc:330] T 11323aa901c0491a82e399f6e6ab79cc P 6e6dcda253eb40db9054aba46933006d: stopping tablet replica
I20260809 12:15:08.005919 21669 raft_consensus.cc:2229] T 11323aa901c0491a82e399f6e6ab79cc P 6e6dcda253eb40db9054aba46933006d [term 1 LEADER]: Raft consensus shutting down.
I20260809 12:15:08.006706 21669 raft_consensus.cc:2258] T 11323aa901c0491a82e399f6e6ab79cc P 6e6dcda253eb40db9054aba46933006d [term 1 FOLLOWER]: Raft consensus is shut down!
I20260809 12:15:08.008265 21668 tablet_replica.cc:330] T 11323aa901c0491a82e399f6e6ab79cc P 574166c798ce434c837b3cc142394423: stopping tablet replica
I20260809 12:15:08.008776 21668 raft_consensus.cc:2229] T 11323aa901c0491a82e399f6e6ab79cc P 574166c798ce434c837b3cc142394423 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260809 12:15:08.010840 21668 raft_consensus.cc:2258] T 11323aa901c0491a82e399f6e6ab79cc P 574166c798ce434c837b3cc142394423 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260809 12:15:08.010139 21669 ts_tablet_manager.cc:1905] T 11323aa901c0491a82e399f6e6ab79cc P 6e6dcda253eb40db9054aba46933006d: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260809 12:15:08.014593 21667 tablet_replica.cc:330] T 11323aa901c0491a82e399f6e6ab79cc P 014d4c8580a640a18402dd826220c63d: stopping tablet replica
I20260809 12:15:08.015256 21667 raft_consensus.cc:2229] T 11323aa901c0491a82e399f6e6ab79cc P 014d4c8580a640a18402dd826220c63d [term 1 FOLLOWER]: Raft consensus shutting down.
I20260809 12:15:08.015931 21667 raft_consensus.cc:2258] T 11323aa901c0491a82e399f6e6ab79cc P 014d4c8580a640a18402dd826220c63d [term 1 FOLLOWER]: Raft consensus is shut down!
I20260809 12:15:08.017841 21667 ts_tablet_manager.cc:1905] T 11323aa901c0491a82e399f6e6ab79cc P 014d4c8580a640a18402dd826220c63d: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260809 12:15:08.018784 21669 ts_tablet_manager.cc:1918] T 11323aa901c0491a82e399f6e6ab79cc P 6e6dcda253eb40db9054aba46933006d: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260809 12:15:08.018953 21669 log.cc:1196] T 11323aa901c0491a82e399f6e6ab79cc P 6e6dcda253eb40db9054aba46933006d: Deleting WAL directory at /tmp/dist-test-taskgH5Pjm/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786277382073938-888-0/minicluster-data/ts-1/wal/wals/11323aa901c0491a82e399f6e6ab79cc
I20260809 12:15:08.019645 21669 ts_tablet_manager.cc:1939] T 11323aa901c0491a82e399f6e6ab79cc P 6e6dcda253eb40db9054aba46933006d: Deleting consensus metadata
I20260809 12:15:08.021539 21119 catalog_manager.cc:4841] TS 6e6dcda253eb40db9054aba46933006d (127.0.222.2:44221): tablet 11323aa901c0491a82e399f6e6ab79cc (table test-table [id=2db15243448f4c29b5a0e791f8b09f1b]) successfully deleted
I20260809 12:15:08.020457 21668 ts_tablet_manager.cc:1905] T 11323aa901c0491a82e399f6e6ab79cc P 574166c798ce434c837b3cc142394423: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260809 12:15:08.023370 21638 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 12:15:08.006891 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35788 (local address 127.0.222.1:44697)
0809 12:15:08.007110 (+ 219us) server_negotiation.cc:206] Beginning negotiation
0809 12:15:08.007117 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0809 12:15:08.008628 (+ 1511us) server_negotiation.cc:407] Connection header received
0809 12:15:08.008709 (+ 81us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 12:15:08.008713 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 12:15:08.008824 (+ 111us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 12:15:08.008952 (+ 128us) server_negotiation.cc:226] Negotiated authn=SASL
0809 12:15:08.009527 (+ 575us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 12:15:08.011070 (+ 1543us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 12:15:08.013790 (+ 2720us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 12:15:08.014557 (+ 767us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 12:15:08.014699 (+ 142us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 12:15:08.021211 (+ 6512us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 12:15:08.021222 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 12:15:08.021239 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0809 12:15:08.021948 (+ 709us) server_negotiation.cc:922] calling sasl_server_start()
0809 12:15:08.022024 (+ 76us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0809 12:15:08.022061 (+ 37us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 12:15:08.022184 (+ 123us) server_negotiation.cc:1035] Waiting for connection context
0809 12:15:08.022811 (+ 627us) server_negotiation.cc:299] Negotiation successful
0809 12:15:08.022929 (+ 118us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":90}
I20260809 12:15:08.024082 21329 generic_service.cc:196] Checking for leaks (request via RPC)
I20260809 12:15:08.026552 21667 ts_tablet_manager.cc:1918] T 11323aa901c0491a82e399f6e6ab79cc P 014d4c8580a640a18402dd826220c63d: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260809 12:15:08.026818 21667 log.cc:1196] T 11323aa901c0491a82e399f6e6ab79cc P 014d4c8580a640a18402dd826220c63d: Deleting WAL directory at /tmp/dist-test-taskgH5Pjm/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786277382073938-888-0/minicluster-data/ts-2/wal/wals/11323aa901c0491a82e399f6e6ab79cc
I20260809 12:15:08.027544 21667 ts_tablet_manager.cc:1939] T 11323aa901c0491a82e399f6e6ab79cc P 014d4c8580a640a18402dd826220c63d: Deleting consensus metadata
I20260809 12:15:08.029414 21116 catalog_manager.cc:4841] TS 014d4c8580a640a18402dd826220c63d (127.0.222.3:33877): tablet 11323aa901c0491a82e399f6e6ab79cc (table test-table [id=2db15243448f4c29b5a0e791f8b09f1b]) successfully deleted
I20260809 12:15:08.119220 888 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskgH5Pjm/build/asan/bin/kudu with pid 21194
I20260809 12:15:08.141378 888 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskgH5Pjm/build/asan/bin/kudu pid 21338
W20260809 12:15:08.141433 21119 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.0.222.1:44697 (error 108)
I20260809 12:15:08.144296 21673 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 12:15:08.143017 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.222.1:44697 (local address 127.0.0.1:35804)
0809 12:15:08.143916 (+ 899us) negotiation.cc:107] Waiting for socket to connect
0809 12:15:08.144056 (+ 140us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.222.1:44697: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":762,"thread_start_us":659,"threads_started":1}
W20260809 12:15:08.144610 21119 catalog_manager.cc:4584] TS 574166c798ce434c837b3cc142394423 (127.0.222.1:44697): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 11323aa901c0491a82e399f6e6ab79cc: Network error: Client connection negotiation failed: client connection to 127.0.222.1:44697: connect: Connection refused (error 111)
I20260809 12:15:08.150122 21636 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 12:15:08.142612 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38104 (local address 127.0.222.2:44221)
0809 12:15:08.142782 (+ 170us) server_negotiation.cc:206] Beginning negotiation
0809 12:15:08.142787 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0809 12:15:08.142803 (+ 16us) server_negotiation.cc:407] Connection header received
0809 12:15:08.142862 (+ 59us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 12:15:08.142865 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 12:15:08.143036 (+ 171us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 12:15:08.143154 (+ 118us) server_negotiation.cc:226] Negotiated authn=SASL
0809 12:15:08.143607 (+ 453us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 12:15:08.144737 (+ 1130us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 12:15:08.146780 (+ 2043us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 12:15:08.147426 (+ 646us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 12:15:08.147576 (+ 150us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 12:15:08.148715 (+ 1139us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 12:15:08.148723 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 12:15:08.148736 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0809 12:15:08.149380 (+ 644us) server_negotiation.cc:922] calling sasl_server_start()
0809 12:15:08.149427 (+ 47us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0809 12:15:08.149449 (+ 22us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 12:15:08.149534 (+ 85us) server_negotiation.cc:1035] Waiting for connection context
0809 12:15:08.149697 (+ 163us) server_negotiation.cc:299] Negotiation successful
0809 12:15:08.149778 (+ 81us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":63}
I20260809 12:15:08.150879 21470 generic_service.cc:196] Checking for leaks (request via RPC)
I20260809 12:15:08.198241 21673 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 12:15:08.197662 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.222.1:44697 (local address 127.0.0.1:35814)
0809 12:15:08.197914 (+ 252us) negotiation.cc:107] Waiting for socket to connect
0809 12:15:08.198039 (+ 125us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.222.1:44697: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":94}
I20260809 12:15:08.199446 21673 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 12:15:08.199010 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.222.1:44697 (local address 127.0.0.1:35818)
0809 12:15:08.199183 (+ 173us) negotiation.cc:107] Waiting for socket to connect
0809 12:15:08.199269 (+ 86us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.222.1:44697: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":53}
I20260809 12:15:08.260293 21673 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 12:15:08.259759 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.222.1:44697 (local address 127.0.0.1:35820)
0809 12:15:08.260005 (+ 246us) negotiation.cc:107] Waiting for socket to connect
0809 12:15:08.260096 (+ 91us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.222.1:44697: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":71}
I20260809 12:15:08.261520 21673 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 12:15:08.261087 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.222.1:44697 (local address 127.0.0.1:35836)
0809 12:15:08.261261 (+ 174us) negotiation.cc:107] Waiting for socket to connect
0809 12:15:08.261347 (+ 86us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.222.1:44697: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":55}
I20260809 12:15:08.271824 888 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskgH5Pjm/build/asan/bin/kudu with pid 21338
I20260809 12:15:08.297317 888 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskgH5Pjm/build/asan/bin/kudu pid 21480
I20260809 12:15:08.304205 21637 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 12:15:08.298141 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53376 (local address 127.0.222.3:33877)
0809 12:15:08.298364 (+ 223us) server_negotiation.cc:206] Beginning negotiation
0809 12:15:08.298370 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0809 12:15:08.298417 (+ 47us) server_negotiation.cc:407] Connection header received
0809 12:15:08.298592 (+ 175us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 12:15:08.298596 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 12:15:08.298695 (+ 99us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 12:15:08.298781 (+ 86us) server_negotiation.cc:226] Negotiated authn=SASL
0809 12:15:08.299227 (+ 446us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 12:15:08.299935 (+ 708us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 12:15:08.301212 (+ 1277us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 12:15:08.301666 (+ 454us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 12:15:08.301759 (+ 93us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 12:15:08.302795 (+ 1036us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 12:15:08.302803 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 12:15:08.302818 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0809 12:15:08.303520 (+ 702us) server_negotiation.cc:922] calling sasl_server_start()
0809 12:15:08.303568 (+ 48us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0809 12:15:08.303594 (+ 26us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 12:15:08.303681 (+ 87us) server_negotiation.cc:1035] Waiting for connection context
0809 12:15:08.303779 (+ 98us) server_negotiation.cc:299] Negotiation successful
0809 12:15:08.303855 (+ 76us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":77}
I20260809 12:15:08.304879 21613 generic_service.cc:196] Checking for leaks (request via RPC)
I20260809 12:15:08.342782 21673 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 12:15:08.342197 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.222.1:44697 (local address 127.0.0.1:35840)
0809 12:15:08.342463 (+ 266us) negotiation.cc:107] Waiting for socket to connect
0809 12:15:08.342593 (+ 130us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.222.1:44697: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":104}
I20260809 12:15:08.344027 21673 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 12:15:08.343536 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.222.1:44697 (local address 127.0.0.1:35852)
0809 12:15:08.343756 (+ 220us) negotiation.cc:107] Waiting for socket to connect
0809 12:15:08.343854 (+ 98us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.222.1:44697: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":112}
I20260809 12:15:08.395843 888 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskgH5Pjm/build/asan/bin/kudu with pid 21480
I20260809 12:15:08.417819 888 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskgH5Pjm/build/asan/bin/kudu pid 21100
I20260809 12:15:08.419100 21163 generic_service.cc:196] Checking for leaks (request via RPC)
I20260809 12:15:08.531559 21673 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 12:15:08.530603 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.222.1:44697 (local address 127.0.0.1:52430)
0809 12:15:08.531131 (+ 528us) negotiation.cc:107] Waiting for socket to connect
0809 12:15:08.531321 (+ 190us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.222.1:44697: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":92}
I20260809 12:15:08.531886 888 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskgH5Pjm/build/asan/bin/kudu with pid 21100
2026-08-09T12:15:08Z chronyd exiting
[ OK ] EncryptionPolicyTest.TestEncryptionPolicy/3 (3830 ms)
[ RUN ] EncryptionPolicyTest.TestEncryptionPolicy/4
Loading random data
Initializing database '/tmp/dist-test-taskgH5Pjm/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786277382073938-888-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 09 12:15:08 dist-test-slave-2w3w krb5kdc[21679](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 09 12:15:08 dist-test-slave-2w3w krb5kdc[21679](info): set up 2 sockets
Aug 09 12:15:08 dist-test-slave-2w3w krb5kdc[21679](info): commencing operation
krb5kdc: starting...
W20260809 12:15:09.717187 888 mini_kdc.cc:121] Time spent starting KDC: real 1.139s user 0.003s sys 0.044s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskgH5Pjm/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786277382073938-888-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskgH5Pjm/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786277382073938-888-0/krb5kdc/test-user.keytab.
Aug 09 12:15:09 dist-test-slave-2w3w krb5kdc[21679](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786277709, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-09T12:15:09Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-09T12:15:09Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.222.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.222.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.222.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskgH5Pjm/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786277382073938-888-0/krb5kdc/kudu_127.0.222.62.keytab.
Entry for principal kudu/127.0.222.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskgH5Pjm/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786277382073938-888-0/krb5kdc/kudu_127.0.222.62.keytab.
I20260809 12:15:10.050657 888 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskgH5Pjm/build/asan/bin/kudu
/tmp/dist-test-taskgH5Pjm/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskgH5Pjm/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786277382073938-888-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskgH5Pjm/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786277382073938-888-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskgH5Pjm/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786277382073938-888-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskgH5Pjm/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786277382073938-888-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.222.62:33407
--webserver_interface=127.0.222.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskgH5Pjm/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786277382073938-888-0/krb5kdc/kudu_127.0.222.62.keytab
--principal=kudu/127.0.222.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.222.20:42815
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.222.62:33407
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskgH5Pjm/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786277382073938-888-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskgH5Pjm/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786277382073938-888-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskgH5Pjm/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786277382073938-888-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260809 12:15:10.403111 21694 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260809 12:15:10.403424 21694 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260809 12:15:10.403607 21694 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260809 12:15:10.411252 21694 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260809 12:15:10.411360 21694 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260809 12:15:10.411490 21694 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260809 12:15:10.411564 21694 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260809 12:15:10.411631 21694 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260809 12:15:10.421993 21694 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.222.20:42815
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskgH5Pjm/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786277382073938-888-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskgH5Pjm/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786277382073938-888-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.222.62:33407
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskgH5Pjm/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786277382073938-888-0/krb5kdc/kudu_127.0.222.62.keytab
--principal=kudu/127.0.222.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.222.62:33407
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskgH5Pjm/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786277382073938-888-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.222.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskgH5Pjm/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786277382073938-888-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 09 Aug 2026 11:06:41 UTC on ccycloud.quasar-bwoogy.root.comops.site
build id 4575991
ASAN enabled
I20260809 12:15:10.423380 21694 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260809 12:15:10.424768 21694 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260809 12:15:10.437001 21701 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 12:15:10.436969 21702 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 12:15:10.436969 21705 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260809 12:15:10.438041 21694 server_base.cc:1034] running on GCE node
I20260809 12:15:10.439106 21694 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260809 12:15:10.440603 21694 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260809 12:15:10.441815 21694 hybrid_clock.cc:648] HybridClock initialized: now 1786277710441751 us; error 53 us; skew 500 ppm
I20260809 12:15:10.442189 21694 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 09 12:15:10 dist-test-slave-2w3w krb5kdc[21679](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786277710, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.222.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260809 12:15:10.445557 21694 init.cc:377] Logged in from keytab as kudu/127.0.222.62@KRBTEST.COM (short username kudu)
I20260809 12:15:10.447228 21694 webserver.cc:460] Webserver started at http://127.0.222.62:37603/ using document root <none> and password file <none>
I20260809 12:15:10.447894 21694 fs_manager.cc:359] Metadata directory not provided
I20260809 12:15:10.448001 21694 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260809 12:15:10.448269 21694 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260809 12:15:10.451360 21694 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskgH5Pjm/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786277382073938-888-0/minicluster-data/master-0/data/instance:
uuid: "e4dcb620a4014a28a6f4c81962b03518"
format_stamp: "Formatted at 2026-08-09 12:15:10 on dist-test-slave-2w3w"
server_key: "4762244e67a8983ece0149a4015778ee"
server_key_iv: "e47a28448608a65a2fbed0daab05e00d"
server_key_version: "encryptionkey@0"
I20260809 12:15:10.452147 21694 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskgH5Pjm/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786277382073938-888-0/minicluster-data/master-0/wal/instance:
uuid: "e4dcb620a4014a28a6f4c81962b03518"
format_stamp: "Formatted at 2026-08-09 12:15:10 on dist-test-slave-2w3w"
server_key: "4762244e67a8983ece0149a4015778ee"
server_key_iv: "e47a28448608a65a2fbed0daab05e00d"
server_key_version: "encryptionkey@0"
I20260809 12:15:10.457397 21694 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.001s sys 0.004s
I20260809 12:15:10.460994 21714 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260809 12:15:10.462324 21694 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260809 12:15:10.462517 21694 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskgH5Pjm/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786277382073938-888-0/minicluster-data/master-0/data,/tmp/dist-test-taskgH5Pjm/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786277382073938-888-0/minicluster-data/master-0/wal
uuid: "e4dcb620a4014a28a6f4c81962b03518"
format_stamp: "Formatted at 2026-08-09 12:15:10 on dist-test-slave-2w3w"
server_key: "4762244e67a8983ece0149a4015778ee"
server_key_iv: "e47a28448608a65a2fbed0daab05e00d"
server_key_version: "encryptionkey@0"
I20260809 12:15:10.462731 21694 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskgH5Pjm/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786277382073938-888-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskgH5Pjm/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786277382073938-888-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskgH5Pjm/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786277382073938-888-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260809 12:15:10.478634 21694 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260809 12:15:10.483341 21694 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260809 12:15:10.483659 21694 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260809 12:15:10.503270 21694 rpc_server.cc:307] RPC server started. Bound to: 127.0.222.62:33407
I20260809 12:15:10.503325 21769 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.222.62:33407 every 8 connection(s)
I20260809 12:15:10.504734 21694 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskgH5Pjm/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786277382073938-888-0/minicluster-data/master-0/data/info.pb
I20260809 12:15:10.508870 21770 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260809 12:15:10.513131 888 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskgH5Pjm/build/asan/bin/kudu as pid 21694
I20260809 12:15:10.513343 888 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskgH5Pjm/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786277382073938-888-0/minicluster-data/master-0/wal/instance
I20260809 12:15:10.513911 888 external_mini_cluster.cc:1428] Setting key 6d480e644d82b214e42b638e2b7d52c4
I20260809 12:15:10.520655 21770 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P e4dcb620a4014a28a6f4c81962b03518: Bootstrap starting.
I20260809 12:15:10.524518 21770 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P e4dcb620a4014a28a6f4c81962b03518: Neither blocks nor log segments found. Creating new log.
I20260809 12:15:10.525700 21770 log.cc:824] T 00000000000000000000000000000000 P e4dcb620a4014a28a6f4c81962b03518: Log is configured to *not* fsync() on all Append() calls
I20260809 12:15:10.528487 21770 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P e4dcb620a4014a28a6f4c81962b03518: No bootstrap required, opened a new log
Aug 09 12:15:10 dist-test-slave-2w3w krb5kdc[21679](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786277709, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.222.62@KRBTEST.COM
I20260809 12:15:10.537915 21770 raft_consensus.cc:348] T 00000000000000000000000000000000 P e4dcb620a4014a28a6f4c81962b03518 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e4dcb620a4014a28a6f4c81962b03518" member_type: VOTER last_known_addr { host: "127.0.222.62" port: 33407 } }
I20260809 12:15:10.538313 21770 raft_consensus.cc:374] T 00000000000000000000000000000000 P e4dcb620a4014a28a6f4c81962b03518 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260809 12:15:10.538445 21770 raft_consensus.cc:729] T 00000000000000000000000000000000 P e4dcb620a4014a28a6f4c81962b03518 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e4dcb620a4014a28a6f4c81962b03518, State: Initialized, Role: FOLLOWER
I20260809 12:15:10.538645 21773 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 12:15:10.515921 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45514 (local address 127.0.222.62:33407)
0809 12:15:10.516744 (+ 823us) server_negotiation.cc:206] Beginning negotiation
0809 12:15:10.516756 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0809 12:15:10.516798 (+ 42us) server_negotiation.cc:407] Connection header received
0809 12:15:10.519158 (+ 2360us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 12:15:10.519180 (+ 22us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 12:15:10.519561 (+ 381us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 12:15:10.519866 (+ 305us) server_negotiation.cc:226] Negotiated authn=SASL
0809 12:15:10.521759 (+ 1893us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 12:15:10.523302 (+ 1543us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 12:15:10.525108 (+ 1806us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 12:15:10.525770 (+ 662us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 12:15:10.526000 (+ 230us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 12:15:10.530688 (+ 4688us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 12:15:10.530698 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 12:15:10.530715 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0809 12:15:10.530753 (+ 38us) server_negotiation.cc:922] calling sasl_server_start()
0809 12:15:10.534142 (+ 3389us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 12:15:10.535127 (+ 985us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 12:15:10.535134 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 12:15:10.535140 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0809 12:15:10.535318 (+ 178us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 12:15:10.535756 (+ 438us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 12:15:10.535759 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 12:15:10.535761 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0809 12:15:10.536343 (+ 582us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 12:15:10.536574 (+ 231us) server_negotiation.cc:1035] Waiting for connection context
0809 12:15:10.537099 (+ 525us) server_negotiation.cc:299] Negotiation successful
0809 12:15:10.537462 (+ 363us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":617,"thread_start_us":398,"threads_started":1}
I20260809 12:15:10.539189 21770 consensus_queue.cc:260] T 00000000000000000000000000000000 P e4dcb620a4014a28a6f4c81962b03518 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e4dcb620a4014a28a6f4c81962b03518" member_type: VOTER last_known_addr { host: "127.0.222.62" port: 33407 } }
I20260809 12:15:10.539515 21770 raft_consensus.cc:388] T 00000000000000000000000000000000 P e4dcb620a4014a28a6f4c81962b03518 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260809 12:15:10.539660 21770 raft_consensus.cc:482] T 00000000000000000000000000000000 P e4dcb620a4014a28a6f4c81962b03518 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260809 12:15:10.539844 21770 raft_consensus.cc:3037] T 00000000000000000000000000000000 P e4dcb620a4014a28a6f4c81962b03518 [term 0 FOLLOWER]: Advancing to term 1
I20260809 12:15:10.542470 21770 raft_consensus.cc:504] T 00000000000000000000000000000000 P e4dcb620a4014a28a6f4c81962b03518 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e4dcb620a4014a28a6f4c81962b03518" member_type: VOTER last_known_addr { host: "127.0.222.62" port: 33407 } }
I20260809 12:15:10.542902 21770 leader_election.cc:302] T 00000000000000000000000000000000 P e4dcb620a4014a28a6f4c81962b03518 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: e4dcb620a4014a28a6f4c81962b03518; no voters:
I20260809 12:15:10.543555 21770 leader_election.cc:288] T 00000000000000000000000000000000 P e4dcb620a4014a28a6f4c81962b03518 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260809 12:15:10.543726 21775 raft_consensus.cc:2781] T 00000000000000000000000000000000 P e4dcb620a4014a28a6f4c81962b03518 [term 1 FOLLOWER]: Leader election won for term 1
I20260809 12:15:10.544404 21775 raft_consensus.cc:686] T 00000000000000000000000000000000 P e4dcb620a4014a28a6f4c81962b03518 [term 1 LEADER]: Becoming Leader. State: Replica: e4dcb620a4014a28a6f4c81962b03518, State: Running, Role: LEADER
I20260809 12:15:10.545011 21775 consensus_queue.cc:237] T 00000000000000000000000000000000 P e4dcb620a4014a28a6f4c81962b03518 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e4dcb620a4014a28a6f4c81962b03518" member_type: VOTER last_known_addr { host: "127.0.222.62" port: 33407 } }
I20260809 12:15:10.545599 21770 sys_catalog.cc:564] T 00000000000000000000000000000000 P e4dcb620a4014a28a6f4c81962b03518 [sys.catalog]: configured and running, proceeding with master startup.
I20260809 12:15:10.548493 21776 sys_catalog.cc:455] T 00000000000000000000000000000000 P e4dcb620a4014a28a6f4c81962b03518 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "e4dcb620a4014a28a6f4c81962b03518" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e4dcb620a4014a28a6f4c81962b03518" member_type: VOTER last_known_addr { host: "127.0.222.62" port: 33407 } } }
I20260809 12:15:10.548871 21776 sys_catalog.cc:458] T 00000000000000000000000000000000 P e4dcb620a4014a28a6f4c81962b03518 [sys.catalog]: This master's current role is: LEADER
I20260809 12:15:10.548442 21777 sys_catalog.cc:455] T 00000000000000000000000000000000 P e4dcb620a4014a28a6f4c81962b03518 [sys.catalog]: SysCatalogTable state changed. Reason: New leader e4dcb620a4014a28a6f4c81962b03518. Latest consensus state: current_term: 1 leader_uuid: "e4dcb620a4014a28a6f4c81962b03518" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e4dcb620a4014a28a6f4c81962b03518" member_type: VOTER last_known_addr { host: "127.0.222.62" port: 33407 } } }
I20260809 12:15:10.549475 21777 sys_catalog.cc:458] T 00000000000000000000000000000000 P e4dcb620a4014a28a6f4c81962b03518 [sys.catalog]: This master's current role is: LEADER
I20260809 12:15:10.551270 21781 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260809 12:15:10.557742 21781 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260809 12:15:10.568323 21781 catalog_manager.cc:1347] Generated new cluster ID: 1d3b2955e2a8405e8690eaaca4e42137
I20260809 12:15:10.568455 21781 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260809 12:15:10.601101 21781 catalog_manager.cc:1370] Generated new certificate authority record
I20260809 12:15:10.603093 21781 catalog_manager.cc:1504] Loading token signing keys...
I20260809 12:15:10.616060 21781 catalog_manager.cc:5868] T 00000000000000000000000000000000 P e4dcb620a4014a28a6f4c81962b03518: Generated new TSK 0
I20260809 12:15:10.617164 21781 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.0.222.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.222.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.222.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskgH5Pjm/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786277382073938-888-0/krb5kdc/kudu_127.0.222.1.keytab.
Entry for principal kudu/127.0.222.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskgH5Pjm/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786277382073938-888-0/krb5kdc/kudu_127.0.222.1.keytab.
I20260809 12:15:10.694092 888 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskgH5Pjm/build/asan/bin/kudu
/tmp/dist-test-taskgH5Pjm/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskgH5Pjm/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786277382073938-888-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskgH5Pjm/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786277382073938-888-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskgH5Pjm/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786277382073938-888-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskgH5Pjm/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786277382073938-888-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.222.1:0
--local_ip_for_outbound_sockets=127.0.222.1
--webserver_interface=127.0.222.1
--webserver_port=0
--tserver_master_addrs=127.0.222.62:33407
--keytab_file=/tmp/dist-test-taskgH5Pjm/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786277382073938-888-0/krb5kdc/kudu_127.0.222.1.keytab
--principal=kudu/127.0.222.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.222.20:42815
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskgH5Pjm/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786277382073938-888-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskgH5Pjm/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786277382073938-888-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskgH5Pjm/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786277382073938-888-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260809 12:15:11.041697 21796 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260809 12:15:11.041994 21796 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260809 12:15:11.042155 21796 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260809 12:15:11.053597 21796 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260809 12:15:11.053784 21796 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260809 12:15:11.053975 21796 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.222.1
I20260809 12:15:11.064682 21796 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.222.20:42815
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskgH5Pjm/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786277382073938-888-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskgH5Pjm/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786277382073938-888-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskgH5Pjm/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786277382073938-888-0/krb5kdc/kudu_127.0.222.1.keytab
--principal=kudu/127.0.222.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.222.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskgH5Pjm/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786277382073938-888-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.222.1
--webserver_port=0
--tserver_master_addrs=127.0.222.62:33407
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.222.1
--log_dir=/tmp/dist-test-taskgH5Pjm/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786277382073938-888-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 09 Aug 2026 11:06:41 UTC on ccycloud.quasar-bwoogy.root.comops.site
build id 4575991
ASAN enabled
I20260809 12:15:11.066036 21796 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260809 12:15:11.067731 21796 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260809 12:15:11.081276 21801 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 12:15:11.081337 21802 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 12:15:11.081307 21804 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260809 12:15:11.082300 21796 server_base.cc:1034] running on GCE node
I20260809 12:15:11.082872 21796 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260809 12:15:11.084117 21796 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260809 12:15:11.085348 21796 hybrid_clock.cc:648] HybridClock initialized: now 1786277711085239 us; error 105 us; skew 500 ppm
I20260809 12:15:11.085731 21796 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 09 12:15:11 dist-test-slave-2w3w krb5kdc[21679](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786277711, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.222.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260809 12:15:11.089115 21796 init.cc:377] Logged in from keytab as kudu/127.0.222.1@KRBTEST.COM (short username kudu)
I20260809 12:15:11.090606 21796 webserver.cc:460] Webserver started at http://127.0.222.1:41561/ using document root <none> and password file <none>
I20260809 12:15:11.091396 21796 fs_manager.cc:359] Metadata directory not provided
I20260809 12:15:11.091552 21796 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260809 12:15:11.091881 21796 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260809 12:15:11.094924 21796 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskgH5Pjm/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786277382073938-888-0/minicluster-data/ts-0/data/instance:
uuid: "09204b15d804406a80c791bee0de9329"
format_stamp: "Formatted at 2026-08-09 12:15:11 on dist-test-slave-2w3w"
server_key: "b423b2807d940ecebefb6bb30add7899"
server_key_iv: "da4ce472df6c63901fc98cf2287f7027"
server_key_version: "encryptionkey@0"
I20260809 12:15:11.095872 21796 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskgH5Pjm/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786277382073938-888-0/minicluster-data/ts-0/wal/instance:
uuid: "09204b15d804406a80c791bee0de9329"
format_stamp: "Formatted at 2026-08-09 12:15:11 on dist-test-slave-2w3w"
server_key: "b423b2807d940ecebefb6bb30add7899"
server_key_iv: "da4ce472df6c63901fc98cf2287f7027"
server_key_version: "encryptionkey@0"
I20260809 12:15:11.101157 21796 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.000s
I20260809 12:15:11.105234 21811 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260809 12:15:11.106580 21796 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.000s sys 0.002s
I20260809 12:15:11.106763 21796 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskgH5Pjm/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786277382073938-888-0/minicluster-data/ts-0/data,/tmp/dist-test-taskgH5Pjm/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786277382073938-888-0/minicluster-data/ts-0/wal
uuid: "09204b15d804406a80c791bee0de9329"
format_stamp: "Formatted at 2026-08-09 12:15:11 on dist-test-slave-2w3w"
server_key: "b423b2807d940ecebefb6bb30add7899"
server_key_iv: "da4ce472df6c63901fc98cf2287f7027"
server_key_version: "encryptionkey@0"
I20260809 12:15:11.107012 21796 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskgH5Pjm/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786277382073938-888-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskgH5Pjm/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786277382073938-888-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskgH5Pjm/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786277382073938-888-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260809 12:15:11.142269 21796 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260809 12:15:11.147109 21796 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260809 12:15:11.147413 21796 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260809 12:15:11.148442 21796 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260809 12:15:11.149832 21796 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260809 12:15:11.149947 21796 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260809 12:15:11.150120 21796 ts_tablet_manager.cc:610] Registered 0 tablets
I20260809 12:15:11.150197 21796 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260809 12:15:11.185431 21796 rpc_server.cc:307] RPC server started. Bound to: 127.0.222.1:44429
I20260809 12:15:11.185491 21934 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.222.1:44429 every 8 connection(s)
I20260809 12:15:11.186789 21796 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskgH5Pjm/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786277382073938-888-0/minicluster-data/ts-0/data/info.pb
I20260809 12:15:11.187877 888 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskgH5Pjm/build/asan/bin/kudu as pid 21796
I20260809 12:15:11.188098 888 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskgH5Pjm/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786277382073938-888-0/minicluster-data/ts-0/wal/instance
I20260809 12:15:11.188571 888 external_mini_cluster.cc:1428] Setting key 9e0998aa57be24e494d1419920f752b3
Aug 09 12:15:11 dist-test-slave-2w3w krb5kdc[21679](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786277711, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.222.1@KRBTEST.COM for kudu/127.0.222.62@KRBTEST.COM
I20260809 12:15:11.209095 21937 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 12:15:11.189705 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.222.1:58595 (local address 127.0.222.62:33407)
0809 12:15:11.190198 (+ 493us) server_negotiation.cc:206] Beginning negotiation
0809 12:15:11.190204 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0809 12:15:11.191416 (+ 1212us) server_negotiation.cc:407] Connection header received
0809 12:15:11.193276 (+ 1860us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 12:15:11.193284 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 12:15:11.193386 (+ 102us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 12:15:11.193507 (+ 121us) server_negotiation.cc:226] Negotiated authn=SASL
0809 12:15:11.195047 (+ 1540us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 12:15:11.196210 (+ 1163us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 12:15:11.197848 (+ 1638us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 12:15:11.198563 (+ 715us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 12:15:11.198712 (+ 149us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 12:15:11.202414 (+ 3702us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 12:15:11.202423 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 12:15:11.202431 (+ 8us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0809 12:15:11.202480 (+ 49us) server_negotiation.cc:922] calling sasl_server_start()
0809 12:15:11.205633 (+ 3153us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 12:15:11.206659 (+ 1026us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 12:15:11.206663 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 12:15:11.206665 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0809 12:15:11.206797 (+ 132us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 12:15:11.207339 (+ 542us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 12:15:11.207342 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 12:15:11.207344 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0809 12:15:11.207598 (+ 254us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 12:15:11.207710 (+ 112us) server_negotiation.cc:1035] Waiting for connection context
0809 12:15:11.208659 (+ 949us) server_negotiation.cc:299] Negotiation successful
0809 12:15:11.208842 (+ 183us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":372,"thread_start_us":250,"threads_started":1}
I20260809 12:15:11.210781 21938 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 12:15:11.190348 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.222.62:33407 (local address 127.0.222.1:58595)
0809 12:15:11.191222 (+ 874us) negotiation.cc:107] Waiting for socket to connect
0809 12:15:11.191277 (+ 55us) client_negotiation.cc:174] Beginning negotiation
0809 12:15:11.193001 (+ 1724us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 12:15:11.193716 (+ 715us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 12:15:11.193724 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 12:15:11.194232 (+ 508us) client_negotiation.cc:189] Negotiated authn=SASL
0809 12:15:11.194846 (+ 614us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 12:15:11.194865 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 12:15:11.196371 (+ 1506us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 12:15:11.196375 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 12:15:11.197676 (+ 1301us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 12:15:11.197689 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 12:15:11.198784 (+ 1095us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 12:15:11.198789 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 12:15:11.199067 (+ 278us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 12:15:11.199774 (+ 707us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0809 12:15:11.199803 (+ 29us) client_negotiation.cc:656] calling sasl_client_start()
0809 12:15:11.202188 (+ 2385us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0809 12:15:11.205787 (+ 3599us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0809 12:15:11.205799 (+ 12us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0809 12:15:11.205819 (+ 20us) client_negotiation.cc:760] calling sasl_client_step()
0809 12:15:11.206528 (+ 709us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0809 12:15:11.206914 (+ 386us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0809 12:15:11.206921 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0809 12:15:11.206928 (+ 7us) client_negotiation.cc:760] calling sasl_client_step()
0809 12:15:11.207217 (+ 289us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0809 12:15:11.207725 (+ 508us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0809 12:15:11.207736 (+ 11us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0809 12:15:11.208219 (+ 483us) client_negotiation.cc:769] Sending connection context
0809 12:15:11.208542 (+ 323us) client_negotiation.cc:240] Negotiation successful
0809 12:15:11.209185 (+ 643us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":559,"thread_start_us":375,"threads_started":1}
I20260809 12:15:11.212708 21935 heartbeater.cc:347] Connected to a master server at 127.0.222.62:33407
I20260809 12:15:11.213050 21935 heartbeater.cc:464] Registering TS with master...
I20260809 12:15:11.213795 21935 heartbeater.cc:511] Master 127.0.222.62:33407 requested a full tablet report, sending...
I20260809 12:15:11.216041 21732 ts_manager.cc:194] Registered new tserver with Master: 09204b15d804406a80c791bee0de9329 (127.0.222.1:44429)
I20260809 12:15:11.217510 21732 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.222.1@KRBTEST.COM'} at 127.0.222.1:58595
WARNING: no policy specified for kudu/127.0.222.2@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.222.2@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.222.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskgH5Pjm/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786277382073938-888-0/krb5kdc/kudu_127.0.222.2.keytab.
Entry for principal kudu/127.0.222.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskgH5Pjm/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786277382073938-888-0/krb5kdc/kudu_127.0.222.2.keytab.
I20260809 12:15:11.266304 888 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskgH5Pjm/build/asan/bin/kudu
/tmp/dist-test-taskgH5Pjm/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskgH5Pjm/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786277382073938-888-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskgH5Pjm/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786277382073938-888-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskgH5Pjm/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786277382073938-888-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskgH5Pjm/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786277382073938-888-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.222.2:0
--local_ip_for_outbound_sockets=127.0.222.2
--webserver_interface=127.0.222.2
--webserver_port=0
--tserver_master_addrs=127.0.222.62:33407
--keytab_file=/tmp/dist-test-taskgH5Pjm/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786277382073938-888-0/krb5kdc/kudu_127.0.222.2.keytab
--principal=kudu/127.0.222.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.222.20:42815
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskgH5Pjm/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786277382073938-888-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskgH5Pjm/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786277382073938-888-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskgH5Pjm/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786277382073938-888-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260809 12:15:11.612193 21941 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260809 12:15:11.612481 21941 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260809 12:15:11.612581 21941 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260809 12:15:11.619874 21941 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260809 12:15:11.619978 21941 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260809 12:15:11.620085 21941 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.222.2
I20260809 12:15:11.629832 21941 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.222.20:42815
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskgH5Pjm/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786277382073938-888-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskgH5Pjm/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786277382073938-888-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskgH5Pjm/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786277382073938-888-0/krb5kdc/kudu_127.0.222.2.keytab
--principal=kudu/127.0.222.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.222.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskgH5Pjm/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786277382073938-888-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.222.2
--webserver_port=0
--tserver_master_addrs=127.0.222.62:33407
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.222.2
--log_dir=/tmp/dist-test-taskgH5Pjm/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786277382073938-888-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 09 Aug 2026 11:06:41 UTC on ccycloud.quasar-bwoogy.root.comops.site
build id 4575991
ASAN enabled
I20260809 12:15:11.631179 21941 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260809 12:15:11.632534 21941 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260809 12:15:11.646162 21949 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 12:15:11.646200 21946 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 12:15:11.646237 21947 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260809 12:15:11.647183 21941 server_base.cc:1034] running on GCE node
I20260809 12:15:11.647782 21941 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260809 12:15:11.648762 21941 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260809 12:15:11.649964 21941 hybrid_clock.cc:648] HybridClock initialized: now 1786277711649926 us; error 39 us; skew 500 ppm
I20260809 12:15:11.650290 21941 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 09 12:15:11 dist-test-slave-2w3w krb5kdc[21679](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786277711, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.222.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260809 12:15:11.653579 21941 init.cc:377] Logged in from keytab as kudu/127.0.222.2@KRBTEST.COM (short username kudu)
I20260809 12:15:11.655197 21941 webserver.cc:460] Webserver started at http://127.0.222.2:40249/ using document root <none> and password file <none>
I20260809 12:15:11.655895 21941 fs_manager.cc:359] Metadata directory not provided
I20260809 12:15:11.656000 21941 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260809 12:15:11.656430 21941 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260809 12:15:11.659418 21941 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskgH5Pjm/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786277382073938-888-0/minicluster-data/ts-1/data/instance:
uuid: "f9753698cc3c4413a465e4ab5db2c61a"
format_stamp: "Formatted at 2026-08-09 12:15:11 on dist-test-slave-2w3w"
server_key: "4bdd3824957ee3f386f20ad2da704f61"
server_key_iv: "fd81cb5f69466765d2c4f14e88ff6635"
server_key_version: "encryptionkey@0"
I20260809 12:15:11.660203 21941 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskgH5Pjm/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786277382073938-888-0/minicluster-data/ts-1/wal/instance:
uuid: "f9753698cc3c4413a465e4ab5db2c61a"
format_stamp: "Formatted at 2026-08-09 12:15:11 on dist-test-slave-2w3w"
server_key: "4bdd3824957ee3f386f20ad2da704f61"
server_key_iv: "fd81cb5f69466765d2c4f14e88ff6635"
server_key_version: "encryptionkey@0"
I20260809 12:15:11.665445 21941 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.001s
I20260809 12:15:11.669217 21956 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260809 12:15:11.670724 21941 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260809 12:15:11.670935 21941 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskgH5Pjm/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786277382073938-888-0/minicluster-data/ts-1/data,/tmp/dist-test-taskgH5Pjm/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786277382073938-888-0/minicluster-data/ts-1/wal
uuid: "f9753698cc3c4413a465e4ab5db2c61a"
format_stamp: "Formatted at 2026-08-09 12:15:11 on dist-test-slave-2w3w"
server_key: "4bdd3824957ee3f386f20ad2da704f61"
server_key_iv: "fd81cb5f69466765d2c4f14e88ff6635"
server_key_version: "encryptionkey@0"
I20260809 12:15:11.671255 21941 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskgH5Pjm/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786277382073938-888-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskgH5Pjm/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786277382073938-888-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskgH5Pjm/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786277382073938-888-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260809 12:15:11.692279 21941 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260809 12:15:11.696987 21941 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260809 12:15:11.697330 21941 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260809 12:15:11.698366 21941 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260809 12:15:11.699880 21941 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260809 12:15:11.699988 21941 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260809 12:15:11.700153 21941 ts_tablet_manager.cc:610] Registered 0 tablets
I20260809 12:15:11.700227 21941 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260809 12:15:11.735598 21941 rpc_server.cc:307] RPC server started. Bound to: 127.0.222.2:32929
I20260809 12:15:11.735697 22080 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.222.2:32929 every 8 connection(s)
I20260809 12:15:11.736953 21941 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskgH5Pjm/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786277382073938-888-0/minicluster-data/ts-1/data/info.pb
I20260809 12:15:11.740542 888 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskgH5Pjm/build/asan/bin/kudu as pid 21941
I20260809 12:15:11.740757 888 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskgH5Pjm/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786277382073938-888-0/minicluster-data/ts-1/wal/instance
I20260809 12:15:11.741407 888 external_mini_cluster.cc:1428] Setting key 61f7120ebf54c9d9acd820f8f05a654b
Aug 09 12:15:11 dist-test-slave-2w3w krb5kdc[21679](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786277711, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.222.2@KRBTEST.COM for kudu/127.0.222.62@KRBTEST.COM
I20260809 12:15:11.760263 22083 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 12:15:11.739772 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.222.2:51865 (local address 127.0.222.62:33407)
0809 12:15:11.740278 (+ 506us) server_negotiation.cc:206] Beginning negotiation
0809 12:15:11.740283 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0809 12:15:11.741404 (+ 1121us) server_negotiation.cc:407] Connection header received
0809 12:15:11.743775 (+ 2371us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 12:15:11.743782 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 12:15:11.743907 (+ 125us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 12:15:11.744028 (+ 121us) server_negotiation.cc:226] Negotiated authn=SASL
0809 12:15:11.745626 (+ 1598us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 12:15:11.746865 (+ 1239us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 12:15:11.748504 (+ 1639us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 12:15:11.749311 (+ 807us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 12:15:11.749456 (+ 145us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 12:15:11.753013 (+ 3557us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 12:15:11.753022 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 12:15:11.753029 (+ 7us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0809 12:15:11.753077 (+ 48us) server_negotiation.cc:922] calling sasl_server_start()
0809 12:15:11.756823 (+ 3746us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 12:15:11.757662 (+ 839us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 12:15:11.757670 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 12:15:11.757676 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0809 12:15:11.757865 (+ 189us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 12:15:11.758418 (+ 553us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 12:15:11.758421 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 12:15:11.758423 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0809 12:15:11.758689 (+ 266us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 12:15:11.758800 (+ 111us) server_negotiation.cc:1035] Waiting for connection context
0809 12:15:11.759783 (+ 983us) server_negotiation.cc:299] Negotiation successful
0809 12:15:11.759984 (+ 201us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":397,"thread_start_us":275,"threads_started":1}
I20260809 12:15:11.761950 22084 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 12:15:11.740338 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.222.62:33407 (local address 127.0.222.2:51865)
0809 12:15:11.741237 (+ 899us) negotiation.cc:107] Waiting for socket to connect
0809 12:15:11.741275 (+ 38us) client_negotiation.cc:174] Beginning negotiation
0809 12:15:11.743339 (+ 2064us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 12:15:11.744241 (+ 902us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 12:15:11.744249 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 12:15:11.744770 (+ 521us) client_negotiation.cc:189] Negotiated authn=SASL
0809 12:15:11.745452 (+ 682us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 12:15:11.745472 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 12:15:11.747063 (+ 1591us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 12:15:11.747067 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 12:15:11.748344 (+ 1277us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 12:15:11.748356 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 12:15:11.749540 (+ 1184us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 12:15:11.749545 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 12:15:11.749739 (+ 194us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 12:15:11.750454 (+ 715us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0809 12:15:11.750481 (+ 27us) client_negotiation.cc:656] calling sasl_client_start()
0809 12:15:11.752765 (+ 2284us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0809 12:15:11.757016 (+ 4251us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0809 12:15:11.757023 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0809 12:15:11.757052 (+ 29us) client_negotiation.cc:760] calling sasl_client_step()
0809 12:15:11.757528 (+ 476us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0809 12:15:11.757984 (+ 456us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0809 12:15:11.757987 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0809 12:15:11.757989 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0809 12:15:11.758278 (+ 289us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0809 12:15:11.758828 (+ 550us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0809 12:15:11.758837 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0809 12:15:11.759365 (+ 528us) client_negotiation.cc:769] Sending connection context
0809 12:15:11.759691 (+ 326us) client_negotiation.cc:240] Negotiation successful
0809 12:15:11.760303 (+ 612us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":649,"thread_start_us":403,"threads_started":1}
I20260809 12:15:11.763583 22081 heartbeater.cc:347] Connected to a master server at 127.0.222.62:33407
I20260809 12:15:11.763913 22081 heartbeater.cc:464] Registering TS with master...
I20260809 12:15:11.764642 22081 heartbeater.cc:511] Master 127.0.222.62:33407 requested a full tablet report, sending...
I20260809 12:15:11.766181 21732 ts_manager.cc:194] Registered new tserver with Master: f9753698cc3c4413a465e4ab5db2c61a (127.0.222.2:32929)
I20260809 12:15:11.767158 21732 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.222.2@KRBTEST.COM'} at 127.0.222.2:51865
WARNING: no policy specified for kudu/127.0.222.3@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.222.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.222.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskgH5Pjm/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786277382073938-888-0/krb5kdc/kudu_127.0.222.3.keytab.
Entry for principal kudu/127.0.222.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskgH5Pjm/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786277382073938-888-0/krb5kdc/kudu_127.0.222.3.keytab.
I20260809 12:15:11.818496 888 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskgH5Pjm/build/asan/bin/kudu
/tmp/dist-test-taskgH5Pjm/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskgH5Pjm/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786277382073938-888-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskgH5Pjm/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786277382073938-888-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskgH5Pjm/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786277382073938-888-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskgH5Pjm/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786277382073938-888-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.222.3:0
--local_ip_for_outbound_sockets=127.0.222.3
--webserver_interface=127.0.222.3
--webserver_port=0
--tserver_master_addrs=127.0.222.62:33407
--keytab_file=/tmp/dist-test-taskgH5Pjm/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786277382073938-888-0/krb5kdc/kudu_127.0.222.3.keytab
--principal=kudu/127.0.222.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.222.20:42815
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskgH5Pjm/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786277382073938-888-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskgH5Pjm/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786277382073938-888-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskgH5Pjm/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786277382073938-888-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260809 12:15:12.160099 22088 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260809 12:15:12.160441 22088 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260809 12:15:12.160614 22088 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260809 12:15:12.168320 22088 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260809 12:15:12.168501 22088 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260809 12:15:12.168794 22088 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.222.3
I20260809 12:15:12.179067 22088 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.222.20:42815
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskgH5Pjm/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786277382073938-888-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskgH5Pjm/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786277382073938-888-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskgH5Pjm/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786277382073938-888-0/krb5kdc/kudu_127.0.222.3.keytab
--principal=kudu/127.0.222.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.222.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskgH5Pjm/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786277382073938-888-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.222.3
--webserver_port=0
--tserver_master_addrs=127.0.222.62:33407
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.222.3
--log_dir=/tmp/dist-test-taskgH5Pjm/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786277382073938-888-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 09 Aug 2026 11:06:41 UTC on ccycloud.quasar-bwoogy.root.comops.site
build id 4575991
ASAN enabled
I20260809 12:15:12.180315 22088 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260809 12:15:12.181664 22088 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260809 12:15:12.196710 22093 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 12:15:12.196718 22096 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 12:15:12.196759 22094 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260809 12:15:12.197588 22088 server_base.cc:1034] running on GCE node
I20260809 12:15:12.198241 22088 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260809 12:15:12.199296 22088 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260809 12:15:12.200539 22088 hybrid_clock.cc:648] HybridClock initialized: now 1786277712200486 us; error 36 us; skew 500 ppm
I20260809 12:15:12.200866 22088 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 09 12:15:12 dist-test-slave-2w3w krb5kdc[21679](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786277712, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.222.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260809 12:15:12.204051 22088 init.cc:377] Logged in from keytab as kudu/127.0.222.3@KRBTEST.COM (short username kudu)
I20260809 12:15:12.205538 22088 webserver.cc:460] Webserver started at http://127.0.222.3:44297/ using document root <none> and password file <none>
I20260809 12:15:12.206257 22088 fs_manager.cc:359] Metadata directory not provided
I20260809 12:15:12.206413 22088 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260809 12:15:12.206740 22088 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260809 12:15:12.209789 22088 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskgH5Pjm/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786277382073938-888-0/minicluster-data/ts-2/data/instance:
uuid: "48112dd60dbc49f49247eb7f61b9cce2"
format_stamp: "Formatted at 2026-08-09 12:15:12 on dist-test-slave-2w3w"
server_key: "db03449211f5279709239520517c5725"
server_key_iv: "0d853deca0c1e6f1b90d3abd27f400ad"
server_key_version: "encryptionkey@0"
I20260809 12:15:12.210623 22088 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskgH5Pjm/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786277382073938-888-0/minicluster-data/ts-2/wal/instance:
uuid: "48112dd60dbc49f49247eb7f61b9cce2"
format_stamp: "Formatted at 2026-08-09 12:15:12 on dist-test-slave-2w3w"
server_key: "db03449211f5279709239520517c5725"
server_key_iv: "0d853deca0c1e6f1b90d3abd27f400ad"
server_key_version: "encryptionkey@0"
I20260809 12:15:12.215941 22088 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.003s sys 0.001s
I20260809 12:15:12.219313 22104 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260809 12:15:12.220525 21935 heartbeater.cc:503] Master 127.0.222.62:33407 was elected leader, sending a full tablet report...
I20260809 12:15:12.220701 22088 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.001s
I20260809 12:15:12.220891 22088 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskgH5Pjm/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786277382073938-888-0/minicluster-data/ts-2/data,/tmp/dist-test-taskgH5Pjm/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786277382073938-888-0/minicluster-data/ts-2/wal
uuid: "48112dd60dbc49f49247eb7f61b9cce2"
format_stamp: "Formatted at 2026-08-09 12:15:12 on dist-test-slave-2w3w"
server_key: "db03449211f5279709239520517c5725"
server_key_iv: "0d853deca0c1e6f1b90d3abd27f400ad"
server_key_version: "encryptionkey@0"
I20260809 12:15:12.221122 22088 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskgH5Pjm/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786277382073938-888-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskgH5Pjm/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786277382073938-888-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskgH5Pjm/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786277382073938-888-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260809 12:15:12.247532 22088 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260809 12:15:12.251575 22088 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260809 12:15:12.251817 22088 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260809 12:15:12.252729 22088 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260809 12:15:12.253988 22088 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260809 12:15:12.254062 22088 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260809 12:15:12.254144 22088 ts_tablet_manager.cc:610] Registered 0 tablets
I20260809 12:15:12.254189 22088 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260809 12:15:12.289386 22088 rpc_server.cc:307] RPC server started. Bound to: 127.0.222.3:33909
I20260809 12:15:12.289480 22229 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.222.3:33909 every 8 connection(s)
I20260809 12:15:12.290710 22088 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskgH5Pjm/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786277382073938-888-0/minicluster-data/ts-2/data/info.pb
I20260809 12:15:12.291913 888 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskgH5Pjm/build/asan/bin/kudu as pid 22088
I20260809 12:15:12.292117 888 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskgH5Pjm/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786277382073938-888-0/minicluster-data/ts-2/wal/instance
I20260809 12:15:12.292608 888 external_mini_cluster.cc:1428] Setting key f1296eb83bdf0dbd2309bf0a7b567d0f
Aug 09 12:15:12 dist-test-slave-2w3w krb5kdc[21679](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786277712, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.222.3@KRBTEST.COM for kudu/127.0.222.62@KRBTEST.COM
I20260809 12:15:12.311533 22232 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 12:15:12.293455 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.222.3:42589 (local address 127.0.222.62:33407)
0809 12:15:12.293902 (+ 447us) server_negotiation.cc:206] Beginning negotiation
0809 12:15:12.293907 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0809 12:15:12.295292 (+ 1385us) server_negotiation.cc:407] Connection header received
0809 12:15:12.297149 (+ 1857us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 12:15:12.297153 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 12:15:12.297228 (+ 75us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 12:15:12.297318 (+ 90us) server_negotiation.cc:226] Negotiated authn=SASL
0809 12:15:12.298836 (+ 1518us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 12:15:12.299655 (+ 819us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 12:15:12.301219 (+ 1564us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 12:15:12.301600 (+ 381us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 12:15:12.301692 (+ 92us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 12:15:12.305122 (+ 3430us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 12:15:12.305127 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 12:15:12.305131 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0809 12:15:12.305164 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0809 12:15:12.308630 (+ 3466us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 12:15:12.309322 (+ 692us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 12:15:12.309326 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 12:15:12.309328 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0809 12:15:12.309452 (+ 124us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 12:15:12.309832 (+ 380us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 12:15:12.309835 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 12:15:12.309837 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0809 12:15:12.310168 (+ 331us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 12:15:12.310292 (+ 124us) server_negotiation.cc:1035] Waiting for connection context
0809 12:15:12.311044 (+ 752us) server_negotiation.cc:299] Negotiation successful
0809 12:15:12.311248 (+ 204us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":338,"thread_start_us":212,"threads_started":1}
I20260809 12:15:12.312420 22233 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 12:15:12.294155 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.222.62:33407 (local address 127.0.222.3:42589)
0809 12:15:12.295119 (+ 964us) negotiation.cc:107] Waiting for socket to connect
0809 12:15:12.295169 (+ 50us) client_negotiation.cc:174] Beginning negotiation
0809 12:15:12.296912 (+ 1743us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 12:15:12.297523 (+ 611us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 12:15:12.297531 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 12:15:12.298097 (+ 566us) client_negotiation.cc:189] Negotiated authn=SASL
0809 12:15:12.298677 (+ 580us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 12:15:12.298694 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 12:15:12.299832 (+ 1138us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 12:15:12.299836 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 12:15:12.301073 (+ 1237us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 12:15:12.301090 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 12:15:12.301833 (+ 743us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 12:15:12.301838 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 12:15:12.302027 (+ 189us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 12:15:12.302663 (+ 636us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0809 12:15:12.302685 (+ 22us) client_negotiation.cc:656] calling sasl_client_start()
0809 12:15:12.304934 (+ 2249us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0809 12:15:12.308750 (+ 3816us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0809 12:15:12.308757 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0809 12:15:12.308769 (+ 12us) client_negotiation.cc:760] calling sasl_client_step()
0809 12:15:12.309222 (+ 453us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0809 12:15:12.309555 (+ 333us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0809 12:15:12.309558 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0809 12:15:12.309560 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0809 12:15:12.309738 (+ 178us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0809 12:15:12.310273 (+ 535us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0809 12:15:12.310278 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0809 12:15:12.310578 (+ 300us) client_negotiation.cc:769] Sending connection context
0809 12:15:12.310874 (+ 296us) client_negotiation.cc:240] Negotiation successful
0809 12:15:12.311326 (+ 452us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":656,"thread_start_us":395,"threads_started":1}
I20260809 12:15:12.313649 22230 heartbeater.cc:347] Connected to a master server at 127.0.222.62:33407
I20260809 12:15:12.313972 22230 heartbeater.cc:464] Registering TS with master...
I20260809 12:15:12.314642 22230 heartbeater.cc:511] Master 127.0.222.62:33407 requested a full tablet report, sending...
I20260809 12:15:12.316084 21732 ts_manager.cc:194] Registered new tserver with Master: 48112dd60dbc49f49247eb7f61b9cce2 (127.0.222.3:33909)
I20260809 12:15:12.316825 21732 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.222.3@KRBTEST.COM'} at 127.0.222.3:42589
I20260809 12:15:12.320681 888 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260809 12:15:12.338470 22232 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 12:15:12.324700 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45524 (local address 127.0.222.62:33407)
0809 12:15:12.324896 (+ 196us) server_negotiation.cc:206] Beginning negotiation
0809 12:15:12.324903 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0809 12:15:12.325419 (+ 516us) server_negotiation.cc:407] Connection header received
0809 12:15:12.325557 (+ 138us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 12:15:12.325560 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 12:15:12.325618 (+ 58us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 12:15:12.325696 (+ 78us) server_negotiation.cc:226] Negotiated authn=SASL
0809 12:15:12.327211 (+ 1515us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 12:15:12.327918 (+ 707us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 12:15:12.329209 (+ 1291us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 12:15:12.329600 (+ 391us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 12:15:12.329689 (+ 89us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 12:15:12.332285 (+ 2596us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 12:15:12.332292 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 12:15:12.332297 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0809 12:15:12.332377 (+ 80us) server_negotiation.cc:922] calling sasl_server_start()
0809 12:15:12.335968 (+ 3591us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 12:15:12.336670 (+ 702us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 12:15:12.336674 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 12:15:12.336676 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0809 12:15:12.336787 (+ 111us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 12:15:12.337132 (+ 345us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 12:15:12.337135 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 12:15:12.337137 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0809 12:15:12.337442 (+ 305us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 12:15:12.337619 (+ 177us) server_negotiation.cc:1035] Waiting for connection context
0809 12:15:12.337934 (+ 315us) server_negotiation.cc:299] Negotiation successful
0809 12:15:12.338118 (+ 184us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":64}
I20260809 12:15:12.342293 21732 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:45524:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260809 12:15:12.349499 21732 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260809 12:15:12.373994 22245 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 12:15:12.362193 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41612 (local address 127.0.222.3:33909)
0809 12:15:12.363956 (+ 1763us) server_negotiation.cc:206] Beginning negotiation
0809 12:15:12.363968 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0809 12:15:12.364010 (+ 42us) server_negotiation.cc:407] Connection header received
0809 12:15:12.364130 (+ 120us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 12:15:12.364141 (+ 11us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 12:15:12.364454 (+ 313us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 12:15:12.364948 (+ 494us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0809 12:15:12.366258 (+ 1310us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 12:15:12.368029 (+ 1771us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 12:15:12.371223 (+ 3194us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 12:15:12.372759 (+ 1536us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 12:15:12.372887 (+ 128us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 12:15:12.373111 (+ 224us) server_negotiation.cc:1035] Waiting for connection context
0809 12:15:12.373469 (+ 358us) server_negotiation.cc:299] Negotiation successful
0809 12:15:12.373647 (+ 178us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1536,"thread_start_us":1329,"threads_started":1}
I20260809 12:15:12.374020 22244 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 12:15:12.361814 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.222.3:33909 (local address 127.0.0.1:41612)
0809 12:15:12.362621 (+ 807us) negotiation.cc:107] Waiting for socket to connect
0809 12:15:12.362678 (+ 57us) client_negotiation.cc:174] Beginning negotiation
0809 12:15:12.362893 (+ 215us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 12:15:12.364751 (+ 1858us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 12:15:12.364762 (+ 11us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 12:15:12.364814 (+ 52us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0809 12:15:12.365765 (+ 951us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 12:15:12.365782 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 12:15:12.368174 (+ 2392us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 12:15:12.368178 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 12:15:12.371017 (+ 2839us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 12:15:12.371039 (+ 22us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 12:15:12.372979 (+ 1940us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 12:15:12.372985 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 12:15:12.373198 (+ 213us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 12:15:12.373260 (+ 62us) client_negotiation.cc:769] Sending connection context
0809 12:15:12.373404 (+ 144us) client_negotiation.cc:240] Negotiation successful
0809 12:15:12.373596 (+ 192us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":536,"mutex_wait_us":43,"thread_start_us":373,"threads_started":1}
I20260809 12:15:12.376220 22248 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 12:15:12.363305 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40382 (local address 127.0.222.1:44429)
0809 12:15:12.367585 (+ 4280us) server_negotiation.cc:206] Beginning negotiation
0809 12:15:12.367605 (+ 20us) server_negotiation.cc:399] Waiting for connection header
0809 12:15:12.367631 (+ 26us) server_negotiation.cc:407] Connection header received
0809 12:15:12.367718 (+ 87us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 12:15:12.367725 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 12:15:12.367947 (+ 222us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 12:15:12.368328 (+ 381us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0809 12:15:12.369370 (+ 1042us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 12:15:12.370950 (+ 1580us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 12:15:12.373570 (+ 2620us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 12:15:12.374904 (+ 1334us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 12:15:12.375089 (+ 185us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 12:15:12.375317 (+ 228us) server_negotiation.cc:1035] Waiting for connection context
0809 12:15:12.375662 (+ 345us) server_negotiation.cc:299] Negotiation successful
0809 12:15:12.375822 (+ 160us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":4070,"thread_start_us":3894,"threads_started":1}
I20260809 12:15:12.376320 22246 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 12:15:12.362610 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.222.1:44429 (local address 127.0.0.1:40382)
0809 12:15:12.366272 (+ 3662us) negotiation.cc:107] Waiting for socket to connect
0809 12:15:12.366289 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0809 12:15:12.366476 (+ 187us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 12:15:12.368168 (+ 1692us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 12:15:12.368172 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 12:15:12.368200 (+ 28us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0809 12:15:12.369056 (+ 856us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 12:15:12.369073 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 12:15:12.371153 (+ 2080us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 12:15:12.371161 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 12:15:12.373354 (+ 2193us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 12:15:12.373394 (+ 40us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 12:15:12.375179 (+ 1785us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 12:15:12.375185 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 12:15:12.375480 (+ 295us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 12:15:12.375513 (+ 33us) client_negotiation.cc:769] Sending connection context
0809 12:15:12.375610 (+ 97us) client_negotiation.cc:240] Negotiation successful
0809 12:15:12.375736 (+ 126us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":3542,"mutex_wait_us":1,"thread_start_us":3339,"threads_started":1}
I20260809 12:15:12.377111 22247 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 12:15:12.363178 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.222.2:32929 (local address 127.0.0.1:48482)
0809 12:15:12.367887 (+ 4709us) negotiation.cc:107] Waiting for socket to connect
0809 12:15:12.367903 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0809 12:15:12.368116 (+ 213us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 12:15:12.368835 (+ 719us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 12:15:12.368839 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 12:15:12.368862 (+ 23us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0809 12:15:12.369444 (+ 582us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 12:15:12.369455 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 12:15:12.370843 (+ 1388us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 12:15:12.370850 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 12:15:12.372920 (+ 2070us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 12:15:12.372934 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 12:15:12.376231 (+ 3297us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 12:15:12.376277 (+ 46us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 12:15:12.376470 (+ 193us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 12:15:12.376526 (+ 56us) client_negotiation.cc:769] Sending connection context
0809 12:15:12.376632 (+ 106us) client_negotiation.cc:240] Negotiation successful
0809 12:15:12.376771 (+ 139us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":4601,"mutex_wait_us":75,"thread_start_us":4466,"threads_started":1}
I20260809 12:15:12.377128 22249 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 12:15:12.363703 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48482 (local address 127.0.222.2:32929)
0809 12:15:12.366947 (+ 3244us) server_negotiation.cc:206] Beginning negotiation
0809 12:15:12.366977 (+ 30us) server_negotiation.cc:399] Waiting for connection header
0809 12:15:12.368008 (+ 1031us) server_negotiation.cc:407] Connection header received
0809 12:15:12.368340 (+ 332us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 12:15:12.368349 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 12:15:12.368559 (+ 210us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 12:15:12.368731 (+ 172us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0809 12:15:12.369618 (+ 887us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 12:15:12.370611 (+ 993us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 12:15:12.373128 (+ 2517us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 12:15:12.375640 (+ 2512us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 12:15:12.375796 (+ 156us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 12:15:12.375982 (+ 186us) server_negotiation.cc:1035] Waiting for connection context
0809 12:15:12.376711 (+ 729us) server_negotiation.cc:299] Negotiation successful
0809 12:15:12.376886 (+ 175us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":3085,"thread_start_us":2963,"threads_started":1}
I20260809 12:15:12.378657 22159 tablet_service.cc:1467] Processing CreateTablet for tablet 6ec53971c4c04e45a9cba284cabb1fa8 (DEFAULT_TABLE table=test-table [id=e985fca0125c487fb98b5978f75ab0e6]), partition=RANGE (key) PARTITION UNBOUNDED
I20260809 12:15:12.380304 22159 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 6ec53971c4c04e45a9cba284cabb1fa8. 1 dirs total, 0 dirs full, 0 dirs failed
I20260809 12:15:12.380645 22009 tablet_service.cc:1467] Processing CreateTablet for tablet 6ec53971c4c04e45a9cba284cabb1fa8 (DEFAULT_TABLE table=test-table [id=e985fca0125c487fb98b5978f75ab0e6]), partition=RANGE (key) PARTITION UNBOUNDED
I20260809 12:15:12.380640 21865 tablet_service.cc:1467] Processing CreateTablet for tablet 6ec53971c4c04e45a9cba284cabb1fa8 (DEFAULT_TABLE table=test-table [id=e985fca0125c487fb98b5978f75ab0e6]), partition=RANGE (key) PARTITION UNBOUNDED
I20260809 12:15:12.382023 21865 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 6ec53971c4c04e45a9cba284cabb1fa8. 1 dirs total, 0 dirs full, 0 dirs failed
I20260809 12:15:12.382025 22009 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 6ec53971c4c04e45a9cba284cabb1fa8. 1 dirs total, 0 dirs full, 0 dirs failed
I20260809 12:15:12.392637 22250 tablet_bootstrap.cc:492] T 6ec53971c4c04e45a9cba284cabb1fa8 P 48112dd60dbc49f49247eb7f61b9cce2: Bootstrap starting.
I20260809 12:15:12.395884 22250 tablet_bootstrap.cc:654] T 6ec53971c4c04e45a9cba284cabb1fa8 P 48112dd60dbc49f49247eb7f61b9cce2: Neither blocks nor log segments found. Creating new log.
I20260809 12:15:12.396802 22252 tablet_bootstrap.cc:492] T 6ec53971c4c04e45a9cba284cabb1fa8 P f9753698cc3c4413a465e4ab5db2c61a: Bootstrap starting.
I20260809 12:15:12.397692 22250 log.cc:824] T 6ec53971c4c04e45a9cba284cabb1fa8 P 48112dd60dbc49f49247eb7f61b9cce2: Log is configured to *not* fsync() on all Append() calls
I20260809 12:15:12.399300 22251 tablet_bootstrap.cc:492] T 6ec53971c4c04e45a9cba284cabb1fa8 P 09204b15d804406a80c791bee0de9329: Bootstrap starting.
I20260809 12:15:12.401225 22252 tablet_bootstrap.cc:654] T 6ec53971c4c04e45a9cba284cabb1fa8 P f9753698cc3c4413a465e4ab5db2c61a: Neither blocks nor log segments found. Creating new log.
I20260809 12:15:12.402503 22252 log.cc:824] T 6ec53971c4c04e45a9cba284cabb1fa8 P f9753698cc3c4413a465e4ab5db2c61a: Log is configured to *not* fsync() on all Append() calls
I20260809 12:15:12.402877 22251 tablet_bootstrap.cc:654] T 6ec53971c4c04e45a9cba284cabb1fa8 P 09204b15d804406a80c791bee0de9329: Neither blocks nor log segments found. Creating new log.
I20260809 12:15:12.404027 22251 log.cc:824] T 6ec53971c4c04e45a9cba284cabb1fa8 P 09204b15d804406a80c791bee0de9329: Log is configured to *not* fsync() on all Append() calls
I20260809 12:15:12.404652 22250 tablet_bootstrap.cc:492] T 6ec53971c4c04e45a9cba284cabb1fa8 P 48112dd60dbc49f49247eb7f61b9cce2: No bootstrap required, opened a new log
I20260809 12:15:12.405014 22250 ts_tablet_manager.cc:1397] T 6ec53971c4c04e45a9cba284cabb1fa8 P 48112dd60dbc49f49247eb7f61b9cce2: Time spent bootstrapping tablet: real 0.013s user 0.008s sys 0.000s
I20260809 12:15:12.405658 22252 tablet_bootstrap.cc:492] T 6ec53971c4c04e45a9cba284cabb1fa8 P f9753698cc3c4413a465e4ab5db2c61a: No bootstrap required, opened a new log
I20260809 12:15:12.406013 22252 ts_tablet_manager.cc:1397] T 6ec53971c4c04e45a9cba284cabb1fa8 P f9753698cc3c4413a465e4ab5db2c61a: Time spent bootstrapping tablet: real 0.009s user 0.000s sys 0.008s
I20260809 12:15:12.407173 22251 tablet_bootstrap.cc:492] T 6ec53971c4c04e45a9cba284cabb1fa8 P 09204b15d804406a80c791bee0de9329: No bootstrap required, opened a new log
I20260809 12:15:12.407425 22251 ts_tablet_manager.cc:1397] T 6ec53971c4c04e45a9cba284cabb1fa8 P 09204b15d804406a80c791bee0de9329: Time spent bootstrapping tablet: real 0.008s user 0.007s sys 0.000s
I20260809 12:15:12.412778 22250 raft_consensus.cc:348] T 6ec53971c4c04e45a9cba284cabb1fa8 P 48112dd60dbc49f49247eb7f61b9cce2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "48112dd60dbc49f49247eb7f61b9cce2" member_type: VOTER last_known_addr { host: "127.0.222.3" port: 33909 } } peers { permanent_uuid: "f9753698cc3c4413a465e4ab5db2c61a" member_type: VOTER last_known_addr { host: "127.0.222.2" port: 32929 } } peers { permanent_uuid: "09204b15d804406a80c791bee0de9329" member_type: VOTER last_known_addr { host: "127.0.222.1" port: 44429 } }
I20260809 12:15:12.413166 22250 raft_consensus.cc:374] T 6ec53971c4c04e45a9cba284cabb1fa8 P 48112dd60dbc49f49247eb7f61b9cce2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260809 12:15:12.413276 22250 raft_consensus.cc:729] T 6ec53971c4c04e45a9cba284cabb1fa8 P 48112dd60dbc49f49247eb7f61b9cce2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 48112dd60dbc49f49247eb7f61b9cce2, State: Initialized, Role: FOLLOWER
I20260809 12:15:12.413920 22250 consensus_queue.cc:260] T 6ec53971c4c04e45a9cba284cabb1fa8 P 48112dd60dbc49f49247eb7f61b9cce2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "48112dd60dbc49f49247eb7f61b9cce2" member_type: VOTER last_known_addr { host: "127.0.222.3" port: 33909 } } peers { permanent_uuid: "f9753698cc3c4413a465e4ab5db2c61a" member_type: VOTER last_known_addr { host: "127.0.222.2" port: 32929 } } peers { permanent_uuid: "09204b15d804406a80c791bee0de9329" member_type: VOTER last_known_addr { host: "127.0.222.1" port: 44429 } }
I20260809 12:15:12.415086 22230 heartbeater.cc:503] Master 127.0.222.62:33407 was elected leader, sending a full tablet report...
I20260809 12:15:12.417101 22250 ts_tablet_manager.cc:1428] T 6ec53971c4c04e45a9cba284cabb1fa8 P 48112dd60dbc49f49247eb7f61b9cce2: Time spent starting tablet: real 0.012s user 0.010s sys 0.000s
I20260809 12:15:12.418346 22252 raft_consensus.cc:348] T 6ec53971c4c04e45a9cba284cabb1fa8 P f9753698cc3c4413a465e4ab5db2c61a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "48112dd60dbc49f49247eb7f61b9cce2" member_type: VOTER last_known_addr { host: "127.0.222.3" port: 33909 } } peers { permanent_uuid: "f9753698cc3c4413a465e4ab5db2c61a" member_type: VOTER last_known_addr { host: "127.0.222.2" port: 32929 } } peers { permanent_uuid: "09204b15d804406a80c791bee0de9329" member_type: VOTER last_known_addr { host: "127.0.222.1" port: 44429 } }
I20260809 12:15:12.418246 22251 raft_consensus.cc:348] T 6ec53971c4c04e45a9cba284cabb1fa8 P 09204b15d804406a80c791bee0de9329 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "48112dd60dbc49f49247eb7f61b9cce2" member_type: VOTER last_known_addr { host: "127.0.222.3" port: 33909 } } peers { permanent_uuid: "f9753698cc3c4413a465e4ab5db2c61a" member_type: VOTER last_known_addr { host: "127.0.222.2" port: 32929 } } peers { permanent_uuid: "09204b15d804406a80c791bee0de9329" member_type: VOTER last_known_addr { host: "127.0.222.1" port: 44429 } }
I20260809 12:15:12.418730 22252 raft_consensus.cc:374] T 6ec53971c4c04e45a9cba284cabb1fa8 P f9753698cc3c4413a465e4ab5db2c61a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260809 12:15:12.418735 22251 raft_consensus.cc:374] T 6ec53971c4c04e45a9cba284cabb1fa8 P 09204b15d804406a80c791bee0de9329 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260809 12:15:12.418833 22251 raft_consensus.cc:729] T 6ec53971c4c04e45a9cba284cabb1fa8 P 09204b15d804406a80c791bee0de9329 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 09204b15d804406a80c791bee0de9329, State: Initialized, Role: FOLLOWER
I20260809 12:15:12.418845 22252 raft_consensus.cc:729] T 6ec53971c4c04e45a9cba284cabb1fa8 P f9753698cc3c4413a465e4ab5db2c61a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f9753698cc3c4413a465e4ab5db2c61a, State: Initialized, Role: FOLLOWER
I20260809 12:15:12.419512 22252 consensus_queue.cc:260] T 6ec53971c4c04e45a9cba284cabb1fa8 P f9753698cc3c4413a465e4ab5db2c61a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "48112dd60dbc49f49247eb7f61b9cce2" member_type: VOTER last_known_addr { host: "127.0.222.3" port: 33909 } } peers { permanent_uuid: "f9753698cc3c4413a465e4ab5db2c61a" member_type: VOTER last_known_addr { host: "127.0.222.2" port: 32929 } } peers { permanent_uuid: "09204b15d804406a80c791bee0de9329" member_type: VOTER last_known_addr { host: "127.0.222.1" port: 44429 } }
I20260809 12:15:12.419647 22251 consensus_queue.cc:260] T 6ec53971c4c04e45a9cba284cabb1fa8 P 09204b15d804406a80c791bee0de9329 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "48112dd60dbc49f49247eb7f61b9cce2" member_type: VOTER last_known_addr { host: "127.0.222.3" port: 33909 } } peers { permanent_uuid: "f9753698cc3c4413a465e4ab5db2c61a" member_type: VOTER last_known_addr { host: "127.0.222.2" port: 32929 } } peers { permanent_uuid: "09204b15d804406a80c791bee0de9329" member_type: VOTER last_known_addr { host: "127.0.222.1" port: 44429 } }
I20260809 12:15:12.421612 22251 ts_tablet_manager.cc:1428] T 6ec53971c4c04e45a9cba284cabb1fa8 P 09204b15d804406a80c791bee0de9329: Time spent starting tablet: real 0.014s user 0.013s sys 0.000s
I20260809 12:15:12.422649 22081 heartbeater.cc:503] Master 127.0.222.62:33407 was elected leader, sending a full tablet report...
I20260809 12:15:12.423285 22252 ts_tablet_manager.cc:1428] T 6ec53971c4c04e45a9cba284cabb1fa8 P f9753698cc3c4413a465e4ab5db2c61a: Time spent starting tablet: real 0.017s user 0.008s sys 0.007s
W20260809 12:15:12.439309 21936 tablet.cc:2369] T 6ec53971c4c04e45a9cba284cabb1fa8 P 09204b15d804406a80c791bee0de9329: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260809 12:15:12.489229 22082 tablet.cc:2369] T 6ec53971c4c04e45a9cba284cabb1fa8 P f9753698cc3c4413a465e4ab5db2c61a: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260809 12:15:12.542448 22231 tablet.cc:2369] T 6ec53971c4c04e45a9cba284cabb1fa8 P 48112dd60dbc49f49247eb7f61b9cce2: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260809 12:15:12.548066 22259 raft_consensus.cc:482] T 6ec53971c4c04e45a9cba284cabb1fa8 P 09204b15d804406a80c791bee0de9329 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260809 12:15:12.548403 22259 raft_consensus.cc:504] T 6ec53971c4c04e45a9cba284cabb1fa8 P 09204b15d804406a80c791bee0de9329 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "48112dd60dbc49f49247eb7f61b9cce2" member_type: VOTER last_known_addr { host: "127.0.222.3" port: 33909 } } peers { permanent_uuid: "f9753698cc3c4413a465e4ab5db2c61a" member_type: VOTER last_known_addr { host: "127.0.222.2" port: 32929 } } peers { permanent_uuid: "09204b15d804406a80c791bee0de9329" member_type: VOTER last_known_addr { host: "127.0.222.1" port: 44429 } }
I20260809 12:15:12.550118 22259 leader_election.cc:288] T 6ec53971c4c04e45a9cba284cabb1fa8 P 09204b15d804406a80c791bee0de9329 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 48112dd60dbc49f49247eb7f61b9cce2 (127.0.222.3:33909), f9753698cc3c4413a465e4ab5db2c61a (127.0.222.2:32929)
I20260809 12:15:12.557724 22245 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 12:15:12.550605 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.222.1:45051 (local address 127.0.222.3:33909)
0809 12:15:12.550795 (+ 190us) server_negotiation.cc:206] Beginning negotiation
0809 12:15:12.550800 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0809 12:15:12.551315 (+ 515us) server_negotiation.cc:407] Connection header received
0809 12:15:12.551561 (+ 246us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 12:15:12.551567 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 12:15:12.551652 (+ 85us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 12:15:12.551728 (+ 76us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0809 12:15:12.552703 (+ 975us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 12:15:12.553795 (+ 1092us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 12:15:12.555700 (+ 1905us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 12:15:12.556585 (+ 885us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 12:15:12.556735 (+ 150us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 12:15:12.556854 (+ 119us) server_negotiation.cc:1035] Waiting for connection context
0809 12:15:12.557319 (+ 465us) server_negotiation.cc:299] Negotiation successful
0809 12:15:12.557457 (+ 138us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":63}
I20260809 12:15:12.558054 22261 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 12:15:12.551263 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.222.2:32929 (local address 127.0.222.1:39135)
0809 12:15:12.551942 (+ 679us) negotiation.cc:107] Waiting for socket to connect
0809 12:15:12.551961 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0809 12:15:12.552219 (+ 258us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 12:15:12.552737 (+ 518us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 12:15:12.552741 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 12:15:12.552767 (+ 26us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0809 12:15:12.553111 (+ 344us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 12:15:12.553118 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 12:15:12.554328 (+ 1210us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 12:15:12.554333 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 12:15:12.555888 (+ 1555us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 12:15:12.555901 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 12:15:12.557215 (+ 1314us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 12:15:12.557224 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 12:15:12.557393 (+ 169us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 12:15:12.557426 (+ 33us) client_negotiation.cc:769] Sending connection context
0809 12:15:12.557574 (+ 148us) client_negotiation.cc:240] Negotiation successful
0809 12:15:12.557695 (+ 121us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":513,"thread_start_us":766,"threads_started":1}
I20260809 12:15:12.558454 22260 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 12:15:12.550414 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.222.3:33909 (local address 127.0.222.1:45051)
0809 12:15:12.551150 (+ 736us) negotiation.cc:107] Waiting for socket to connect
0809 12:15:12.551190 (+ 40us) client_negotiation.cc:174] Beginning negotiation
0809 12:15:12.551350 (+ 160us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 12:15:12.551974 (+ 624us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 12:15:12.551995 (+ 21us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 12:15:12.552031 (+ 36us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0809 12:15:12.552485 (+ 454us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 12:15:12.552502 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 12:15:12.553918 (+ 1416us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 12:15:12.553922 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 12:15:12.555542 (+ 1620us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 12:15:12.555555 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 12:15:12.556798 (+ 1243us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 12:15:12.556804 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 12:15:12.557102 (+ 298us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 12:15:12.557150 (+ 48us) client_negotiation.cc:769] Sending connection context
0809 12:15:12.557949 (+ 799us) client_negotiation.cc:240] Negotiation successful
0809 12:15:12.558096 (+ 147us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":589,"thread_start_us":372,"threads_started":1}
I20260809 12:15:12.558485 22249 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 12:15:12.551629 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.222.1:39135 (local address 127.0.222.2:32929)
0809 12:15:12.551842 (+ 213us) server_negotiation.cc:206] Beginning negotiation
0809 12:15:12.551848 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0809 12:15:12.552083 (+ 235us) server_negotiation.cc:407] Connection header received
0809 12:15:12.552438 (+ 355us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 12:15:12.552443 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 12:15:12.552597 (+ 154us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 12:15:12.552726 (+ 129us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0809 12:15:12.553268 (+ 542us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 12:15:12.554174 (+ 906us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 12:15:12.556039 (+ 1865us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 12:15:12.556987 (+ 948us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 12:15:12.557133 (+ 146us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 12:15:12.557253 (+ 120us) server_negotiation.cc:1035] Waiting for connection context
0809 12:15:12.557635 (+ 382us) server_negotiation.cc:299] Negotiation successful
0809 12:15:12.557770 (+ 135us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":78}
I20260809 12:15:12.559449 22031 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6ec53971c4c04e45a9cba284cabb1fa8" candidate_uuid: "09204b15d804406a80c791bee0de9329" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f9753698cc3c4413a465e4ab5db2c61a" is_pre_election: true
I20260809 12:15:12.559530 22181 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6ec53971c4c04e45a9cba284cabb1fa8" candidate_uuid: "09204b15d804406a80c791bee0de9329" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "48112dd60dbc49f49247eb7f61b9cce2" is_pre_election: true
I20260809 12:15:12.560024 22031 raft_consensus.cc:2454] T 6ec53971c4c04e45a9cba284cabb1fa8 P f9753698cc3c4413a465e4ab5db2c61a [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 09204b15d804406a80c791bee0de9329 in term 0.
I20260809 12:15:12.560091 22181 raft_consensus.cc:2454] T 6ec53971c4c04e45a9cba284cabb1fa8 P 48112dd60dbc49f49247eb7f61b9cce2 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 09204b15d804406a80c791bee0de9329 in term 0.
I20260809 12:15:12.560751 21813 leader_election.cc:302] T 6ec53971c4c04e45a9cba284cabb1fa8 P 09204b15d804406a80c791bee0de9329 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 09204b15d804406a80c791bee0de9329, f9753698cc3c4413a465e4ab5db2c61a; no voters:
I20260809 12:15:12.561156 22259 raft_consensus.cc:2781] T 6ec53971c4c04e45a9cba284cabb1fa8 P 09204b15d804406a80c791bee0de9329 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260809 12:15:12.561321 22259 raft_consensus.cc:482] T 6ec53971c4c04e45a9cba284cabb1fa8 P 09204b15d804406a80c791bee0de9329 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260809 12:15:12.561465 22259 raft_consensus.cc:3037] T 6ec53971c4c04e45a9cba284cabb1fa8 P 09204b15d804406a80c791bee0de9329 [term 0 FOLLOWER]: Advancing to term 1
I20260809 12:15:12.563807 22259 raft_consensus.cc:504] T 6ec53971c4c04e45a9cba284cabb1fa8 P 09204b15d804406a80c791bee0de9329 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "48112dd60dbc49f49247eb7f61b9cce2" member_type: VOTER last_known_addr { host: "127.0.222.3" port: 33909 } } peers { permanent_uuid: "f9753698cc3c4413a465e4ab5db2c61a" member_type: VOTER last_known_addr { host: "127.0.222.2" port: 32929 } } peers { permanent_uuid: "09204b15d804406a80c791bee0de9329" member_type: VOTER last_known_addr { host: "127.0.222.1" port: 44429 } }
I20260809 12:15:12.564443 22259 leader_election.cc:288] T 6ec53971c4c04e45a9cba284cabb1fa8 P 09204b15d804406a80c791bee0de9329 [CANDIDATE]: Term 1 election: Requested vote from peers 48112dd60dbc49f49247eb7f61b9cce2 (127.0.222.3:33909), f9753698cc3c4413a465e4ab5db2c61a (127.0.222.2:32929)
I20260809 12:15:12.564867 22181 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6ec53971c4c04e45a9cba284cabb1fa8" candidate_uuid: "09204b15d804406a80c791bee0de9329" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "48112dd60dbc49f49247eb7f61b9cce2"
I20260809 12:15:12.564956 22031 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6ec53971c4c04e45a9cba284cabb1fa8" candidate_uuid: "09204b15d804406a80c791bee0de9329" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f9753698cc3c4413a465e4ab5db2c61a"
I20260809 12:15:12.565073 22181 raft_consensus.cc:3037] T 6ec53971c4c04e45a9cba284cabb1fa8 P 48112dd60dbc49f49247eb7f61b9cce2 [term 0 FOLLOWER]: Advancing to term 1
I20260809 12:15:12.565210 22031 raft_consensus.cc:3037] T 6ec53971c4c04e45a9cba284cabb1fa8 P f9753698cc3c4413a465e4ab5db2c61a [term 0 FOLLOWER]: Advancing to term 1
I20260809 12:15:12.567682 22181 raft_consensus.cc:2454] T 6ec53971c4c04e45a9cba284cabb1fa8 P 48112dd60dbc49f49247eb7f61b9cce2 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 09204b15d804406a80c791bee0de9329 in term 1.
I20260809 12:15:12.567723 22031 raft_consensus.cc:2454] T 6ec53971c4c04e45a9cba284cabb1fa8 P f9753698cc3c4413a465e4ab5db2c61a [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 09204b15d804406a80c791bee0de9329 in term 1.
I20260809 12:15:12.568081 21813 leader_election.cc:302] T 6ec53971c4c04e45a9cba284cabb1fa8 P 09204b15d804406a80c791bee0de9329 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 09204b15d804406a80c791bee0de9329, 48112dd60dbc49f49247eb7f61b9cce2; no voters:
I20260809 12:15:12.568378 22259 raft_consensus.cc:2781] T 6ec53971c4c04e45a9cba284cabb1fa8 P 09204b15d804406a80c791bee0de9329 [term 1 FOLLOWER]: Leader election won for term 1
I20260809 12:15:12.568838 22259 raft_consensus.cc:686] T 6ec53971c4c04e45a9cba284cabb1fa8 P 09204b15d804406a80c791bee0de9329 [term 1 LEADER]: Becoming Leader. State: Replica: 09204b15d804406a80c791bee0de9329, State: Running, Role: LEADER
I20260809 12:15:12.569331 22259 consensus_queue.cc:237] T 6ec53971c4c04e45a9cba284cabb1fa8 P 09204b15d804406a80c791bee0de9329 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "48112dd60dbc49f49247eb7f61b9cce2" member_type: VOTER last_known_addr { host: "127.0.222.3" port: 33909 } } peers { permanent_uuid: "f9753698cc3c4413a465e4ab5db2c61a" member_type: VOTER last_known_addr { host: "127.0.222.2" port: 32929 } } peers { permanent_uuid: "09204b15d804406a80c791bee0de9329" member_type: VOTER last_known_addr { host: "127.0.222.1" port: 44429 } }
I20260809 12:15:12.574326 21731 catalog_manager.cc:5495] T 6ec53971c4c04e45a9cba284cabb1fa8 P 09204b15d804406a80c791bee0de9329 reported cstate change: term changed from 0 to 1, leader changed from <none> to 09204b15d804406a80c791bee0de9329 (127.0.222.1). New cstate: current_term: 1 leader_uuid: "09204b15d804406a80c791bee0de9329" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "48112dd60dbc49f49247eb7f61b9cce2" member_type: VOTER last_known_addr { host: "127.0.222.3" port: 33909 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "f9753698cc3c4413a465e4ab5db2c61a" member_type: VOTER last_known_addr { host: "127.0.222.2" port: 32929 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "09204b15d804406a80c791bee0de9329" member_type: VOTER last_known_addr { host: "127.0.222.1" port: 44429 } health_report { overall_health: HEALTHY } } }
I20260809 12:15:12.616854 22248 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 12:15:12.611107 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40394 (local address 127.0.222.1:44429)
0809 12:15:12.611369 (+ 262us) server_negotiation.cc:206] Beginning negotiation
0809 12:15:12.611375 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0809 12:15:12.611456 (+ 81us) server_negotiation.cc:407] Connection header received
0809 12:15:12.611697 (+ 241us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 12:15:12.611701 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 12:15:12.611782 (+ 81us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 12:15:12.611952 (+ 170us) server_negotiation.cc:226] Negotiated authn=TOKEN
0809 12:15:12.612338 (+ 386us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 12:15:12.613068 (+ 730us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 12:15:12.614534 (+ 1466us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 12:15:12.615025 (+ 491us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 12:15:12.615161 (+ 136us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 12:15:12.615697 (+ 536us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0809 12:15:12.616219 (+ 522us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0809 12:15:12.616435 (+ 216us) server_negotiation.cc:1035] Waiting for connection context
0809 12:15:12.616490 (+ 55us) server_negotiation.cc:299] Negotiation successful
0809 12:15:12.616577 (+ 87us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":107}
I20260809 12:15:12.625602 22181 raft_consensus.cc:1261] T 6ec53971c4c04e45a9cba284cabb1fa8 P 48112dd60dbc49f49247eb7f61b9cce2 [term 1 FOLLOWER]: Refusing update from remote peer 09204b15d804406a80c791bee0de9329: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260809 12:15:12.625595 22031 raft_consensus.cc:1261] T 6ec53971c4c04e45a9cba284cabb1fa8 P f9753698cc3c4413a465e4ab5db2c61a [term 1 FOLLOWER]: Refusing update from remote peer 09204b15d804406a80c791bee0de9329: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260809 12:15:12.626641 22262 consensus_queue.cc:1035] T 6ec53971c4c04e45a9cba284cabb1fa8 P 09204b15d804406a80c791bee0de9329 [LEADER]: Connected to new peer: Peer: permanent_uuid: "48112dd60dbc49f49247eb7f61b9cce2" member_type: VOTER last_known_addr { host: "127.0.222.3" port: 33909 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260809 12:15:12.627029 22259 consensus_queue.cc:1035] T 6ec53971c4c04e45a9cba284cabb1fa8 P 09204b15d804406a80c791bee0de9329 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f9753698cc3c4413a465e4ab5db2c61a" member_type: VOTER last_known_addr { host: "127.0.222.2" port: 32929 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260809 12:15:12.638983 22266 mvcc.cc:205] Tried to move back new op lower bound from 7316593510902865920 to 7316593510687698944. Current Snapshot: MvccSnapshot[applied={T|T < 7316593510902865920}]
I20260809 12:15:12.642691 22267 mvcc.cc:205] Tried to move back new op lower bound from 7316593510902865920 to 7316593510687698944. Current Snapshot: MvccSnapshot[applied={T|T < 7316593510902865920}]
I20260809 12:15:12.658874 21731 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:45524:
table { table_name: "test-table" } modify_external_catalogs: true
I20260809 12:15:12.659229 21731 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:45524:
table { table_name: "test-table" } modify_external_catalogs: true
I20260809 12:15:12.662914 21731 catalog_manager.cc:5782] T 00000000000000000000000000000000 P e4dcb620a4014a28a6f4c81962b03518: Sending DeleteTablet for 3 replicas of tablet 6ec53971c4c04e45a9cba284cabb1fa8
I20260809 12:15:12.664149 21865 tablet_service.cc:1514] Processing DeleteTablet for tablet 6ec53971c4c04e45a9cba284cabb1fa8 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-09 12:15:12 UTC) from {username='kudu', principal='kudu/127.0.222.62@KRBTEST.COM'} at 127.0.0.1:40382
I20260809 12:15:12.664139 22159 tablet_service.cc:1514] Processing DeleteTablet for tablet 6ec53971c4c04e45a9cba284cabb1fa8 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-09 12:15:12 UTC) from {username='kudu', principal='kudu/127.0.222.62@KRBTEST.COM'} at 127.0.0.1:41612
I20260809 12:15:12.664186 22009 tablet_service.cc:1514] Processing DeleteTablet for tablet 6ec53971c4c04e45a9cba284cabb1fa8 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-09 12:15:12 UTC) from {username='kudu', principal='kudu/127.0.222.62@KRBTEST.COM'} at 127.0.0.1:48482
I20260809 12:15:12.665203 22278 tablet_replica.cc:330] T 6ec53971c4c04e45a9cba284cabb1fa8 P f9753698cc3c4413a465e4ab5db2c61a: stopping tablet replica
I20260809 12:15:12.665405 22275 tablet_replica.cc:330] T 6ec53971c4c04e45a9cba284cabb1fa8 P 09204b15d804406a80c791bee0de9329: stopping tablet replica
I20260809 12:15:12.665777 888 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskgH5Pjm/build/asan/bin/kudu pid 21796
I20260809 12:15:12.665648 22278 raft_consensus.cc:2229] T 6ec53971c4c04e45a9cba284cabb1fa8 P f9753698cc3c4413a465e4ab5db2c61a [term 1 FOLLOWER]: Raft consensus shutting down.
I20260809 12:15:12.666203 22276 tablet_replica.cc:330] T 6ec53971c4c04e45a9cba284cabb1fa8 P 48112dd60dbc49f49247eb7f61b9cce2: stopping tablet replica
I20260809 12:15:12.666561 22278 raft_consensus.cc:2258] T 6ec53971c4c04e45a9cba284cabb1fa8 P f9753698cc3c4413a465e4ab5db2c61a [term 1 FOLLOWER]: Raft consensus is shut down!
I20260809 12:15:12.666605 22276 raft_consensus.cc:2229] T 6ec53971c4c04e45a9cba284cabb1fa8 P 48112dd60dbc49f49247eb7f61b9cce2 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260809 12:15:12.666951 22275 raft_consensus.cc:2229] T 6ec53971c4c04e45a9cba284cabb1fa8 P 09204b15d804406a80c791bee0de9329 [term 1 LEADER]: Raft consensus shutting down.
I20260809 12:15:12.667917 22276 raft_consensus.cc:2258] T 6ec53971c4c04e45a9cba284cabb1fa8 P 48112dd60dbc49f49247eb7f61b9cce2 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260809 12:15:12.669596 22278 ts_tablet_manager.cc:1905] T 6ec53971c4c04e45a9cba284cabb1fa8 P f9753698cc3c4413a465e4ab5db2c61a: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260809 12:15:12.669950 22276 ts_tablet_manager.cc:1905] T 6ec53971c4c04e45a9cba284cabb1fa8 P 48112dd60dbc49f49247eb7f61b9cce2: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260809 12:15:12.670387 22275 raft_consensus.cc:2258] T 6ec53971c4c04e45a9cba284cabb1fa8 P 09204b15d804406a80c791bee0de9329 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260809 12:15:12.675534 22275 ts_tablet_manager.cc:1905] T 6ec53971c4c04e45a9cba284cabb1fa8 P 09204b15d804406a80c791bee0de9329: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260809 12:15:12.677217 22276 ts_tablet_manager.cc:1918] T 6ec53971c4c04e45a9cba284cabb1fa8 P 48112dd60dbc49f49247eb7f61b9cce2: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260809 12:15:12.677439 22276 log.cc:1196] T 6ec53971c4c04e45a9cba284cabb1fa8 P 48112dd60dbc49f49247eb7f61b9cce2: Deleting WAL directory at /tmp/dist-test-taskgH5Pjm/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786277382073938-888-0/minicluster-data/ts-2/wal/wals/6ec53971c4c04e45a9cba284cabb1fa8
I20260809 12:15:12.677896 22278 ts_tablet_manager.cc:1918] T 6ec53971c4c04e45a9cba284cabb1fa8 P f9753698cc3c4413a465e4ab5db2c61a: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260809 12:15:12.678083 22276 ts_tablet_manager.cc:1939] T 6ec53971c4c04e45a9cba284cabb1fa8 P 48112dd60dbc49f49247eb7f61b9cce2: Deleting consensus metadata
I20260809 12:15:12.678171 22278 log.cc:1196] T 6ec53971c4c04e45a9cba284cabb1fa8 P f9753698cc3c4413a465e4ab5db2c61a: Deleting WAL directory at /tmp/dist-test-taskgH5Pjm/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786277382073938-888-0/minicluster-data/ts-1/wal/wals/6ec53971c4c04e45a9cba284cabb1fa8
I20260809 12:15:12.679198 22278 ts_tablet_manager.cc:1939] T 6ec53971c4c04e45a9cba284cabb1fa8 P f9753698cc3c4413a465e4ab5db2c61a: Deleting consensus metadata
I20260809 12:15:12.680513 21716 catalog_manager.cc:4841] TS 48112dd60dbc49f49247eb7f61b9cce2 (127.0.222.3:33909): tablet 6ec53971c4c04e45a9cba284cabb1fa8 (table test-table [id=e985fca0125c487fb98b5978f75ab0e6]) successfully deleted
I20260809 12:15:12.680914 21716 catalog_manager.cc:4841] TS f9753698cc3c4413a465e4ab5db2c61a (127.0.222.2:32929): tablet 6ec53971c4c04e45a9cba284cabb1fa8 (table test-table [id=e985fca0125c487fb98b5978f75ab0e6]) successfully deleted
I20260809 12:15:12.683465 22275 ts_tablet_manager.cc:1918] T 6ec53971c4c04e45a9cba284cabb1fa8 P 09204b15d804406a80c791bee0de9329: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260809 12:15:12.683665 22275 log.cc:1196] T 6ec53971c4c04e45a9cba284cabb1fa8 P 09204b15d804406a80c791bee0de9329: Deleting WAL directory at /tmp/dist-test-taskgH5Pjm/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786277382073938-888-0/minicluster-data/ts-0/wal/wals/6ec53971c4c04e45a9cba284cabb1fa8
I20260809 12:15:12.684250 22275 ts_tablet_manager.cc:1939] T 6ec53971c4c04e45a9cba284cabb1fa8 P 09204b15d804406a80c791bee0de9329: Deleting consensus metadata
Aug 09 12:15:12 dist-test-slave-2w3w krb5kdc[21679](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786277709, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.222.1@KRBTEST.COM
I20260809 12:15:12.686084 21715 catalog_manager.cc:4841] TS 09204b15d804406a80c791bee0de9329 (127.0.222.1:44429): tablet 6ec53971c4c04e45a9cba284cabb1fa8 (table test-table [id=e985fca0125c487fb98b5978f75ab0e6]) successfully deleted
I20260809 12:15:12.693197 22248 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 12:15:12.666820 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40408 (local address 127.0.222.1:44429)
0809 12:15:12.667022 (+ 202us) server_negotiation.cc:206] Beginning negotiation
0809 12:15:12.667031 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0809 12:15:12.667958 (+ 927us) server_negotiation.cc:407] Connection header received
0809 12:15:12.668150 (+ 192us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 12:15:12.668157 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 12:15:12.668236 (+ 79us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 12:15:12.668342 (+ 106us) server_negotiation.cc:226] Negotiated authn=SASL
0809 12:15:12.670377 (+ 2035us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 12:15:12.671828 (+ 1451us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 12:15:12.681263 (+ 9435us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 12:15:12.681805 (+ 542us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 12:15:12.681979 (+ 174us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 12:15:12.686810 (+ 4831us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 12:15:12.686821 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 12:15:12.686837 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0809 12:15:12.686883 (+ 46us) server_negotiation.cc:922] calling sasl_server_start()
0809 12:15:12.690079 (+ 3196us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 12:15:12.690844 (+ 765us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 12:15:12.690852 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 12:15:12.690854 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0809 12:15:12.691107 (+ 253us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 12:15:12.691530 (+ 423us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 12:15:12.691535 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 12:15:12.691539 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0809 12:15:12.691916 (+ 377us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 12:15:12.692232 (+ 316us) server_negotiation.cc:1035] Waiting for connection context
0809 12:15:12.692586 (+ 354us) server_negotiation.cc:299] Negotiation successful
0809 12:15:12.692839 (+ 253us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":53}
I20260809 12:15:12.693964 21930 generic_service.cc:196] Checking for leaks (request via RPC)
I20260809 12:15:12.808671 888 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskgH5Pjm/build/asan/bin/kudu with pid 21796
I20260809 12:15:12.830435 888 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskgH5Pjm/build/asan/bin/kudu pid 21941
Aug 09 12:15:12 dist-test-slave-2w3w krb5kdc[21679](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786277709, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.222.2@KRBTEST.COM
I20260809 12:15:12.846302 22249 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 12:15:12.831238 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48486 (local address 127.0.222.2:32929)
0809 12:15:12.831413 (+ 175us) server_negotiation.cc:206] Beginning negotiation
0809 12:15:12.831423 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0809 12:15:12.831447 (+ 24us) server_negotiation.cc:407] Connection header received
0809 12:15:12.831626 (+ 179us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 12:15:12.831630 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 12:15:12.831703 (+ 73us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 12:15:12.831792 (+ 89us) server_negotiation.cc:226] Negotiated authn=SASL
0809 12:15:12.833105 (+ 1313us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 12:15:12.833847 (+ 742us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 12:15:12.835235 (+ 1388us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 12:15:12.835629 (+ 394us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 12:15:12.835724 (+ 95us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 12:15:12.839980 (+ 4256us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 12:15:12.839997 (+ 17us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 12:15:12.840014 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0809 12:15:12.840049 (+ 35us) server_negotiation.cc:922] calling sasl_server_start()
0809 12:15:12.843659 (+ 3610us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 12:15:12.844380 (+ 721us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 12:15:12.844388 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 12:15:12.844390 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0809 12:15:12.844517 (+ 127us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 12:15:12.844871 (+ 354us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 12:15:12.844875 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 12:15:12.844878 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0809 12:15:12.845270 (+ 392us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 12:15:12.845510 (+ 240us) server_negotiation.cc:1035] Waiting for connection context
0809 12:15:12.845768 (+ 258us) server_negotiation.cc:299] Negotiation successful
0809 12:15:12.845958 (+ 190us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":60}
I20260809 12:15:12.847048 22075 generic_service.cc:196] Checking for leaks (request via RPC)
I20260809 12:15:12.937461 888 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskgH5Pjm/build/asan/bin/kudu with pid 21941
I20260809 12:15:12.958488 888 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskgH5Pjm/build/asan/bin/kudu pid 22088
Aug 09 12:15:12 dist-test-slave-2w3w krb5kdc[21679](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786277709, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.222.3@KRBTEST.COM
I20260809 12:15:12.974445 22245 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 12:15:12.959261 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41618 (local address 127.0.222.3:33909)
0809 12:15:12.959443 (+ 182us) server_negotiation.cc:206] Beginning negotiation
0809 12:15:12.959448 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0809 12:15:12.959468 (+ 20us) server_negotiation.cc:407] Connection header received
0809 12:15:12.959636 (+ 168us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 12:15:12.959641 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 12:15:12.959710 (+ 69us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 12:15:12.959798 (+ 88us) server_negotiation.cc:226] Negotiated authn=SASL
0809 12:15:12.961200 (+ 1402us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 12:15:12.961902 (+ 702us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 12:15:12.963371 (+ 1469us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 12:15:12.963784 (+ 413us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 12:15:12.963886 (+ 102us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 12:15:12.969044 (+ 5158us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 12:15:12.969060 (+ 16us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 12:15:12.969077 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0809 12:15:12.969117 (+ 40us) server_negotiation.cc:922] calling sasl_server_start()
0809 12:15:12.971936 (+ 2819us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 12:15:12.972575 (+ 639us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 12:15:12.972583 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 12:15:12.972585 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0809 12:15:12.972720 (+ 135us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 12:15:12.973054 (+ 334us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 12:15:12.973057 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 12:15:12.973059 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0809 12:15:12.973407 (+ 348us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 12:15:12.973630 (+ 223us) server_negotiation.cc:1035] Waiting for connection context
0809 12:15:12.973899 (+ 269us) server_negotiation.cc:299] Negotiation successful
0809 12:15:12.974070 (+ 171us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":58}
I20260809 12:15:12.975175 22225 generic_service.cc:196] Checking for leaks (request via RPC)
I20260809 12:15:13.067685 888 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskgH5Pjm/build/asan/bin/kudu with pid 22088
I20260809 12:15:13.087639 888 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskgH5Pjm/build/asan/bin/kudu pid 21694
I20260809 12:15:13.088790 21764 generic_service.cc:196] Checking for leaks (request via RPC)
I20260809 12:15:13.202621 888 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskgH5Pjm/build/asan/bin/kudu with pid 21694
2026-08-09T12:15:13Z chronyd exiting
[ OK ] EncryptionPolicyTest.TestEncryptionPolicy/4 (4674 ms)
[ RUN ] EncryptionPolicyTest.TestEncryptionPolicy/5
2026-08-09T12:15:13Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-09T12:15:13Z Disabled control of system clock
I20260809 12:15:13.303761 888 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskgH5Pjm/build/asan/bin/kudu
/tmp/dist-test-taskgH5Pjm/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskgH5Pjm/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786277382073938-888-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskgH5Pjm/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786277382073938-888-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskgH5Pjm/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786277382073938-888-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskgH5Pjm/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786277382073938-888-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.222.62:33597
--webserver_interface=127.0.222.62
--webserver_port=0
--builtin_ntp_servers=127.0.222.20:40477
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.222.62:33597
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20260809 12:15:13.654600 22287 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260809 12:15:13.654928 22287 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260809 12:15:13.655037 22287 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260809 12:15:13.662274 22287 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260809 12:15:13.662403 22287 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260809 12:15:13.662448 22287 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260809 12:15:13.662501 22287 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260809 12:15:13.662549 22287 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260809 12:15:13.671455 22287 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.222.20:40477
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskgH5Pjm/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786277382073938-888-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskgH5Pjm/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786277382073938-888-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.222.62:33597
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.222.62:33597
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskgH5Pjm/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786277382073938-888-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.222.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskgH5Pjm/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786277382073938-888-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 09 Aug 2026 11:06:41 UTC on ccycloud.quasar-bwoogy.root.comops.site
build id 4575991
ASAN enabled
I20260809 12:15:13.672721 22287 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260809 12:15:13.674436 22287 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260809 12:15:13.686476 22297 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 12:15:13.686594 22295 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 12:15:13.686594 22294 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260809 12:15:13.687669 22287 server_base.cc:1034] running on GCE node
I20260809 12:15:13.688522 22287 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260809 12:15:13.690022 22287 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260809 12:15:13.691270 22287 hybrid_clock.cc:648] HybridClock initialized: now 1786277713691204 us; error 55 us; skew 500 ppm
I20260809 12:15:13.691635 22287 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260809 12:15:13.697585 22287 webserver.cc:460] Webserver started at http://127.0.222.62:35553/ using document root <none> and password file <none>
I20260809 12:15:13.698340 22287 fs_manager.cc:359] Metadata directory not provided
I20260809 12:15:13.698484 22287 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260809 12:15:13.698832 22287 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260809 12:15:13.701912 22287 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskgH5Pjm/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786277382073938-888-0/minicluster-data/master-0/data/instance:
uuid: "8583eb54d4754b7eb7dd5adff1560715"
format_stamp: "Formatted at 2026-08-09 12:15:13 on dist-test-slave-2w3w"
server_key: "fb2bc7c4eb6a10061e24a26484246ee6"
server_key_iv: "5af4b3c8b3a40a984f4e8a51811a377f"
server_key_version: "encryptionkey@0"
I20260809 12:15:13.702826 22287 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskgH5Pjm/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786277382073938-888-0/minicluster-data/master-0/wal/instance:
uuid: "8583eb54d4754b7eb7dd5adff1560715"
format_stamp: "Formatted at 2026-08-09 12:15:13 on dist-test-slave-2w3w"
server_key: "fb2bc7c4eb6a10061e24a26484246ee6"
server_key_iv: "5af4b3c8b3a40a984f4e8a51811a377f"
server_key_version: "encryptionkey@0"
I20260809 12:15:13.708144 22287 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.003s sys 0.001s
I20260809 12:15:13.711611 22303 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260809 12:15:13.712971 22287 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260809 12:15:13.713171 22287 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskgH5Pjm/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786277382073938-888-0/minicluster-data/master-0/data,/tmp/dist-test-taskgH5Pjm/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786277382073938-888-0/minicluster-data/master-0/wal
uuid: "8583eb54d4754b7eb7dd5adff1560715"
format_stamp: "Formatted at 2026-08-09 12:15:13 on dist-test-slave-2w3w"
server_key: "fb2bc7c4eb6a10061e24a26484246ee6"
server_key_iv: "5af4b3c8b3a40a984f4e8a51811a377f"
server_key_version: "encryptionkey@0"
I20260809 12:15:13.713404 22287 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskgH5Pjm/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786277382073938-888-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskgH5Pjm/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786277382073938-888-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskgH5Pjm/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786277382073938-888-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260809 12:15:13.724903 22287 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260809 12:15:13.725709 22287 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260809 12:15:13.726020 22287 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260809 12:15:13.745591 22287 rpc_server.cc:307] RPC server started. Bound to: 127.0.222.62:33597
I20260809 12:15:13.745625 22358 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.222.62:33597 every 8 connection(s)
I20260809 12:15:13.746935 22287 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskgH5Pjm/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786277382073938-888-0/minicluster-data/master-0/data/info.pb
I20260809 12:15:13.750608 22359 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260809 12:15:13.755355 888 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskgH5Pjm/build/asan/bin/kudu as pid 22287
I20260809 12:15:13.755545 888 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskgH5Pjm/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786277382073938-888-0/minicluster-data/master-0/wal/instance
I20260809 12:15:13.756170 888 external_mini_cluster.cc:1428] Setting key d101edeec1403a2c340e884eae0e44cc
I20260809 12:15:13.762205 22359 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 8583eb54d4754b7eb7dd5adff1560715: Bootstrap starting.
I20260809 12:15:13.765897 22359 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 8583eb54d4754b7eb7dd5adff1560715: Neither blocks nor log segments found. Creating new log.
I20260809 12:15:13.767233 22359 log.cc:824] T 00000000000000000000000000000000 P 8583eb54d4754b7eb7dd5adff1560715: Log is configured to *not* fsync() on all Append() calls
I20260809 12:15:13.770416 22359 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 8583eb54d4754b7eb7dd5adff1560715: No bootstrap required, opened a new log
I20260809 12:15:13.770884 22362 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 12:15:13.758098 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37672 (local address 127.0.222.62:33597)
0809 12:15:13.759220 (+ 1122us) server_negotiation.cc:206] Beginning negotiation
0809 12:15:13.759230 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0809 12:15:13.759274 (+ 44us) server_negotiation.cc:407] Connection header received
0809 12:15:13.761030 (+ 1756us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 12:15:13.761051 (+ 21us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 12:15:13.761514 (+ 463us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 12:15:13.761820 (+ 306us) server_negotiation.cc:226] Negotiated authn=SASL
0809 12:15:13.762528 (+ 708us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 12:15:13.764575 (+ 2047us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 12:15:13.765941 (+ 1366us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 12:15:13.767084 (+ 1143us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 12:15:13.767277 (+ 193us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 12:15:13.768403 (+ 1126us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 12:15:13.768414 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 12:15:13.768433 (+ 19us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0809 12:15:13.769077 (+ 644us) server_negotiation.cc:922] calling sasl_server_start()
0809 12:15:13.769164 (+ 87us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0809 12:15:13.769186 (+ 22us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 12:15:13.769274 (+ 88us) server_negotiation.cc:1035] Waiting for connection context
0809 12:15:13.769559 (+ 285us) server_negotiation.cc:299] Negotiation successful
0809 12:15:13.769879 (+ 320us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":665,"thread_start_us":417,"threads_started":1}
I20260809 12:15:13.778666 22359 raft_consensus.cc:348] T 00000000000000000000000000000000 P 8583eb54d4754b7eb7dd5adff1560715 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8583eb54d4754b7eb7dd5adff1560715" member_type: VOTER last_known_addr { host: "127.0.222.62" port: 33597 } }
I20260809 12:15:13.779057 22359 raft_consensus.cc:374] T 00000000000000000000000000000000 P 8583eb54d4754b7eb7dd5adff1560715 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260809 12:15:13.779172 22359 raft_consensus.cc:729] T 00000000000000000000000000000000 P 8583eb54d4754b7eb7dd5adff1560715 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8583eb54d4754b7eb7dd5adff1560715, State: Initialized, Role: FOLLOWER
I20260809 12:15:13.779762 22359 consensus_queue.cc:260] T 00000000000000000000000000000000 P 8583eb54d4754b7eb7dd5adff1560715 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8583eb54d4754b7eb7dd5adff1560715" member_type: VOTER last_known_addr { host: "127.0.222.62" port: 33597 } }
I20260809 12:15:13.780030 22359 raft_consensus.cc:388] T 00000000000000000000000000000000 P 8583eb54d4754b7eb7dd5adff1560715 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260809 12:15:13.780169 22359 raft_consensus.cc:482] T 00000000000000000000000000000000 P 8583eb54d4754b7eb7dd5adff1560715 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260809 12:15:13.780361 22359 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 8583eb54d4754b7eb7dd5adff1560715 [term 0 FOLLOWER]: Advancing to term 1
I20260809 12:15:13.782610 22359 raft_consensus.cc:504] T 00000000000000000000000000000000 P 8583eb54d4754b7eb7dd5adff1560715 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8583eb54d4754b7eb7dd5adff1560715" member_type: VOTER last_known_addr { host: "127.0.222.62" port: 33597 } }
I20260809 12:15:13.783231 22359 leader_election.cc:302] T 00000000000000000000000000000000 P 8583eb54d4754b7eb7dd5adff1560715 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 8583eb54d4754b7eb7dd5adff1560715; no voters:
I20260809 12:15:13.783820 22359 leader_election.cc:288] T 00000000000000000000000000000000 P 8583eb54d4754b7eb7dd5adff1560715 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260809 12:15:13.783955 22364 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 8583eb54d4754b7eb7dd5adff1560715 [term 1 FOLLOWER]: Leader election won for term 1
I20260809 12:15:13.784700 22364 raft_consensus.cc:686] T 00000000000000000000000000000000 P 8583eb54d4754b7eb7dd5adff1560715 [term 1 LEADER]: Becoming Leader. State: Replica: 8583eb54d4754b7eb7dd5adff1560715, State: Running, Role: LEADER
I20260809 12:15:13.785262 22364 consensus_queue.cc:237] T 00000000000000000000000000000000 P 8583eb54d4754b7eb7dd5adff1560715 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8583eb54d4754b7eb7dd5adff1560715" member_type: VOTER last_known_addr { host: "127.0.222.62" port: 33597 } }
I20260809 12:15:13.785954 22359 sys_catalog.cc:564] T 00000000000000000000000000000000 P 8583eb54d4754b7eb7dd5adff1560715 [sys.catalog]: configured and running, proceeding with master startup.
I20260809 12:15:13.788326 22367 sys_catalog.cc:455] T 00000000000000000000000000000000 P 8583eb54d4754b7eb7dd5adff1560715 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 8583eb54d4754b7eb7dd5adff1560715. Latest consensus state: current_term: 1 leader_uuid: "8583eb54d4754b7eb7dd5adff1560715" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8583eb54d4754b7eb7dd5adff1560715" member_type: VOTER last_known_addr { host: "127.0.222.62" port: 33597 } } }
I20260809 12:15:13.788627 22365 sys_catalog.cc:455] T 00000000000000000000000000000000 P 8583eb54d4754b7eb7dd5adff1560715 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "8583eb54d4754b7eb7dd5adff1560715" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8583eb54d4754b7eb7dd5adff1560715" member_type: VOTER last_known_addr { host: "127.0.222.62" port: 33597 } } }
I20260809 12:15:13.788842 22367 sys_catalog.cc:458] T 00000000000000000000000000000000 P 8583eb54d4754b7eb7dd5adff1560715 [sys.catalog]: This master's current role is: LEADER
I20260809 12:15:13.788908 22365 sys_catalog.cc:458] T 00000000000000000000000000000000 P 8583eb54d4754b7eb7dd5adff1560715 [sys.catalog]: This master's current role is: LEADER
I20260809 12:15:13.790596 22375 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260809 12:15:13.796129 22375 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260809 12:15:13.805501 22375 catalog_manager.cc:1347] Generated new cluster ID: 3215689e6c364c869752095dbf06f978
I20260809 12:15:13.805630 22375 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260809 12:15:13.815407 22375 catalog_manager.cc:1370] Generated new certificate authority record
I20260809 12:15:13.816673 22375 catalog_manager.cc:1504] Loading token signing keys...
I20260809 12:15:13.823216 22375 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 8583eb54d4754b7eb7dd5adff1560715: Generated new TSK 0
I20260809 12:15:13.824301 22375 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260809 12:15:13.841670 888 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskgH5Pjm/build/asan/bin/kudu
/tmp/dist-test-taskgH5Pjm/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskgH5Pjm/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786277382073938-888-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskgH5Pjm/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786277382073938-888-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskgH5Pjm/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786277382073938-888-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskgH5Pjm/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786277382073938-888-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.222.1:0
--local_ip_for_outbound_sockets=127.0.222.1
--webserver_interface=127.0.222.1
--webserver_port=0
--tserver_master_addrs=127.0.222.62:33597
--builtin_ntp_servers=127.0.222.20:40477
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20260809 12:15:14.188431 22385 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260809 12:15:14.188705 22385 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260809 12:15:14.188884 22385 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260809 12:15:14.196451 22385 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260809 12:15:14.196574 22385 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260809 12:15:14.196754 22385 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.222.1
I20260809 12:15:14.206167 22385 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.222.20:40477
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskgH5Pjm/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786277382073938-888-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskgH5Pjm/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786277382073938-888-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.222.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskgH5Pjm/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786277382073938-888-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.222.1
--webserver_port=0
--tserver_master_addrs=127.0.222.62:33597
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.222.1
--log_dir=/tmp/dist-test-taskgH5Pjm/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786277382073938-888-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 09 Aug 2026 11:06:41 UTC on ccycloud.quasar-bwoogy.root.comops.site
build id 4575991
ASAN enabled
I20260809 12:15:14.207559 22385 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260809 12:15:14.209439 22385 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260809 12:15:14.223369 22393 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 12:15:14.223284 22396 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 12:15:14.223265 22394 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260809 12:15:14.224444 22385 server_base.cc:1034] running on GCE node
I20260809 12:15:14.225088 22385 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260809 12:15:14.226038 22385 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260809 12:15:14.227275 22385 hybrid_clock.cc:648] HybridClock initialized: now 1786277714227231 us; error 56 us; skew 500 ppm
I20260809 12:15:14.227614 22385 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260809 12:15:14.230113 22385 webserver.cc:460] Webserver started at http://127.0.222.1:35473/ using document root <none> and password file <none>
I20260809 12:15:14.230854 22385 fs_manager.cc:359] Metadata directory not provided
I20260809 12:15:14.231027 22385 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260809 12:15:14.231383 22385 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260809 12:15:14.234299 22385 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskgH5Pjm/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786277382073938-888-0/minicluster-data/ts-0/data/instance:
uuid: "1fcdf59a28b54be7a8f7c793b2c475ff"
format_stamp: "Formatted at 2026-08-09 12:15:14 on dist-test-slave-2w3w"
server_key: "74be8b3cb87e677e2daa942be57b2ee0"
server_key_iv: "3ff9eecfdce8c2d22c2cac2adacb104b"
server_key_version: "encryptionkey@0"
I20260809 12:15:14.235203 22385 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskgH5Pjm/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786277382073938-888-0/minicluster-data/ts-0/wal/instance:
uuid: "1fcdf59a28b54be7a8f7c793b2c475ff"
format_stamp: "Formatted at 2026-08-09 12:15:14 on dist-test-slave-2w3w"
server_key: "74be8b3cb87e677e2daa942be57b2ee0"
server_key_iv: "3ff9eecfdce8c2d22c2cac2adacb104b"
server_key_version: "encryptionkey@0"
I20260809 12:15:14.240399 22385 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.004s sys 0.000s
I20260809 12:15:14.243858 22403 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260809 12:15:14.245191 22385 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260809 12:15:14.245376 22385 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskgH5Pjm/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786277382073938-888-0/minicluster-data/ts-0/data,/tmp/dist-test-taskgH5Pjm/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786277382073938-888-0/minicluster-data/ts-0/wal
uuid: "1fcdf59a28b54be7a8f7c793b2c475ff"
format_stamp: "Formatted at 2026-08-09 12:15:14 on dist-test-slave-2w3w"
server_key: "74be8b3cb87e677e2daa942be57b2ee0"
server_key_iv: "3ff9eecfdce8c2d22c2cac2adacb104b"
server_key_version: "encryptionkey@0"
I20260809 12:15:14.245602 22385 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskgH5Pjm/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786277382073938-888-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskgH5Pjm/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786277382073938-888-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskgH5Pjm/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786277382073938-888-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260809 12:15:14.281327 22385 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260809 12:15:14.282241 22385 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260809 12:15:14.282541 22385 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260809 12:15:14.283641 22385 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260809 12:15:14.285149 22385 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260809 12:15:14.285280 22385 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260809 12:15:14.285409 22385 ts_tablet_manager.cc:610] Registered 0 tablets
I20260809 12:15:14.285501 22385 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260809 12:15:14.319944 22385 rpc_server.cc:307] RPC server started. Bound to: 127.0.222.1:40663
I20260809 12:15:14.320036 22525 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.222.1:40663 every 8 connection(s)
I20260809 12:15:14.321338 22385 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskgH5Pjm/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786277382073938-888-0/minicluster-data/ts-0/data/info.pb
I20260809 12:15:14.326797 888 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskgH5Pjm/build/asan/bin/kudu as pid 22385
I20260809 12:15:14.327013 888 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskgH5Pjm/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786277382073938-888-0/minicluster-data/ts-0/wal/instance
I20260809 12:15:14.327594 888 external_mini_cluster.cc:1428] Setting key 5e94a11692544d540780be01cf5104ca
I20260809 12:15:14.330700 888 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskgH5Pjm/build/asan/bin/kudu
/tmp/dist-test-taskgH5Pjm/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskgH5Pjm/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786277382073938-888-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskgH5Pjm/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786277382073938-888-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskgH5Pjm/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786277382073938-888-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskgH5Pjm/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786277382073938-888-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.222.2:0
--local_ip_for_outbound_sockets=127.0.222.2
--webserver_interface=127.0.222.2
--webserver_port=0
--tserver_master_addrs=127.0.222.62:33597
--builtin_ntp_servers=127.0.222.20:40477
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
I20260809 12:15:14.335831 22529 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 12:15:14.324108 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.222.1:46805 (local address 127.0.222.62:33597)
0809 12:15:14.324554 (+ 446us) server_negotiation.cc:206] Beginning negotiation
0809 12:15:14.324559 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0809 12:15:14.325615 (+ 1056us) server_negotiation.cc:407] Connection header received
0809 12:15:14.328071 (+ 2456us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 12:15:14.328075 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 12:15:14.328158 (+ 83us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 12:15:14.328292 (+ 134us) server_negotiation.cc:226] Negotiated authn=SASL
0809 12:15:14.329520 (+ 1228us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 12:15:14.330295 (+ 775us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 12:15:14.332871 (+ 2576us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 12:15:14.333275 (+ 404us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 12:15:14.333379 (+ 104us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 12:15:14.335073 (+ 1694us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 12:15:14.335078 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 12:15:14.335081 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0809 12:15:14.335100 (+ 19us) server_negotiation.cc:922] calling sasl_server_start()
0809 12:15:14.335140 (+ 40us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0809 12:15:14.335151 (+ 11us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 12:15:14.335220 (+ 69us) server_negotiation.cc:1035] Waiting for connection context
0809 12:15:14.335480 (+ 260us) server_negotiation.cc:299] Negotiation successful
0809 12:15:14.335559 (+ 79us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":347,"thread_start_us":252,"threads_started":1}
I20260809 12:15:14.337333 22530 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 12:15:14.324556 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.222.62:33597 (local address 127.0.222.1:46805)
0809 12:15:14.325440 (+ 884us) negotiation.cc:107] Waiting for socket to connect
0809 12:15:14.325497 (+ 57us) client_negotiation.cc:174] Beginning negotiation
0809 12:15:14.327772 (+ 2275us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 12:15:14.328513 (+ 741us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 12:15:14.328526 (+ 13us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 12:15:14.328642 (+ 116us) client_negotiation.cc:189] Negotiated authn=SASL
0809 12:15:14.329332 (+ 690us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 12:15:14.329358 (+ 26us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 12:15:14.330470 (+ 1112us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 12:15:14.330479 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 12:15:14.332703 (+ 2224us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 12:15:14.332722 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 12:15:14.333521 (+ 799us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 12:15:14.333531 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 12:15:14.333860 (+ 329us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 12:15:14.334640 (+ 780us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0809 12:15:14.334646 (+ 6us) client_negotiation.cc:656] calling sasl_client_start()
0809 12:15:14.334699 (+ 53us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0809 12:15:14.334820 (+ 121us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0809 12:15:14.335251 (+ 431us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0809 12:15:14.335265 (+ 14us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0809 12:15:14.335275 (+ 10us) client_negotiation.cc:769] Sending connection context
0809 12:15:14.335465 (+ 190us) client_negotiation.cc:240] Negotiation successful
0809 12:15:14.335894 (+ 429us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":459,"thread_start_us":272,"threads_started":1}
I20260809 12:15:14.338912 22526 heartbeater.cc:347] Connected to a master server at 127.0.222.62:33597
I20260809 12:15:14.339254 22526 heartbeater.cc:464] Registering TS with master...
I20260809 12:15:14.340253 22526 heartbeater.cc:511] Master 127.0.222.62:33597 requested a full tablet report, sending...
I20260809 12:15:14.342849 22321 ts_manager.cc:194] Registered new tserver with Master: 1fcdf59a28b54be7a8f7c793b2c475ff (127.0.222.1:40663)
I20260809 12:15:14.345186 22321 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.222.1:46805
W20260809 12:15:14.671916 22531 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260809 12:15:14.672209 22531 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260809 12:15:14.672369 22531 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260809 12:15:14.679546 22531 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260809 12:15:14.679639 22531 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260809 12:15:14.679845 22531 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.222.2
I20260809 12:15:14.688969 22531 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.222.20:40477
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskgH5Pjm/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786277382073938-888-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskgH5Pjm/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786277382073938-888-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.222.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskgH5Pjm/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786277382073938-888-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.222.2
--webserver_port=0
--tserver_master_addrs=127.0.222.62:33597
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.222.2
--log_dir=/tmp/dist-test-taskgH5Pjm/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786277382073938-888-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 09 Aug 2026 11:06:41 UTC on ccycloud.quasar-bwoogy.root.comops.site
build id 4575991
ASAN enabled
I20260809 12:15:14.690311 22531 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260809 12:15:14.692127 22531 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260809 12:15:14.705224 22541 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 12:15:14.705324 22537 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 12:15:14.705245 22539 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260809 12:15:14.706099 22531 server_base.cc:1034] running on GCE node
I20260809 12:15:14.706653 22531 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260809 12:15:14.707682 22531 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260809 12:15:14.708925 22531 hybrid_clock.cc:648] HybridClock initialized: now 1786277714708889 us; error 61 us; skew 500 ppm
I20260809 12:15:14.709260 22531 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260809 12:15:14.711669 22531 webserver.cc:460] Webserver started at http://127.0.222.2:44421/ using document root <none> and password file <none>
I20260809 12:15:14.712383 22531 fs_manager.cc:359] Metadata directory not provided
I20260809 12:15:14.712527 22531 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260809 12:15:14.712872 22531 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260809 12:15:14.715940 22531 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskgH5Pjm/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786277382073938-888-0/minicluster-data/ts-1/data/instance:
uuid: "f6d49565647a486eb53ccf7685f9321a"
format_stamp: "Formatted at 2026-08-09 12:15:14 on dist-test-slave-2w3w"
server_key: "e6af857f8e50f85cd0fce1b9fcb1b694"
server_key_iv: "3c137a6e180262201ff2153c63b093c0"
server_key_version: "encryptionkey@0"
I20260809 12:15:14.716773 22531 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskgH5Pjm/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786277382073938-888-0/minicluster-data/ts-1/wal/instance:
uuid: "f6d49565647a486eb53ccf7685f9321a"
format_stamp: "Formatted at 2026-08-09 12:15:14 on dist-test-slave-2w3w"
server_key: "e6af857f8e50f85cd0fce1b9fcb1b694"
server_key_iv: "3c137a6e180262201ff2153c63b093c0"
server_key_version: "encryptionkey@0"
I20260809 12:15:14.722359 22531 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.004s sys 0.002s
I20260809 12:15:14.726020 22548 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260809 12:15:14.727416 22531 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260809 12:15:14.727588 22531 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskgH5Pjm/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786277382073938-888-0/minicluster-data/ts-1/data,/tmp/dist-test-taskgH5Pjm/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786277382073938-888-0/minicluster-data/ts-1/wal
uuid: "f6d49565647a486eb53ccf7685f9321a"
format_stamp: "Formatted at 2026-08-09 12:15:14 on dist-test-slave-2w3w"
server_key: "e6af857f8e50f85cd0fce1b9fcb1b694"
server_key_iv: "3c137a6e180262201ff2153c63b093c0"
server_key_version: "encryptionkey@0"
I20260809 12:15:14.727819 22531 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskgH5Pjm/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786277382073938-888-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskgH5Pjm/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786277382073938-888-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskgH5Pjm/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786277382073938-888-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260809 12:15:14.739133 22531 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260809 12:15:14.739894 22531 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260809 12:15:14.740181 22531 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260809 12:15:14.741160 22531 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260809 12:15:14.742563 22531 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260809 12:15:14.742700 22531 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260809 12:15:14.742841 22531 ts_tablet_manager.cc:610] Registered 0 tablets
I20260809 12:15:14.742933 22531 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260809 12:15:14.778590 22531 rpc_server.cc:307] RPC server started. Bound to: 127.0.222.2:39473
I20260809 12:15:14.778695 22668 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.222.2:39473 every 8 connection(s)
I20260809 12:15:14.779959 22531 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskgH5Pjm/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786277382073938-888-0/minicluster-data/ts-1/data/info.pb
I20260809 12:15:14.786279 888 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskgH5Pjm/build/asan/bin/kudu as pid 22531
I20260809 12:15:14.786482 888 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskgH5Pjm/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786277382073938-888-0/minicluster-data/ts-1/wal/instance
I20260809 12:15:14.787045 888 external_mini_cluster.cc:1428] Setting key cc85af55a47ad276fad6cb93d69b9cbe
I20260809 12:15:14.790493 888 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskgH5Pjm/build/asan/bin/kudu
/tmp/dist-test-taskgH5Pjm/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskgH5Pjm/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786277382073938-888-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskgH5Pjm/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786277382073938-888-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskgH5Pjm/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786277382073938-888-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskgH5Pjm/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786277382073938-888-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.222.3:0
--local_ip_for_outbound_sockets=127.0.222.3
--webserver_interface=127.0.222.3
--webserver_port=0
--tserver_master_addrs=127.0.222.62:33597
--builtin_ntp_servers=127.0.222.20:40477
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
I20260809 12:15:14.794315 22529 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 12:15:14.782511 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.222.2:48551 (local address 127.0.222.62:33597)
0809 12:15:14.782697 (+ 186us) server_negotiation.cc:206] Beginning negotiation
0809 12:15:14.782702 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0809 12:15:14.784074 (+ 1372us) server_negotiation.cc:407] Connection header received
0809 12:15:14.785784 (+ 1710us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 12:15:14.785789 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 12:15:14.785865 (+ 76us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 12:15:14.785953 (+ 88us) server_negotiation.cc:226] Negotiated authn=SASL
0809 12:15:14.787564 (+ 1611us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 12:15:14.788474 (+ 910us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 12:15:14.790790 (+ 2316us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 12:15:14.791596 (+ 806us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 12:15:14.791715 (+ 119us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 12:15:14.793605 (+ 1890us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 12:15:14.793610 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 12:15:14.793614 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0809 12:15:14.793631 (+ 17us) server_negotiation.cc:922] calling sasl_server_start()
0809 12:15:14.793664 (+ 33us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0809 12:15:14.793675 (+ 11us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 12:15:14.793760 (+ 85us) server_negotiation.cc:1035] Waiting for connection context
0809 12:15:14.794004 (+ 244us) server_negotiation.cc:299] Negotiation successful
0809 12:15:14.794084 (+ 80us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":57}
I20260809 12:15:14.795912 22671 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 12:15:14.783013 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.222.62:33597 (local address 127.0.222.2:48551)
0809 12:15:14.783923 (+ 910us) negotiation.cc:107] Waiting for socket to connect
0809 12:15:14.783959 (+ 36us) client_negotiation.cc:174] Beginning negotiation
0809 12:15:14.785531 (+ 1572us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 12:15:14.786239 (+ 708us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 12:15:14.786251 (+ 12us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 12:15:14.786389 (+ 138us) client_negotiation.cc:189] Negotiated authn=SASL
0809 12:15:14.787340 (+ 951us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 12:15:14.787368 (+ 28us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 12:15:14.788621 (+ 1253us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 12:15:14.788625 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 12:15:14.790612 (+ 1987us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 12:15:14.790628 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 12:15:14.792051 (+ 1423us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 12:15:14.792060 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 12:15:14.792437 (+ 377us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 12:15:14.793220 (+ 783us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0809 12:15:14.793225 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0809 12:15:14.793285 (+ 60us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0809 12:15:14.793399 (+ 114us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0809 12:15:14.793780 (+ 381us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0809 12:15:14.793793 (+ 13us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0809 12:15:14.793803 (+ 10us) client_negotiation.cc:769] Sending connection context
0809 12:15:14.793997 (+ 194us) client_negotiation.cc:240] Negotiation successful
0809 12:15:14.794431 (+ 434us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":486,"thread_start_us":289,"threads_started":1}
I20260809 12:15:14.797427 22669 heartbeater.cc:347] Connected to a master server at 127.0.222.62:33597
I20260809 12:15:14.797747 22669 heartbeater.cc:464] Registering TS with master...
I20260809 12:15:14.798674 22669 heartbeater.cc:511] Master 127.0.222.62:33597 requested a full tablet report, sending...
I20260809 12:15:14.800319 22321 ts_manager.cc:194] Registered new tserver with Master: f6d49565647a486eb53ccf7685f9321a (127.0.222.2:39473)
I20260809 12:15:14.801154 22321 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.222.2:48551
W20260809 12:15:15.139432 22672 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260809 12:15:15.139776 22672 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260809 12:15:15.139962 22672 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260809 12:15:15.147302 22672 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260809 12:15:15.147400 22672 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260809 12:15:15.147624 22672 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.222.3
I20260809 12:15:15.156935 22672 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.222.20:40477
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskgH5Pjm/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786277382073938-888-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskgH5Pjm/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786277382073938-888-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.222.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskgH5Pjm/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786277382073938-888-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.222.3
--webserver_port=0
--tserver_master_addrs=127.0.222.62:33597
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.222.3
--log_dir=/tmp/dist-test-taskgH5Pjm/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786277382073938-888-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 09 Aug 2026 11:06:41 UTC on ccycloud.quasar-bwoogy.root.comops.site
build id 4575991
ASAN enabled
I20260809 12:15:15.158229 22672 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260809 12:15:15.160089 22672 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260809 12:15:15.173264 22682 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 12:15:15.173431 22678 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 12:15:15.173496 22679 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260809 12:15:15.174293 22672 server_base.cc:1034] running on GCE node
I20260809 12:15:15.174858 22672 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260809 12:15:15.175827 22672 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260809 12:15:15.177031 22672 hybrid_clock.cc:648] HybridClock initialized: now 1786277715176982 us; error 41 us; skew 500 ppm
I20260809 12:15:15.177348 22672 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260809 12:15:15.179726 22672 webserver.cc:460] Webserver started at http://127.0.222.3:45049/ using document root <none> and password file <none>
I20260809 12:15:15.180426 22672 fs_manager.cc:359] Metadata directory not provided
I20260809 12:15:15.180527 22672 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260809 12:15:15.180902 22672 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260809 12:15:15.183825 22672 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskgH5Pjm/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786277382073938-888-0/minicluster-data/ts-2/data/instance:
uuid: "b8128aa81c32435eb3bda70dd5ecbd3d"
format_stamp: "Formatted at 2026-08-09 12:15:15 on dist-test-slave-2w3w"
server_key: "9f89d3f624fd96a423c5d8c6973d2280"
server_key_iv: "3961dcd274f163710598b2e49dcad2b9"
server_key_version: "encryptionkey@0"
I20260809 12:15:15.184643 22672 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskgH5Pjm/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786277382073938-888-0/minicluster-data/ts-2/wal/instance:
uuid: "b8128aa81c32435eb3bda70dd5ecbd3d"
format_stamp: "Formatted at 2026-08-09 12:15:15 on dist-test-slave-2w3w"
server_key: "9f89d3f624fd96a423c5d8c6973d2280"
server_key_iv: "3961dcd274f163710598b2e49dcad2b9"
server_key_version: "encryptionkey@0"
I20260809 12:15:15.190357 22672 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.001s
I20260809 12:15:15.194056 22690 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260809 12:15:15.195585 22672 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260809 12:15:15.195781 22672 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskgH5Pjm/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786277382073938-888-0/minicluster-data/ts-2/data,/tmp/dist-test-taskgH5Pjm/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786277382073938-888-0/minicluster-data/ts-2/wal
uuid: "b8128aa81c32435eb3bda70dd5ecbd3d"
format_stamp: "Formatted at 2026-08-09 12:15:15 on dist-test-slave-2w3w"
server_key: "9f89d3f624fd96a423c5d8c6973d2280"
server_key_iv: "3961dcd274f163710598b2e49dcad2b9"
server_key_version: "encryptionkey@0"
I20260809 12:15:15.196214 22672 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskgH5Pjm/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786277382073938-888-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskgH5Pjm/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786277382073938-888-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskgH5Pjm/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786277382073938-888-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260809 12:15:15.224997 22672 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260809 12:15:15.225926 22672 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260809 12:15:15.226226 22672 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260809 12:15:15.227370 22672 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260809 12:15:15.228937 22672 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260809 12:15:15.229066 22672 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260809 12:15:15.229218 22672 ts_tablet_manager.cc:610] Registered 0 tablets
I20260809 12:15:15.229311 22672 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260809 12:15:15.264528 22672 rpc_server.cc:307] RPC server started. Bound to: 127.0.222.3:33683
I20260809 12:15:15.264619 22810 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.222.3:33683 every 8 connection(s)
I20260809 12:15:15.265858 22672 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskgH5Pjm/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786277382073938-888-0/minicluster-data/ts-2/data/info.pb
I20260809 12:15:15.275444 888 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskgH5Pjm/build/asan/bin/kudu as pid 22672
I20260809 12:15:15.275632 888 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskgH5Pjm/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786277382073938-888-0/minicluster-data/ts-2/wal/instance
I20260809 12:15:15.276134 888 external_mini_cluster.cc:1428] Setting key b5a3f9dc0ed7bc8e09eff2ecbd1708aa
I20260809 12:15:15.278739 22529 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 12:15:15.268555 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.222.3:33381 (local address 127.0.222.62:33597)
0809 12:15:15.268742 (+ 187us) server_negotiation.cc:206] Beginning negotiation
0809 12:15:15.268748 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0809 12:15:15.270115 (+ 1367us) server_negotiation.cc:407] Connection header received
0809 12:15:15.271992 (+ 1877us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 12:15:15.271996 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 12:15:15.272080 (+ 84us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 12:15:15.272169 (+ 89us) server_negotiation.cc:226] Negotiated authn=SASL
0809 12:15:15.273187 (+ 1018us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 12:15:15.273964 (+ 777us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 12:15:15.275843 (+ 1879us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 12:15:15.276306 (+ 463us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 12:15:15.276410 (+ 104us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 12:15:15.277792 (+ 1382us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 12:15:15.277797 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 12:15:15.277800 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0809 12:15:15.277840 (+ 40us) server_negotiation.cc:922] calling sasl_server_start()
0809 12:15:15.277880 (+ 40us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0809 12:15:15.277897 (+ 17us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 12:15:15.278009 (+ 112us) server_negotiation.cc:1035] Waiting for connection context
0809 12:15:15.278260 (+ 251us) server_negotiation.cc:299] Negotiation successful
0809 12:15:15.278371 (+ 111us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":71}
I20260809 12:15:15.279712 22813 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 12:15:15.269043 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.222.62:33597 (local address 127.0.222.3:33381)
0809 12:15:15.269948 (+ 905us) negotiation.cc:107] Waiting for socket to connect
0809 12:15:15.269994 (+ 46us) client_negotiation.cc:174] Beginning negotiation
0809 12:15:15.271730 (+ 1736us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 12:15:15.272370 (+ 640us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 12:15:15.272379 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 12:15:15.272458 (+ 79us) client_negotiation.cc:189] Negotiated authn=SASL
0809 12:15:15.273023 (+ 565us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 12:15:15.273043 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 12:15:15.274099 (+ 1056us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 12:15:15.274102 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 12:15:15.275698 (+ 1596us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 12:15:15.275715 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 12:15:15.276536 (+ 821us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 12:15:15.276544 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 12:15:15.276781 (+ 237us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 12:15:15.277450 (+ 669us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0809 12:15:15.277452 (+ 2us) client_negotiation.cc:656] calling sasl_client_start()
0809 12:15:15.277493 (+ 41us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0809 12:15:15.277606 (+ 113us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0809 12:15:15.278012 (+ 406us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0809 12:15:15.278023 (+ 11us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0809 12:15:15.278030 (+ 7us) client_negotiation.cc:769] Sending connection context
0809 12:15:15.278234 (+ 204us) client_negotiation.cc:240] Negotiation successful
0809 12:15:15.278523 (+ 289us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":477,"thread_start_us":296,"threads_started":1}
I20260809 12:15:15.281080 22811 heartbeater.cc:347] Connected to a master server at 127.0.222.62:33597
I20260809 12:15:15.281371 22811 heartbeater.cc:464] Registering TS with master...
I20260809 12:15:15.282089 22811 heartbeater.cc:511] Master 127.0.222.62:33597 requested a full tablet report, sending...
I20260809 12:15:15.283547 22321 ts_manager.cc:194] Registered new tserver with Master: b8128aa81c32435eb3bda70dd5ecbd3d (127.0.222.3:33683)
I20260809 12:15:15.284390 22321 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.222.3:33381
I20260809 12:15:15.291652 888 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260809 12:15:15.301707 22529 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 12:15:15.295706 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37684 (local address 127.0.222.62:33597)
0809 12:15:15.295938 (+ 232us) server_negotiation.cc:206] Beginning negotiation
0809 12:15:15.295946 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0809 12:15:15.296350 (+ 404us) server_negotiation.cc:407] Connection header received
0809 12:15:15.296537 (+ 187us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 12:15:15.296541 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 12:15:15.296616 (+ 75us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 12:15:15.296699 (+ 83us) server_negotiation.cc:226] Negotiated authn=SASL
0809 12:15:15.297161 (+ 462us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 12:15:15.297851 (+ 690us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 12:15:15.299235 (+ 1384us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 12:15:15.299703 (+ 468us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 12:15:15.299807 (+ 104us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 12:15:15.300986 (+ 1179us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 12:15:15.300991 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 12:15:15.300994 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0809 12:15:15.301010 (+ 16us) server_negotiation.cc:922] calling sasl_server_start()
0809 12:15:15.301040 (+ 30us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0809 12:15:15.301051 (+ 11us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 12:15:15.301124 (+ 73us) server_negotiation.cc:1035] Waiting for connection context
0809 12:15:15.301279 (+ 155us) server_negotiation.cc:299] Negotiation successful
0809 12:15:15.301387 (+ 108us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":65}
I20260809 12:15:15.305408 22321 catalog_manager.cc:1908] Servicing CreateTable request from {username='slave'} at 127.0.0.1:37684:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260809 12:15:15.312980 22321 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260809 12:15:15.341651 22828 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 12:15:15.325803 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.222.2:39473 (local address 127.0.0.1:60408)
0809 12:15:15.332839 (+ 7036us) negotiation.cc:107] Waiting for socket to connect
0809 12:15:15.332853 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0809 12:15:15.333000 (+ 147us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 12:15:15.334089 (+ 1089us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 12:15:15.334097 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 12:15:15.334130 (+ 33us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0809 12:15:15.334522 (+ 392us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 12:15:15.334535 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 12:15:15.336660 (+ 2125us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 12:15:15.336669 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 12:15:15.338771 (+ 2102us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 12:15:15.338786 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 12:15:15.340607 (+ 1821us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 12:15:15.340614 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 12:15:15.340829 (+ 215us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 12:15:15.340894 (+ 65us) client_negotiation.cc:769] Sending connection context
0809 12:15:15.341043 (+ 149us) client_negotiation.cc:240] Negotiation successful
0809 12:15:15.341227 (+ 184us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":6892,"mutex_wait_us":1,"thread_start_us":5387,"threads_started":1}
I20260809 12:15:15.341708 22831 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 12:15:15.326418 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60408 (local address 127.0.222.2:39473)
0809 12:15:15.327208 (+ 790us) server_negotiation.cc:206] Beginning negotiation
0809 12:15:15.327221 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0809 12:15:15.332968 (+ 5747us) server_negotiation.cc:407] Connection header received
0809 12:15:15.333198 (+ 230us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 12:15:15.333210 (+ 12us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 12:15:15.333533 (+ 323us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 12:15:15.333737 (+ 204us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0809 12:15:15.334731 (+ 994us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 12:15:15.336465 (+ 1734us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 12:15:15.339037 (+ 2572us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 12:15:15.340395 (+ 1358us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 12:15:15.340537 (+ 142us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 12:15:15.340781 (+ 244us) server_negotiation.cc:1035] Waiting for connection context
0809 12:15:15.341124 (+ 343us) server_negotiation.cc:299] Negotiation successful
0809 12:15:15.341315 (+ 191us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":558,"thread_start_us":631,"threads_started":1}
I20260809 12:15:15.341794 22832 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 12:15:15.326922 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.222.1:40663 (local address 127.0.0.1:59536)
0809 12:15:15.333531 (+ 6609us) negotiation.cc:107] Waiting for socket to connect
0809 12:15:15.333549 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0809 12:15:15.333754 (+ 205us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 12:15:15.334443 (+ 689us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 12:15:15.334447 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 12:15:15.334480 (+ 33us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0809 12:15:15.334923 (+ 443us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 12:15:15.334938 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 12:15:15.336612 (+ 1674us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 12:15:15.336618 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 12:15:15.338934 (+ 2316us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 12:15:15.338949 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 12:15:15.341047 (+ 2098us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 12:15:15.341053 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 12:15:15.341206 (+ 153us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 12:15:15.341235 (+ 29us) client_negotiation.cc:769] Sending connection context
0809 12:15:15.341309 (+ 74us) client_negotiation.cc:240] Negotiation successful
0809 12:15:15.341424 (+ 115us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":6498,"mutex_wait_us":57,"thread_start_us":6341,"threads_started":1}
I20260809 12:15:15.342471 22833 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 12:15:15.327219 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59536 (local address 127.0.222.1:40663)
0809 12:15:15.327952 (+ 733us) server_negotiation.cc:206] Beginning negotiation
0809 12:15:15.327963 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0809 12:15:15.333681 (+ 5718us) server_negotiation.cc:407] Connection header received
0809 12:15:15.333973 (+ 292us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 12:15:15.333983 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 12:15:15.334226 (+ 243us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 12:15:15.334446 (+ 220us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0809 12:15:15.335175 (+ 729us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 12:15:15.336469 (+ 1294us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 12:15:15.339179 (+ 2710us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 12:15:15.340457 (+ 1278us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 12:15:15.340601 (+ 144us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 12:15:15.340869 (+ 268us) server_negotiation.cc:1035] Waiting for connection context
0809 12:15:15.341938 (+ 1069us) server_negotiation.cc:299] Negotiation successful
0809 12:15:15.342096 (+ 158us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":495,"thread_start_us":351,"threads_started":1}
I20260809 12:15:15.343092 22829 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 12:15:15.325423 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.222.3:33683 (local address 127.0.0.1:34494)
0809 12:15:15.331845 (+ 6422us) negotiation.cc:107] Waiting for socket to connect
0809 12:15:15.331900 (+ 55us) client_negotiation.cc:174] Beginning negotiation
0809 12:15:15.332139 (+ 239us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 12:15:15.335036 (+ 2897us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 12:15:15.335042 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 12:15:15.335076 (+ 34us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0809 12:15:15.335466 (+ 390us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 12:15:15.335479 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 12:15:15.337598 (+ 2119us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 12:15:15.337604 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 12:15:15.339875 (+ 2271us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 12:15:15.339891 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 12:15:15.341664 (+ 1773us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 12:15:15.341672 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 12:15:15.341890 (+ 218us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 12:15:15.341924 (+ 34us) client_negotiation.cc:769] Sending connection context
0809 12:15:15.341993 (+ 69us) client_negotiation.cc:240] Negotiation successful
0809 12:15:15.342094 (+ 101us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":5997,"mutex_wait_us":38,"thread_start_us":7126,"threads_started":1}
I20260809 12:15:15.344049 22830 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 12:15:15.325690 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34494 (local address 127.0.222.3:33683)
0809 12:15:15.334342 (+ 8652us) server_negotiation.cc:206] Beginning negotiation
0809 12:15:15.334351 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0809 12:15:15.334389 (+ 38us) server_negotiation.cc:407] Connection header received
0809 12:15:15.334501 (+ 112us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 12:15:15.334508 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 12:15:15.334719 (+ 211us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 12:15:15.335664 (+ 945us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0809 12:15:15.335951 (+ 287us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 12:15:15.337426 (+ 1475us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 12:15:15.340100 (+ 2674us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 12:15:15.341412 (+ 1312us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 12:15:15.343069 (+ 1657us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 12:15:15.343419 (+ 350us) server_negotiation.cc:1035] Waiting for connection context
0809 12:15:15.343499 (+ 80us) server_negotiation.cc:299] Negotiation successful
0809 12:15:15.343654 (+ 155us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":8477,"thread_start_us":8309,"threads_started":1}
I20260809 12:15:15.347167 22597 tablet_service.cc:1467] Processing CreateTablet for tablet fc7aa1df79e647ff834067e8daef8985 (DEFAULT_TABLE table=test-table [id=1bd6c1bac46849838d328f21113200fc]), partition=RANGE (key) PARTITION UNBOUNDED
I20260809 12:15:15.347661 22739 tablet_service.cc:1467] Processing CreateTablet for tablet fc7aa1df79e647ff834067e8daef8985 (DEFAULT_TABLE table=test-table [id=1bd6c1bac46849838d328f21113200fc]), partition=RANGE (key) PARTITION UNBOUNDED
I20260809 12:15:15.347797 22455 tablet_service.cc:1467] Processing CreateTablet for tablet fc7aa1df79e647ff834067e8daef8985 (DEFAULT_TABLE table=test-table [id=1bd6c1bac46849838d328f21113200fc]), partition=RANGE (key) PARTITION UNBOUNDED
I20260809 12:15:15.348735 22597 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet fc7aa1df79e647ff834067e8daef8985. 1 dirs total, 0 dirs full, 0 dirs failed
I20260809 12:15:15.348870 22526 heartbeater.cc:503] Master 127.0.222.62:33597 was elected leader, sending a full tablet report...
I20260809 12:15:15.349257 22455 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet fc7aa1df79e647ff834067e8daef8985. 1 dirs total, 0 dirs full, 0 dirs failed
I20260809 12:15:15.349850 22739 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet fc7aa1df79e647ff834067e8daef8985. 1 dirs total, 0 dirs full, 0 dirs failed
I20260809 12:15:15.362360 22834 tablet_bootstrap.cc:492] T fc7aa1df79e647ff834067e8daef8985 P b8128aa81c32435eb3bda70dd5ecbd3d: Bootstrap starting.
I20260809 12:15:15.363935 22835 tablet_bootstrap.cc:492] T fc7aa1df79e647ff834067e8daef8985 P f6d49565647a486eb53ccf7685f9321a: Bootstrap starting.
I20260809 12:15:15.366925 22834 tablet_bootstrap.cc:654] T fc7aa1df79e647ff834067e8daef8985 P b8128aa81c32435eb3bda70dd5ecbd3d: Neither blocks nor log segments found. Creating new log.
I20260809 12:15:15.367693 22835 tablet_bootstrap.cc:654] T fc7aa1df79e647ff834067e8daef8985 P f6d49565647a486eb53ccf7685f9321a: Neither blocks nor log segments found. Creating new log.
I20260809 12:15:15.368451 22834 log.cc:824] T fc7aa1df79e647ff834067e8daef8985 P b8128aa81c32435eb3bda70dd5ecbd3d: Log is configured to *not* fsync() on all Append() calls
I20260809 12:15:15.368595 22836 tablet_bootstrap.cc:492] T fc7aa1df79e647ff834067e8daef8985 P 1fcdf59a28b54be7a8f7c793b2c475ff: Bootstrap starting.
I20260809 12:15:15.369138 22835 log.cc:824] T fc7aa1df79e647ff834067e8daef8985 P f6d49565647a486eb53ccf7685f9321a: Log is configured to *not* fsync() on all Append() calls
I20260809 12:15:15.372309 22836 tablet_bootstrap.cc:654] T fc7aa1df79e647ff834067e8daef8985 P 1fcdf59a28b54be7a8f7c793b2c475ff: Neither blocks nor log segments found. Creating new log.
I20260809 12:15:15.372363 22835 tablet_bootstrap.cc:492] T fc7aa1df79e647ff834067e8daef8985 P f6d49565647a486eb53ccf7685f9321a: No bootstrap required, opened a new log
I20260809 12:15:15.372408 22834 tablet_bootstrap.cc:492] T fc7aa1df79e647ff834067e8daef8985 P b8128aa81c32435eb3bda70dd5ecbd3d: No bootstrap required, opened a new log
I20260809 12:15:15.372644 22835 ts_tablet_manager.cc:1397] T fc7aa1df79e647ff834067e8daef8985 P f6d49565647a486eb53ccf7685f9321a: Time spent bootstrapping tablet: real 0.009s user 0.006s sys 0.000s
I20260809 12:15:15.372694 22834 ts_tablet_manager.cc:1397] T fc7aa1df79e647ff834067e8daef8985 P b8128aa81c32435eb3bda70dd5ecbd3d: Time spent bootstrapping tablet: real 0.011s user 0.002s sys 0.007s
I20260809 12:15:15.373536 22836 log.cc:824] T fc7aa1df79e647ff834067e8daef8985 P 1fcdf59a28b54be7a8f7c793b2c475ff: Log is configured to *not* fsync() on all Append() calls
I20260809 12:15:15.376623 22836 tablet_bootstrap.cc:492] T fc7aa1df79e647ff834067e8daef8985 P 1fcdf59a28b54be7a8f7c793b2c475ff: No bootstrap required, opened a new log
I20260809 12:15:15.376885 22836 ts_tablet_manager.cc:1397] T fc7aa1df79e647ff834067e8daef8985 P 1fcdf59a28b54be7a8f7c793b2c475ff: Time spent bootstrapping tablet: real 0.008s user 0.006s sys 0.000s
I20260809 12:15:15.380237 22835 raft_consensus.cc:348] T fc7aa1df79e647ff834067e8daef8985 P f6d49565647a486eb53ccf7685f9321a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b8128aa81c32435eb3bda70dd5ecbd3d" member_type: VOTER last_known_addr { host: "127.0.222.3" port: 33683 } } peers { permanent_uuid: "f6d49565647a486eb53ccf7685f9321a" member_type: VOTER last_known_addr { host: "127.0.222.2" port: 39473 } } peers { permanent_uuid: "1fcdf59a28b54be7a8f7c793b2c475ff" member_type: VOTER last_known_addr { host: "127.0.222.1" port: 40663 } }
I20260809 12:15:15.380631 22835 raft_consensus.cc:374] T fc7aa1df79e647ff834067e8daef8985 P f6d49565647a486eb53ccf7685f9321a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260809 12:15:15.380759 22835 raft_consensus.cc:729] T fc7aa1df79e647ff834067e8daef8985 P f6d49565647a486eb53ccf7685f9321a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f6d49565647a486eb53ccf7685f9321a, State: Initialized, Role: FOLLOWER
I20260809 12:15:15.381433 22835 consensus_queue.cc:260] T fc7aa1df79e647ff834067e8daef8985 P f6d49565647a486eb53ccf7685f9321a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b8128aa81c32435eb3bda70dd5ecbd3d" member_type: VOTER last_known_addr { host: "127.0.222.3" port: 33683 } } peers { permanent_uuid: "f6d49565647a486eb53ccf7685f9321a" member_type: VOTER last_known_addr { host: "127.0.222.2" port: 39473 } } peers { permanent_uuid: "1fcdf59a28b54be7a8f7c793b2c475ff" member_type: VOTER last_known_addr { host: "127.0.222.1" port: 40663 } }
I20260809 12:15:15.383046 22834 raft_consensus.cc:348] T fc7aa1df79e647ff834067e8daef8985 P b8128aa81c32435eb3bda70dd5ecbd3d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b8128aa81c32435eb3bda70dd5ecbd3d" member_type: VOTER last_known_addr { host: "127.0.222.3" port: 33683 } } peers { permanent_uuid: "f6d49565647a486eb53ccf7685f9321a" member_type: VOTER last_known_addr { host: "127.0.222.2" port: 39473 } } peers { permanent_uuid: "1fcdf59a28b54be7a8f7c793b2c475ff" member_type: VOTER last_known_addr { host: "127.0.222.1" port: 40663 } }
I20260809 12:15:15.383448 22669 heartbeater.cc:503] Master 127.0.222.62:33597 was elected leader, sending a full tablet report...
I20260809 12:15:15.383574 22834 raft_consensus.cc:374] T fc7aa1df79e647ff834067e8daef8985 P b8128aa81c32435eb3bda70dd5ecbd3d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260809 12:15:15.383720 22834 raft_consensus.cc:729] T fc7aa1df79e647ff834067e8daef8985 P b8128aa81c32435eb3bda70dd5ecbd3d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b8128aa81c32435eb3bda70dd5ecbd3d, State: Initialized, Role: FOLLOWER
I20260809 12:15:15.384142 22835 ts_tablet_manager.cc:1428] T fc7aa1df79e647ff834067e8daef8985 P f6d49565647a486eb53ccf7685f9321a: Time spent starting tablet: real 0.011s user 0.003s sys 0.009s
I20260809 12:15:15.384457 22834 consensus_queue.cc:260] T fc7aa1df79e647ff834067e8daef8985 P b8128aa81c32435eb3bda70dd5ecbd3d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b8128aa81c32435eb3bda70dd5ecbd3d" member_type: VOTER last_known_addr { host: "127.0.222.3" port: 33683 } } peers { permanent_uuid: "f6d49565647a486eb53ccf7685f9321a" member_type: VOTER last_known_addr { host: "127.0.222.2" port: 39473 } } peers { permanent_uuid: "1fcdf59a28b54be7a8f7c793b2c475ff" member_type: VOTER last_known_addr { host: "127.0.222.1" port: 40663 } }
I20260809 12:15:15.385651 22811 heartbeater.cc:503] Master 127.0.222.62:33597 was elected leader, sending a full tablet report...
I20260809 12:15:15.386325 22834 ts_tablet_manager.cc:1428] T fc7aa1df79e647ff834067e8daef8985 P b8128aa81c32435eb3bda70dd5ecbd3d: Time spent starting tablet: real 0.014s user 0.012s sys 0.000s
I20260809 12:15:15.388407 22836 raft_consensus.cc:348] T fc7aa1df79e647ff834067e8daef8985 P 1fcdf59a28b54be7a8f7c793b2c475ff [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b8128aa81c32435eb3bda70dd5ecbd3d" member_type: VOTER last_known_addr { host: "127.0.222.3" port: 33683 } } peers { permanent_uuid: "f6d49565647a486eb53ccf7685f9321a" member_type: VOTER last_known_addr { host: "127.0.222.2" port: 39473 } } peers { permanent_uuid: "1fcdf59a28b54be7a8f7c793b2c475ff" member_type: VOTER last_known_addr { host: "127.0.222.1" port: 40663 } }
I20260809 12:15:15.388866 22836 raft_consensus.cc:374] T fc7aa1df79e647ff834067e8daef8985 P 1fcdf59a28b54be7a8f7c793b2c475ff [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260809 12:15:15.388978 22836 raft_consensus.cc:729] T fc7aa1df79e647ff834067e8daef8985 P 1fcdf59a28b54be7a8f7c793b2c475ff [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1fcdf59a28b54be7a8f7c793b2c475ff, State: Initialized, Role: FOLLOWER
I20260809 12:15:15.389600 22836 consensus_queue.cc:260] T fc7aa1df79e647ff834067e8daef8985 P 1fcdf59a28b54be7a8f7c793b2c475ff [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b8128aa81c32435eb3bda70dd5ecbd3d" member_type: VOTER last_known_addr { host: "127.0.222.3" port: 33683 } } peers { permanent_uuid: "f6d49565647a486eb53ccf7685f9321a" member_type: VOTER last_known_addr { host: "127.0.222.2" port: 39473 } } peers { permanent_uuid: "1fcdf59a28b54be7a8f7c793b2c475ff" member_type: VOTER last_known_addr { host: "127.0.222.1" port: 40663 } }
I20260809 12:15:15.391620 22836 ts_tablet_manager.cc:1428] T fc7aa1df79e647ff834067e8daef8985 P 1fcdf59a28b54be7a8f7c793b2c475ff: Time spent starting tablet: real 0.015s user 0.015s sys 0.000s
W20260809 12:15:15.517673 22812 tablet.cc:2369] T fc7aa1df79e647ff834067e8daef8985 P b8128aa81c32435eb3bda70dd5ecbd3d: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260809 12:15:15.532086 22670 tablet.cc:2369] T fc7aa1df79e647ff834067e8daef8985 P f6d49565647a486eb53ccf7685f9321a: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260809 12:15:15.552033 22842 raft_consensus.cc:482] T fc7aa1df79e647ff834067e8daef8985 P b8128aa81c32435eb3bda70dd5ecbd3d [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260809 12:15:15.552353 22842 raft_consensus.cc:504] T fc7aa1df79e647ff834067e8daef8985 P b8128aa81c32435eb3bda70dd5ecbd3d [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b8128aa81c32435eb3bda70dd5ecbd3d" member_type: VOTER last_known_addr { host: "127.0.222.3" port: 33683 } } peers { permanent_uuid: "f6d49565647a486eb53ccf7685f9321a" member_type: VOTER last_known_addr { host: "127.0.222.2" port: 39473 } } peers { permanent_uuid: "1fcdf59a28b54be7a8f7c793b2c475ff" member_type: VOTER last_known_addr { host: "127.0.222.1" port: 40663 } }
I20260809 12:15:15.554018 22842 leader_election.cc:288] T fc7aa1df79e647ff834067e8daef8985 P b8128aa81c32435eb3bda70dd5ecbd3d [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers f6d49565647a486eb53ccf7685f9321a (127.0.222.2:39473), 1fcdf59a28b54be7a8f7c793b2c475ff (127.0.222.1:40663)
I20260809 12:15:15.560680 22833 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 12:15:15.554535 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.222.3:44747 (local address 127.0.222.1:40663)
0809 12:15:15.554759 (+ 224us) server_negotiation.cc:206] Beginning negotiation
0809 12:15:15.554766 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0809 12:15:15.554800 (+ 34us) server_negotiation.cc:407] Connection header received
0809 12:15:15.555106 (+ 306us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 12:15:15.555116 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 12:15:15.555231 (+ 115us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 12:15:15.555353 (+ 122us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0809 12:15:15.555783 (+ 430us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 12:15:15.556816 (+ 1033us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 12:15:15.558783 (+ 1967us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 12:15:15.559894 (+ 1111us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 12:15:15.559986 (+ 92us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 12:15:15.560058 (+ 72us) server_negotiation.cc:1035] Waiting for connection context
0809 12:15:15.560416 (+ 358us) server_negotiation.cc:299] Negotiation successful
0809 12:15:15.560503 (+ 87us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":69}
I20260809 12:15:15.560856 22813 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 12:15:15.554543 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.222.1:40663 (local address 127.0.222.3:44747)
0809 12:15:15.554709 (+ 166us) negotiation.cc:107] Waiting for socket to connect
0809 12:15:15.554731 (+ 22us) client_negotiation.cc:174] Beginning negotiation
0809 12:15:15.554897 (+ 166us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 12:15:15.555430 (+ 533us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 12:15:15.555435 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 12:15:15.555475 (+ 40us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0809 12:15:15.555646 (+ 171us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 12:15:15.555654 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 12:15:15.556976 (+ 1322us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 12:15:15.556989 (+ 13us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 12:15:15.558603 (+ 1614us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 12:15:15.558615 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 12:15:15.560056 (+ 1441us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 12:15:15.560062 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 12:15:15.560236 (+ 174us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 12:15:15.560276 (+ 40us) client_negotiation.cc:769] Sending connection context
0809 12:15:15.560375 (+ 99us) client_negotiation.cc:240] Negotiation successful
0809 12:15:15.560513 (+ 138us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":40,"mutex_wait_us":1}
I20260809 12:15:15.561527 22831 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 12:15:15.554583 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.222.3:44905 (local address 127.0.222.2:39473)
0809 12:15:15.554784 (+ 201us) server_negotiation.cc:206] Beginning negotiation
0809 12:15:15.554792 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0809 12:15:15.555671 (+ 879us) server_negotiation.cc:407] Connection header received
0809 12:15:15.555882 (+ 211us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 12:15:15.555886 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 12:15:15.555957 (+ 71us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 12:15:15.556029 (+ 72us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0809 12:15:15.556648 (+ 619us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 12:15:15.557612 (+ 964us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 12:15:15.559618 (+ 2006us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 12:15:15.560640 (+ 1022us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 12:15:15.560761 (+ 121us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 12:15:15.560854 (+ 93us) server_negotiation.cc:1035] Waiting for connection context
0809 12:15:15.561200 (+ 346us) server_negotiation.cc:299] Negotiation successful
0809 12:15:15.561299 (+ 99us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":65}
I20260809 12:15:15.562177 22844 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 12:15:15.554559 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.222.2:39473 (local address 127.0.222.3:44905)
0809 12:15:15.555560 (+ 1001us) negotiation.cc:107] Waiting for socket to connect
0809 12:15:15.555576 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0809 12:15:15.555727 (+ 151us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 12:15:15.556087 (+ 360us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 12:15:15.556092 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 12:15:15.556129 (+ 37us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0809 12:15:15.556493 (+ 364us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 12:15:15.556504 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 12:15:15.557776 (+ 1272us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 12:15:15.557780 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 12:15:15.559428 (+ 1648us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 12:15:15.559444 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 12:15:15.560830 (+ 1386us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 12:15:15.560838 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 12:15:15.560995 (+ 157us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 12:15:15.561031 (+ 36us) client_negotiation.cc:769] Sending connection context
0809 12:15:15.561717 (+ 686us) client_negotiation.cc:240] Negotiation successful
0809 12:15:15.561844 (+ 127us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":879,"mutex_wait_us":168,"thread_start_us":424,"threads_started":1}
I20260809 12:15:15.562002 22476 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fc7aa1df79e647ff834067e8daef8985" candidate_uuid: "b8128aa81c32435eb3bda70dd5ecbd3d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1fcdf59a28b54be7a8f7c793b2c475ff" is_pre_election: true
I20260809 12:15:15.562570 22476 raft_consensus.cc:2454] T fc7aa1df79e647ff834067e8daef8985 P 1fcdf59a28b54be7a8f7c793b2c475ff [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b8128aa81c32435eb3bda70dd5ecbd3d in term 0.
I20260809 12:15:15.563287 22694 leader_election.cc:302] T fc7aa1df79e647ff834067e8daef8985 P b8128aa81c32435eb3bda70dd5ecbd3d [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1fcdf59a28b54be7a8f7c793b2c475ff, b8128aa81c32435eb3bda70dd5ecbd3d; no voters:
I20260809 12:15:15.563293 22619 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fc7aa1df79e647ff834067e8daef8985" candidate_uuid: "b8128aa81c32435eb3bda70dd5ecbd3d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f6d49565647a486eb53ccf7685f9321a" is_pre_election: true
I20260809 12:15:15.563720 22842 raft_consensus.cc:2781] T fc7aa1df79e647ff834067e8daef8985 P b8128aa81c32435eb3bda70dd5ecbd3d [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260809 12:15:15.563889 22842 raft_consensus.cc:482] T fc7aa1df79e647ff834067e8daef8985 P b8128aa81c32435eb3bda70dd5ecbd3d [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260809 12:15:15.563865 22619 raft_consensus.cc:2454] T fc7aa1df79e647ff834067e8daef8985 P f6d49565647a486eb53ccf7685f9321a [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b8128aa81c32435eb3bda70dd5ecbd3d in term 0.
I20260809 12:15:15.564021 22842 raft_consensus.cc:3037] T fc7aa1df79e647ff834067e8daef8985 P b8128aa81c32435eb3bda70dd5ecbd3d [term 0 FOLLOWER]: Advancing to term 1
I20260809 12:15:15.566397 22842 raft_consensus.cc:504] T fc7aa1df79e647ff834067e8daef8985 P b8128aa81c32435eb3bda70dd5ecbd3d [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b8128aa81c32435eb3bda70dd5ecbd3d" member_type: VOTER last_known_addr { host: "127.0.222.3" port: 33683 } } peers { permanent_uuid: "f6d49565647a486eb53ccf7685f9321a" member_type: VOTER last_known_addr { host: "127.0.222.2" port: 39473 } } peers { permanent_uuid: "1fcdf59a28b54be7a8f7c793b2c475ff" member_type: VOTER last_known_addr { host: "127.0.222.1" port: 40663 } }
I20260809 12:15:15.567024 22842 leader_election.cc:288] T fc7aa1df79e647ff834067e8daef8985 P b8128aa81c32435eb3bda70dd5ecbd3d [CANDIDATE]: Term 1 election: Requested vote from peers f6d49565647a486eb53ccf7685f9321a (127.0.222.2:39473), 1fcdf59a28b54be7a8f7c793b2c475ff (127.0.222.1:40663)
I20260809 12:15:15.567366 22619 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fc7aa1df79e647ff834067e8daef8985" candidate_uuid: "b8128aa81c32435eb3bda70dd5ecbd3d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f6d49565647a486eb53ccf7685f9321a"
I20260809 12:15:15.567508 22476 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fc7aa1df79e647ff834067e8daef8985" candidate_uuid: "b8128aa81c32435eb3bda70dd5ecbd3d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1fcdf59a28b54be7a8f7c793b2c475ff"
I20260809 12:15:15.567580 22619 raft_consensus.cc:3037] T fc7aa1df79e647ff834067e8daef8985 P f6d49565647a486eb53ccf7685f9321a [term 0 FOLLOWER]: Advancing to term 1
I20260809 12:15:15.567735 22476 raft_consensus.cc:3037] T fc7aa1df79e647ff834067e8daef8985 P 1fcdf59a28b54be7a8f7c793b2c475ff [term 0 FOLLOWER]: Advancing to term 1
I20260809 12:15:15.570046 22619 raft_consensus.cc:2454] T fc7aa1df79e647ff834067e8daef8985 P f6d49565647a486eb53ccf7685f9321a [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b8128aa81c32435eb3bda70dd5ecbd3d in term 1.
I20260809 12:15:15.570050 22476 raft_consensus.cc:2454] T fc7aa1df79e647ff834067e8daef8985 P 1fcdf59a28b54be7a8f7c793b2c475ff [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b8128aa81c32435eb3bda70dd5ecbd3d in term 1.
I20260809 12:15:15.570461 22691 leader_election.cc:302] T fc7aa1df79e647ff834067e8daef8985 P b8128aa81c32435eb3bda70dd5ecbd3d [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: b8128aa81c32435eb3bda70dd5ecbd3d, f6d49565647a486eb53ccf7685f9321a; no voters:
I20260809 12:15:15.570778 22842 raft_consensus.cc:2781] T fc7aa1df79e647ff834067e8daef8985 P b8128aa81c32435eb3bda70dd5ecbd3d [term 1 FOLLOWER]: Leader election won for term 1
I20260809 12:15:15.571324 22842 raft_consensus.cc:686] T fc7aa1df79e647ff834067e8daef8985 P b8128aa81c32435eb3bda70dd5ecbd3d [term 1 LEADER]: Becoming Leader. State: Replica: b8128aa81c32435eb3bda70dd5ecbd3d, State: Running, Role: LEADER
I20260809 12:15:15.571846 22842 consensus_queue.cc:237] T fc7aa1df79e647ff834067e8daef8985 P b8128aa81c32435eb3bda70dd5ecbd3d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b8128aa81c32435eb3bda70dd5ecbd3d" member_type: VOTER last_known_addr { host: "127.0.222.3" port: 33683 } } peers { permanent_uuid: "f6d49565647a486eb53ccf7685f9321a" member_type: VOTER last_known_addr { host: "127.0.222.2" port: 39473 } } peers { permanent_uuid: "1fcdf59a28b54be7a8f7c793b2c475ff" member_type: VOTER last_known_addr { host: "127.0.222.1" port: 40663 } }
W20260809 12:15:15.573963 22527 tablet.cc:2369] T fc7aa1df79e647ff834067e8daef8985 P 1fcdf59a28b54be7a8f7c793b2c475ff: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260809 12:15:15.576676 22321 catalog_manager.cc:5495] T fc7aa1df79e647ff834067e8daef8985 P b8128aa81c32435eb3bda70dd5ecbd3d reported cstate change: term changed from 0 to 1, leader changed from <none> to b8128aa81c32435eb3bda70dd5ecbd3d (127.0.222.3). New cstate: current_term: 1 leader_uuid: "b8128aa81c32435eb3bda70dd5ecbd3d" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b8128aa81c32435eb3bda70dd5ecbd3d" member_type: VOTER last_known_addr { host: "127.0.222.3" port: 33683 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "f6d49565647a486eb53ccf7685f9321a" member_type: VOTER last_known_addr { host: "127.0.222.2" port: 39473 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "1fcdf59a28b54be7a8f7c793b2c475ff" member_type: VOTER last_known_addr { host: "127.0.222.1" port: 40663 } health_report { overall_health: UNKNOWN } } }
I20260809 12:15:15.640763 22830 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 12:15:15.635427 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34504 (local address 127.0.222.3:33683)
0809 12:15:15.635609 (+ 182us) server_negotiation.cc:206] Beginning negotiation
0809 12:15:15.635614 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0809 12:15:15.635632 (+ 18us) server_negotiation.cc:407] Connection header received
0809 12:15:15.635816 (+ 184us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 12:15:15.635820 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 12:15:15.635900 (+ 80us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 12:15:15.636024 (+ 124us) server_negotiation.cc:226] Negotiated authn=TOKEN
0809 12:15:15.636396 (+ 372us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 12:15:15.637120 (+ 724us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 12:15:15.638624 (+ 1504us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 12:15:15.639090 (+ 466us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 12:15:15.639187 (+ 97us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 12:15:15.639656 (+ 469us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0809 12:15:15.640148 (+ 492us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0809 12:15:15.640316 (+ 168us) server_negotiation.cc:1035] Waiting for connection context
0809 12:15:15.640413 (+ 97us) server_negotiation.cc:299] Negotiation successful
0809 12:15:15.640505 (+ 92us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":56}
I20260809 12:15:15.649962 22619 raft_consensus.cc:1261] T fc7aa1df79e647ff834067e8daef8985 P f6d49565647a486eb53ccf7685f9321a [term 1 FOLLOWER]: Refusing update from remote peer b8128aa81c32435eb3bda70dd5ecbd3d: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260809 12:15:15.650045 22476 raft_consensus.cc:1261] T fc7aa1df79e647ff834067e8daef8985 P 1fcdf59a28b54be7a8f7c793b2c475ff [term 1 FOLLOWER]: Refusing update from remote peer b8128aa81c32435eb3bda70dd5ecbd3d: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260809 12:15:15.651126 22845 consensus_queue.cc:1035] T fc7aa1df79e647ff834067e8daef8985 P b8128aa81c32435eb3bda70dd5ecbd3d [LEADER]: Connected to new peer: Peer: permanent_uuid: "1fcdf59a28b54be7a8f7c793b2c475ff" member_type: VOTER last_known_addr { host: "127.0.222.1" port: 40663 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260809 12:15:15.651872 22842 consensus_queue.cc:1035] T fc7aa1df79e647ff834067e8daef8985 P b8128aa81c32435eb3bda70dd5ecbd3d [LEADER]: Connected to new peer: Peer: permanent_uuid: "f6d49565647a486eb53ccf7685f9321a" member_type: VOTER last_known_addr { host: "127.0.222.2" port: 39473 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260809 12:15:15.678426 22321 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:37684:
table { table_name: "test-table" } modify_external_catalogs: true
I20260809 12:15:15.678745 22321 catalog_manager.cc:2645] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:37684:
table { table_name: "test-table" } modify_external_catalogs: true
I20260809 12:15:15.682932 22321 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 8583eb54d4754b7eb7dd5adff1560715: Sending DeleteTablet for 3 replicas of tablet fc7aa1df79e647ff834067e8daef8985
I20260809 12:15:15.684355 22597 tablet_service.cc:1514] Processing DeleteTablet for tablet fc7aa1df79e647ff834067e8daef8985 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-09 12:15:15 UTC) from {username='slave'} at 127.0.0.1:60408
I20260809 12:15:15.684355 22739 tablet_service.cc:1514] Processing DeleteTablet for tablet fc7aa1df79e647ff834067e8daef8985 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-09 12:15:15 UTC) from {username='slave'} at 127.0.0.1:34494
I20260809 12:15:15.684484 22455 tablet_service.cc:1514] Processing DeleteTablet for tablet fc7aa1df79e647ff834067e8daef8985 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-09 12:15:15 UTC) from {username='slave'} at 127.0.0.1:59536
I20260809 12:15:15.686981 888 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskgH5Pjm/build/asan/bin/kudu pid 22385
I20260809 12:15:15.695251 22858 tablet_replica.cc:330] T fc7aa1df79e647ff834067e8daef8985 P f6d49565647a486eb53ccf7685f9321a: stopping tablet replica
I20260809 12:15:15.695680 22858 raft_consensus.cc:2229] T fc7aa1df79e647ff834067e8daef8985 P f6d49565647a486eb53ccf7685f9321a [term 1 FOLLOWER]: Raft consensus shutting down.
I20260809 12:15:15.696182 22858 raft_consensus.cc:2258] T fc7aa1df79e647ff834067e8daef8985 P f6d49565647a486eb53ccf7685f9321a [term 1 FOLLOWER]: Raft consensus is shut down!
I20260809 12:15:15.699003 22858 ts_tablet_manager.cc:1905] T fc7aa1df79e647ff834067e8daef8985 P f6d49565647a486eb53ccf7685f9321a: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260809 12:15:15.701711 22859 tablet_replica.cc:330] T fc7aa1df79e647ff834067e8daef8985 P b8128aa81c32435eb3bda70dd5ecbd3d: stopping tablet replica
I20260809 12:15:15.702275 22859 raft_consensus.cc:2229] T fc7aa1df79e647ff834067e8daef8985 P b8128aa81c32435eb3bda70dd5ecbd3d [term 1 LEADER]: Raft consensus shutting down.
I20260809 12:15:15.702996 22859 raft_consensus.cc:2258] T fc7aa1df79e647ff834067e8daef8985 P b8128aa81c32435eb3bda70dd5ecbd3d [term 1 FOLLOWER]: Raft consensus is shut down!
I20260809 12:15:15.705199 22859 ts_tablet_manager.cc:1905] T fc7aa1df79e647ff834067e8daef8985 P b8128aa81c32435eb3bda70dd5ecbd3d: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260809 12:15:15.706877 22858 ts_tablet_manager.cc:1918] T fc7aa1df79e647ff834067e8daef8985 P f6d49565647a486eb53ccf7685f9321a: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260809 12:15:15.707126 22858 log.cc:1196] T fc7aa1df79e647ff834067e8daef8985 P f6d49565647a486eb53ccf7685f9321a: Deleting WAL directory at /tmp/dist-test-taskgH5Pjm/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786277382073938-888-0/minicluster-data/ts-1/wal/wals/fc7aa1df79e647ff834067e8daef8985
I20260809 12:15:15.707773 22858 ts_tablet_manager.cc:1939] T fc7aa1df79e647ff834067e8daef8985 P f6d49565647a486eb53ccf7685f9321a: Deleting consensus metadata
I20260809 12:15:15.709836 22304 catalog_manager.cc:4841] TS f6d49565647a486eb53ccf7685f9321a (127.0.222.2:39473): tablet fc7aa1df79e647ff834067e8daef8985 (table test-table [id=1bd6c1bac46849838d328f21113200fc]) successfully deleted
I20260809 12:15:15.712302 22860 tablet_replica.cc:330] T fc7aa1df79e647ff834067e8daef8985 P 1fcdf59a28b54be7a8f7c793b2c475ff: stopping tablet replica
I20260809 12:15:15.712288 22859 ts_tablet_manager.cc:1918] T fc7aa1df79e647ff834067e8daef8985 P b8128aa81c32435eb3bda70dd5ecbd3d: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260809 12:15:15.712474 22859 log.cc:1196] T fc7aa1df79e647ff834067e8daef8985 P b8128aa81c32435eb3bda70dd5ecbd3d: Deleting WAL directory at /tmp/dist-test-taskgH5Pjm/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786277382073938-888-0/minicluster-data/ts-2/wal/wals/fc7aa1df79e647ff834067e8daef8985
I20260809 12:15:15.713003 22860 raft_consensus.cc:2229] T fc7aa1df79e647ff834067e8daef8985 P 1fcdf59a28b54be7a8f7c793b2c475ff [term 1 FOLLOWER]: Raft consensus shutting down.
I20260809 12:15:15.713141 22859 ts_tablet_manager.cc:1939] T fc7aa1df79e647ff834067e8daef8985 P b8128aa81c32435eb3bda70dd5ecbd3d: Deleting consensus metadata
I20260809 12:15:15.713701 22860 raft_consensus.cc:2258] T fc7aa1df79e647ff834067e8daef8985 P 1fcdf59a28b54be7a8f7c793b2c475ff [term 1 FOLLOWER]: Raft consensus is shut down!
I20260809 12:15:15.715159 22306 catalog_manager.cc:4841] TS b8128aa81c32435eb3bda70dd5ecbd3d (127.0.222.3:33683): tablet fc7aa1df79e647ff834067e8daef8985 (table test-table [id=1bd6c1bac46849838d328f21113200fc]) successfully deleted
I20260809 12:15:15.716048 22860 ts_tablet_manager.cc:1905] T fc7aa1df79e647ff834067e8daef8985 P 1fcdf59a28b54be7a8f7c793b2c475ff: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260809 12:15:15.720659 22833 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 12:15:15.688697 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59552 (local address 127.0.222.1:40663)
0809 12:15:15.688916 (+ 219us) server_negotiation.cc:206] Beginning negotiation
0809 12:15:15.688923 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0809 12:15:15.711573 (+ 22650us) server_negotiation.cc:407] Connection header received
0809 12:15:15.711788 (+ 215us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 12:15:15.711795 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 12:15:15.711957 (+ 162us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 12:15:15.712087 (+ 130us) server_negotiation.cc:226] Negotiated authn=SASL
0809 12:15:15.712926 (+ 839us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 12:15:15.714359 (+ 1433us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 12:15:15.716645 (+ 2286us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 12:15:15.717280 (+ 635us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 12:15:15.717445 (+ 165us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 12:15:15.718817 (+ 1372us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 12:15:15.718828 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 12:15:15.718847 (+ 19us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0809 12:15:15.719692 (+ 845us) server_negotiation.cc:922] calling sasl_server_start()
0809 12:15:15.719755 (+ 63us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0809 12:15:15.719781 (+ 26us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 12:15:15.719929 (+ 148us) server_negotiation.cc:1035] Waiting for connection context
0809 12:15:15.720107 (+ 178us) server_negotiation.cc:299] Negotiation successful
0809 12:15:15.720231 (+ 124us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":74}
I20260809 12:15:15.721558 22520 generic_service.cc:196] Checking for leaks (request via RPC)
I20260809 12:15:15.815331 22860 ts_tablet_manager.cc:1918] T fc7aa1df79e647ff834067e8daef8985 P 1fcdf59a28b54be7a8f7c793b2c475ff: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260809 12:15:15.815622 22860 log.cc:1196] T fc7aa1df79e647ff834067e8daef8985 P 1fcdf59a28b54be7a8f7c793b2c475ff: Deleting WAL directory at /tmp/dist-test-taskgH5Pjm/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786277382073938-888-0/minicluster-data/ts-0/wal/wals/fc7aa1df79e647ff834067e8daef8985
I20260809 12:15:15.815953 888 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskgH5Pjm/build/asan/bin/kudu with pid 22385
W20260809 12:15:15.843853 22307 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.0.222.1:40663 (error 108)
I20260809 12:15:15.844632 888 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskgH5Pjm/build/asan/bin/kudu pid 22531
I20260809 12:15:15.846174 22864 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 12:15:15.845180 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.222.1:40663 (local address 127.0.0.1:59560)
0809 12:15:15.845819 (+ 639us) negotiation.cc:107] Waiting for socket to connect
0809 12:15:15.845971 (+ 152us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.222.1:40663: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":521,"thread_start_us":423,"threads_started":1}
W20260809 12:15:15.846562 22307 catalog_manager.cc:4584] TS 1fcdf59a28b54be7a8f7c793b2c475ff (127.0.222.1:40663): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet fc7aa1df79e647ff834067e8daef8985: Network error: Client connection negotiation failed: client connection to 127.0.222.1:40663: connect: Connection refused (error 111)
I20260809 12:15:15.853057 22831 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 12:15:15.845408 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60412 (local address 127.0.222.2:39473)
0809 12:15:15.845647 (+ 239us) server_negotiation.cc:206] Beginning negotiation
0809 12:15:15.845654 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0809 12:15:15.845932 (+ 278us) server_negotiation.cc:407] Connection header received
0809 12:15:15.846021 (+ 89us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 12:15:15.846027 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 12:15:15.846158 (+ 131us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 12:15:15.846282 (+ 124us) server_negotiation.cc:226] Negotiated authn=SASL
0809 12:15:15.846833 (+ 551us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 12:15:15.848040 (+ 1207us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 12:15:15.849309 (+ 1269us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 12:15:15.850020 (+ 711us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 12:15:15.850152 (+ 132us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 12:15:15.851419 (+ 1267us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 12:15:15.851431 (+ 12us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 12:15:15.851449 (+ 18us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0809 12:15:15.852228 (+ 779us) server_negotiation.cc:922] calling sasl_server_start()
0809 12:15:15.852299 (+ 71us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0809 12:15:15.852325 (+ 26us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 12:15:15.852441 (+ 116us) server_negotiation.cc:1035] Waiting for connection context
0809 12:15:15.852584 (+ 143us) server_negotiation.cc:299] Negotiation successful
0809 12:15:15.852697 (+ 113us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":54}
I20260809 12:15:15.853866 22663 generic_service.cc:196] Checking for leaks (request via RPC)
I20260809 12:15:15.900198 22864 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 12:15:15.899636 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.222.1:40663 (local address 127.0.0.1:59568)
0809 12:15:15.899888 (+ 252us) negotiation.cc:107] Waiting for socket to connect
0809 12:15:15.900008 (+ 120us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.222.1:40663: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":83}
I20260809 12:15:15.901539 22864 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 12:15:15.901075 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.222.1:40663 (local address 127.0.0.1:59570)
0809 12:15:15.901244 (+ 169us) negotiation.cc:107] Waiting for socket to connect
0809 12:15:15.901356 (+ 112us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.222.1:40663: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":39}
I20260809 12:15:15.946493 888 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskgH5Pjm/build/asan/bin/kudu with pid 22531
I20260809 12:15:15.962572 22864 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 12:15:15.961907 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.222.1:40663 (local address 127.0.0.1:59574)
0809 12:15:15.962207 (+ 300us) negotiation.cc:107] Waiting for socket to connect
0809 12:15:15.962340 (+ 133us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.222.1:40663: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":97}
I20260809 12:15:15.963991 22864 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 12:15:15.963503 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.222.1:40663 (local address 127.0.0.1:59576)
0809 12:15:15.963693 (+ 190us) negotiation.cc:107] Waiting for socket to connect
0809 12:15:15.963795 (+ 102us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.222.1:40663: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":66}
I20260809 12:15:15.969594 888 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskgH5Pjm/build/asan/bin/kudu pid 22672
I20260809 12:15:15.977521 22830 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 12:15:15.970407 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34506 (local address 127.0.222.3:33683)
0809 12:15:15.970583 (+ 176us) server_negotiation.cc:206] Beginning negotiation
0809 12:15:15.970587 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0809 12:15:15.970696 (+ 109us) server_negotiation.cc:407] Connection header received
0809 12:15:15.970874 (+ 178us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 12:15:15.970877 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 12:15:15.970955 (+ 78us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 12:15:15.971107 (+ 152us) server_negotiation.cc:226] Negotiated authn=SASL
0809 12:15:15.971589 (+ 482us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 12:15:15.972393 (+ 804us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 12:15:15.974360 (+ 1967us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 12:15:15.974790 (+ 430us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 12:15:15.974886 (+ 96us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 12:15:15.976112 (+ 1226us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 12:15:15.976121 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 12:15:15.976135 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0809 12:15:15.976698 (+ 563us) server_negotiation.cc:922] calling sasl_server_start()
0809 12:15:15.976744 (+ 46us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0809 12:15:15.976767 (+ 23us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 12:15:15.976857 (+ 90us) server_negotiation.cc:1035] Waiting for connection context
0809 12:15:15.977036 (+ 179us) server_negotiation.cc:299] Negotiation successful
0809 12:15:15.977121 (+ 85us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":65}
I20260809 12:15:15.978312 22806 generic_service.cc:196] Checking for leaks (request via RPC)
I20260809 12:15:16.044727 22864 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 12:15:16.044114 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.222.1:40663 (local address 127.0.0.1:59590)
0809 12:15:16.044417 (+ 303us) negotiation.cc:107] Waiting for socket to connect
0809 12:15:16.044536 (+ 119us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.222.1:40663: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":101}
I20260809 12:15:16.045977 22864 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 12:15:16.045478 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.222.1:40663 (local address 127.0.0.1:59600)
0809 12:15:16.045699 (+ 221us) negotiation.cc:107] Waiting for socket to connect
0809 12:15:16.045791 (+ 92us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.222.1:40663: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":111}
I20260809 12:15:16.101632 888 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskgH5Pjm/build/asan/bin/kudu with pid 22672
I20260809 12:15:16.123834 888 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskgH5Pjm/build/asan/bin/kudu pid 22287
I20260809 12:15:16.125018 22353 generic_service.cc:196] Checking for leaks (request via RPC)
I20260809 12:15:16.236845 22864 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 12:15:16.236389 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.222.1:40663 (local address 127.0.0.1:59608)
0809 12:15:16.236535 (+ 146us) negotiation.cc:107] Waiting for socket to connect
0809 12:15:16.236641 (+ 106us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.222.1:40663: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":37}
I20260809 12:15:16.236879 888 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskgH5Pjm/build/asan/bin/kudu with pid 22287
2026-08-09T12:15:16Z chronyd exiting
[ OK ] EncryptionPolicyTest.TestEncryptionPolicy/5 (3031 ms)
[----------] 6 tests from EncryptionPolicyTest (24216 ms total)
[----------] 18 tests from AuthTokenIssuingTest
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/0
Loading random data
Initializing database '/tmp/dist-test-taskgH5Pjm/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1786277382073938-888-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 09 12:15:16 dist-test-slave-2w3w krb5kdc[22872](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 09 12:15:16 dist-test-slave-2w3w krb5kdc[22872](info): set up 2 sockets
Aug 09 12:15:16 dist-test-slave-2w3w krb5kdc[22872](info): commencing operation
krb5kdc: starting...
W20260809 12:15:17.439998 888 mini_kdc.cc:121] Time spent starting KDC: real 1.155s user 0.003s sys 0.044s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskgH5Pjm/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1786277382073938-888-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskgH5Pjm/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1786277382073938-888-0/krb5kdc/test-user.keytab.
Aug 09 12:15:17 dist-test-slave-2w3w krb5kdc[22872](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786277717, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-09T12:15:17Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-09T12:15:17Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskgH5Pjm/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1786277382073938-888-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskgH5Pjm/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1786277382073938-888-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260809 12:15:17.770008 888 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskgH5Pjm/build/asan/bin/kudu
/tmp/dist-test-taskgH5Pjm/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskgH5Pjm/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1786277382073938-888-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskgH5Pjm/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1786277382073938-888-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskgH5Pjm/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1786277382073938-888-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskgH5Pjm/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1786277382073938-888-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:41211
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskgH5Pjm/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1786277382073938-888-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:36075
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:41211
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=required
--rpc_encryption=required
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taskgH5Pjm/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1786277382073938-888-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskgH5Pjm/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1786277382073938-888-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskgH5Pjm/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1786277382073938-888-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260809 12:15:18.113754 22885 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260809 12:15:18.114053 22885 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260809 12:15:18.114146 22885 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260809 12:15:18.121212 22885 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260809 12:15:18.121318 22885 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260809 12:15:18.121373 22885 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260809 12:15:18.121436 22885 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260809 12:15:18.121481 22885 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260809 12:15:18.131004 22885 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:36075
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskgH5Pjm/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1786277382073938-888-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskgH5Pjm/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1786277382073938-888-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:41211
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskgH5Pjm/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1786277382073938-888-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:41211
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskgH5Pjm/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1786277382073938-888-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskgH5Pjm/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1786277382073938-888-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 09 Aug 2026 11:06:41 UTC on ccycloud.quasar-bwoogy.root.comops.site
build id 4575991
ASAN enabled
I20260809 12:15:18.132223 22885 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260809 12:15:18.133493 22885 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260809 12:15:18.145325 22893 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 12:15:18.145493 22890 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 12:15:18.145417 22891 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260809 12:15:18.146546 22885 server_base.cc:1034] running on GCE node
I20260809 12:15:18.147382 22885 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260809 12:15:18.148727 22885 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260809 12:15:18.149966 22885 hybrid_clock.cc:648] HybridClock initialized: now 1786277718149929 us; error 49 us; skew 500 ppm
I20260809 12:15:18.150305 22885 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 09 12:15:18 dist-test-slave-2w3w krb5kdc[22872](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786277718, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260809 12:15:18.153491 22885 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260809 12:15:18.155092 22885 webserver.cc:460] Webserver started at http://127.0.0.1:42447/ using document root <none> and password file <none>
I20260809 12:15:18.155766 22885 fs_manager.cc:359] Metadata directory not provided
I20260809 12:15:18.155920 22885 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260809 12:15:18.156235 22885 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260809 12:15:18.159291 22885 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskgH5Pjm/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1786277382073938-888-0/minicluster-data/master-0/data/instance:
uuid: "52c3bdc007c54d99a0646ddc0950d948"
format_stamp: "Formatted at 2026-08-09 12:15:18 on dist-test-slave-2w3w"
server_key: "1dccf8f33ae0c838d222f246250c1e3b"
server_key_iv: "eb817808baa36e5aaee3321889604eca"
server_key_version: "encryptionkey@0"
I20260809 12:15:18.160061 22885 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskgH5Pjm/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1786277382073938-888-0/minicluster-data/master-0/wal/instance:
uuid: "52c3bdc007c54d99a0646ddc0950d948"
format_stamp: "Formatted at 2026-08-09 12:15:18 on dist-test-slave-2w3w"
server_key: "1dccf8f33ae0c838d222f246250c1e3b"
server_key_iv: "eb817808baa36e5aaee3321889604eca"
server_key_version: "encryptionkey@0"
I20260809 12:15:18.165612 22885 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.002s sys 0.003s
I20260809 12:15:18.169077 22902 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260809 12:15:18.170331 22885 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260809 12:15:18.170512 22885 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskgH5Pjm/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1786277382073938-888-0/minicluster-data/master-0/data,/tmp/dist-test-taskgH5Pjm/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1786277382073938-888-0/minicluster-data/master-0/wal
uuid: "52c3bdc007c54d99a0646ddc0950d948"
format_stamp: "Formatted at 2026-08-09 12:15:18 on dist-test-slave-2w3w"
server_key: "1dccf8f33ae0c838d222f246250c1e3b"
server_key_iv: "eb817808baa36e5aaee3321889604eca"
server_key_version: "encryptionkey@0"
I20260809 12:15:18.170711 22885 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskgH5Pjm/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1786277382073938-888-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskgH5Pjm/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1786277382073938-888-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskgH5Pjm/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1786277382073938-888-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260809 12:15:18.185025 22885 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260809 12:15:18.189700 22885 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260809 12:15:18.190006 22885 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260809 12:15:18.209132 22885 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:41211
I20260809 12:15:18.209168 22960 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:41211 every 8 connection(s)
I20260809 12:15:18.210565 22885 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskgH5Pjm/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1786277382073938-888-0/minicluster-data/master-0/data/info.pb
I20260809 12:15:18.212504 888 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskgH5Pjm/build/asan/bin/kudu as pid 22885
I20260809 12:15:18.212731 888 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskgH5Pjm/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1786277382073938-888-0/minicluster-data/master-0/wal/instance
I20260809 12:15:18.213377 888 external_mini_cluster.cc:1428] Setting key 37e6d2d910cae212f808d86c0f263411
I20260809 12:15:18.214793 22961 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
Aug 09 12:15:18 dist-test-slave-2w3w krb5kdc[22872](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786277717, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.0.1@KRBTEST.COM
I20260809 12:15:18.229295 22961 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 52c3bdc007c54d99a0646ddc0950d948: Bootstrap starting.
I20260809 12:15:18.232589 22961 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 52c3bdc007c54d99a0646ddc0950d948: Neither blocks nor log segments found. Creating new log.
I20260809 12:15:18.233726 22961 log.cc:824] T 00000000000000000000000000000000 P 52c3bdc007c54d99a0646ddc0950d948: Log is configured to *not* fsync() on all Append() calls
I20260809 12:15:18.236879 22961 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 52c3bdc007c54d99a0646ddc0950d948: No bootstrap required, opened a new log
I20260809 12:15:18.237401 22964 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 12:15:18.215370 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48829 (local address 127.0.0.1:41211)
0809 12:15:18.216174 (+ 804us) server_negotiation.cc:206] Beginning negotiation
0809 12:15:18.216186 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0809 12:15:18.216229 (+ 43us) server_negotiation.cc:407] Connection header received
0809 12:15:18.217868 (+ 1639us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 12:15:18.217889 (+ 21us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 12:15:18.218284 (+ 395us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 12:15:18.218608 (+ 324us) server_negotiation.cc:226] Negotiated authn=SASL
0809 12:15:18.220471 (+ 1863us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 12:15:18.221833 (+ 1362us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 12:15:18.223702 (+ 1869us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 12:15:18.224361 (+ 659us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 12:15:18.224492 (+ 131us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 12:15:18.229532 (+ 5040us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 12:15:18.229540 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 12:15:18.229554 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0809 12:15:18.229599 (+ 45us) server_negotiation.cc:922] calling sasl_server_start()
0809 12:15:18.232799 (+ 3200us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 12:15:18.233755 (+ 956us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 12:15:18.233763 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 12:15:18.233769 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0809 12:15:18.233984 (+ 215us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 12:15:18.234488 (+ 504us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 12:15:18.234493 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 12:15:18.234496 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0809 12:15:18.235169 (+ 673us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 12:15:18.235512 (+ 343us) server_negotiation.cc:1035] Waiting for connection context
0809 12:15:18.236025 (+ 513us) server_negotiation.cc:299] Negotiation successful
0809 12:15:18.236349 (+ 324us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":589,"thread_start_us":371,"threads_started":1}
I20260809 12:15:18.241389 888 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260809 12:15:18.244999 22961 raft_consensus.cc:348] T 00000000000000000000000000000000 P 52c3bdc007c54d99a0646ddc0950d948 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "52c3bdc007c54d99a0646ddc0950d948" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 41211 } }
I20260809 12:15:18.245354 22961 raft_consensus.cc:374] T 00000000000000000000000000000000 P 52c3bdc007c54d99a0646ddc0950d948 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260809 12:15:18.245473 22961 raft_consensus.cc:729] T 00000000000000000000000000000000 P 52c3bdc007c54d99a0646ddc0950d948 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 52c3bdc007c54d99a0646ddc0950d948, State: Initialized, Role: FOLLOWER
I20260809 12:15:18.246029 22961 consensus_queue.cc:260] T 00000000000000000000000000000000 P 52c3bdc007c54d99a0646ddc0950d948 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "52c3bdc007c54d99a0646ddc0950d948" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 41211 } }
I20260809 12:15:18.246282 22961 raft_consensus.cc:388] T 00000000000000000000000000000000 P 52c3bdc007c54d99a0646ddc0950d948 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260809 12:15:18.246436 22961 raft_consensus.cc:482] T 00000000000000000000000000000000 P 52c3bdc007c54d99a0646ddc0950d948 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260809 12:15:18.246686 22961 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 52c3bdc007c54d99a0646ddc0950d948 [term 0 FOLLOWER]: Advancing to term 1
I20260809 12:15:18.250157 22961 raft_consensus.cc:504] T 00000000000000000000000000000000 P 52c3bdc007c54d99a0646ddc0950d948 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "52c3bdc007c54d99a0646ddc0950d948" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 41211 } }
I20260809 12:15:18.250765 22961 leader_election.cc:302] T 00000000000000000000000000000000 P 52c3bdc007c54d99a0646ddc0950d948 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 52c3bdc007c54d99a0646ddc0950d948; no voters:
I20260809 12:15:18.251451 22961 leader_election.cc:288] T 00000000000000000000000000000000 P 52c3bdc007c54d99a0646ddc0950d948 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260809 12:15:18.251564 22972 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 52c3bdc007c54d99a0646ddc0950d948 [term 1 FOLLOWER]: Leader election won for term 1
I20260809 12:15:18.252899 22972 raft_consensus.cc:686] T 00000000000000000000000000000000 P 52c3bdc007c54d99a0646ddc0950d948 [term 1 LEADER]: Becoming Leader. State: Replica: 52c3bdc007c54d99a0646ddc0950d948, State: Running, Role: LEADER
I20260809 12:15:18.253695 22961 sys_catalog.cc:564] T 00000000000000000000000000000000 P 52c3bdc007c54d99a0646ddc0950d948 [sys.catalog]: configured and running, proceeding with master startup.
I20260809 12:15:18.253562 22972 consensus_queue.cc:237] T 00000000000000000000000000000000 P 52c3bdc007c54d99a0646ddc0950d948 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "52c3bdc007c54d99a0646ddc0950d948" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 41211 } }
I20260809 12:15:18.257223 22973 sys_catalog.cc:455] T 00000000000000000000000000000000 P 52c3bdc007c54d99a0646ddc0950d948 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "52c3bdc007c54d99a0646ddc0950d948" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "52c3bdc007c54d99a0646ddc0950d948" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 41211 } } }
I20260809 12:15:18.257282 22974 sys_catalog.cc:455] T 00000000000000000000000000000000 P 52c3bdc007c54d99a0646ddc0950d948 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 52c3bdc007c54d99a0646ddc0950d948. Latest consensus state: current_term: 1 leader_uuid: "52c3bdc007c54d99a0646ddc0950d948" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "52c3bdc007c54d99a0646ddc0950d948" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 41211 } } }
I20260809 12:15:18.257694 22974 sys_catalog.cc:458] T 00000000000000000000000000000000 P 52c3bdc007c54d99a0646ddc0950d948 [sys.catalog]: This master's current role is: LEADER
I20260809 12:15:18.257694 22973 sys_catalog.cc:458] T 00000000000000000000000000000000 P 52c3bdc007c54d99a0646ddc0950d948 [sys.catalog]: This master's current role is: LEADER
I20260809 12:15:18.258911 22980 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260809 12:15:18.266974 22964 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 12:15:18.246720 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37335 (local address 127.0.0.1:41211)
0809 12:15:18.246975 (+ 255us) server_negotiation.cc:206] Beginning negotiation
0809 12:15:18.246984 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0809 12:15:18.247450 (+ 466us) server_negotiation.cc:407] Connection header received
0809 12:15:18.247640 (+ 190us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 12:15:18.247645 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 12:15:18.247713 (+ 68us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 12:15:18.247836 (+ 123us) server_negotiation.cc:226] Negotiated authn=SASL
0809 12:15:18.249658 (+ 1822us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 12:15:18.250500 (+ 842us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 12:15:18.252454 (+ 1954us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 12:15:18.253176 (+ 722us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 12:15:18.253302 (+ 126us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 12:15:18.256965 (+ 3663us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 12:15:18.256974 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 12:15:18.256982 (+ 8us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0809 12:15:18.257036 (+ 54us) server_negotiation.cc:922] calling sasl_server_start()
0809 12:15:18.262004 (+ 4968us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 12:15:18.263283 (+ 1279us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 12:15:18.263292 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 12:15:18.263298 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0809 12:15:18.263579 (+ 281us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 12:15:18.264319 (+ 740us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 12:15:18.264327 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 12:15:18.264333 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0809 12:15:18.264824 (+ 491us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 12:15:18.265219 (+ 395us) server_negotiation.cc:1035] Waiting for connection context
0809 12:15:18.266094 (+ 875us) server_negotiation.cc:299] Negotiation successful
0809 12:15:18.266363 (+ 269us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":81}
I20260809 12:15:18.275907 22980 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260809 12:15:18.286415 22980 catalog_manager.cc:1347] Generated new cluster ID: f185e74dc22145098a076344a8c8212e
I20260809 12:15:18.286551 22980 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260809 12:15:18.314358 22980 catalog_manager.cc:1370] Generated new certificate authority record
I20260809 12:15:18.316260 22980 catalog_manager.cc:1504] Loading token signing keys...
I20260809 12:15:18.324555 22980 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 52c3bdc007c54d99a0646ddc0950d948: Generated new TSK 0
I20260809 12:15:18.325407 22980 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260809 12:15:18.339007 888 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskgH5Pjm/build/asan/bin/kudu pid 22885
I20260809 12:15:18.340694 22955 generic_service.cc:196] Checking for leaks (request via RPC)
I20260809 12:15:18.439450 888 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskgH5Pjm/build/asan/bin/kudu with pid 22885
W20260809 12:15:18.461403 22868 connection.cc:535] client connection to 127.0.0.1:41211 recv error: Network error: failed to read from TLS socket (remote: 127.0.0.1:41211): Connection reset by peer (error 104)
2026-08-09T12:15:18Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/0 (2196 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/1
Loading random data
Initializing database '/tmp/dist-test-taskgH5Pjm/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1786277382073938-888-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 09 12:15:18 dist-test-slave-2w3w krb5kdc[22996](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 09 12:15:18 dist-test-slave-2w3w krb5kdc[22996](info): set up 2 sockets
Aug 09 12:15:18 dist-test-slave-2w3w krb5kdc[22996](info): commencing operation
krb5kdc: starting...
W20260809 12:15:19.608596 888 mini_kdc.cc:121] Time spent starting KDC: real 1.128s user 0.000s sys 0.046s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskgH5Pjm/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1786277382073938-888-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskgH5Pjm/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1786277382073938-888-0/krb5kdc/test-user.keytab.
Aug 09 12:15:19 dist-test-slave-2w3w krb5kdc[22996](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786277719, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-09T12:15:19Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-09T12:15:19Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskgH5Pjm/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1786277382073938-888-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskgH5Pjm/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1786277382073938-888-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260809 12:15:19.938863 888 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskgH5Pjm/build/asan/bin/kudu
/tmp/dist-test-taskgH5Pjm/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskgH5Pjm/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1786277382073938-888-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskgH5Pjm/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1786277382073938-888-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskgH5Pjm/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1786277382073938-888-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskgH5Pjm/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1786277382073938-888-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:41177
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskgH5Pjm/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1786277382073938-888-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:40261
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:41177
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=required
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taskgH5Pjm/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1786277382073938-888-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskgH5Pjm/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1786277382073938-888-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskgH5Pjm/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1786277382073938-888-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260809 12:15:20.283740 23010 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260809 12:15:20.284112 23010 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260809 12:15:20.284282 23010 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260809 12:15:20.291676 23010 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260809 12:15:20.291822 23010 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260809 12:15:20.291935 23010 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260809 12:15:20.292060 23010 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260809 12:15:20.292143 23010 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260809 12:15:20.302325 23010 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:40261
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskgH5Pjm/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1786277382073938-888-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskgH5Pjm/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1786277382073938-888-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:41177
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskgH5Pjm/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1786277382073938-888-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:41177
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskgH5Pjm/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1786277382073938-888-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskgH5Pjm/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1786277382073938-888-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 09 Aug 2026 11:06:41 UTC on ccycloud.quasar-bwoogy.root.comops.site
build id 4575991
ASAN enabled
I20260809 12:15:20.303731 23010 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260809 12:15:20.305112 23010 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260809 12:15:20.316782 23015 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 12:15:20.316956 23016 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 12:15:20.317018 23019 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260809 12:15:20.318055 23010 server_base.cc:1034] running on GCE node
I20260809 12:15:20.318840 23010 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260809 12:15:20.320251 23010 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260809 12:15:20.321503 23010 hybrid_clock.cc:648] HybridClock initialized: now 1786277720321463 us; error 51 us; skew 500 ppm
I20260809 12:15:20.321894 23010 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 09 12:15:20 dist-test-slave-2w3w krb5kdc[22996](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786277720, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260809 12:15:20.330001 23010 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260809 12:15:20.331560 23010 webserver.cc:460] Webserver started at http://127.0.0.1:33119/ using document root <none> and password file <none>
I20260809 12:15:20.332283 23010 fs_manager.cc:359] Metadata directory not provided
I20260809 12:15:20.332436 23010 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260809 12:15:20.332751 23010 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260809 12:15:20.335898 23010 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskgH5Pjm/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1786277382073938-888-0/minicluster-data/master-0/data/instance:
uuid: "3810d8e19dcf4b7486e12a7c35adb432"
format_stamp: "Formatted at 2026-08-09 12:15:20 on dist-test-slave-2w3w"
server_key: "5f23976039627e88a949ea32d3693666"
server_key_iv: "3fb5db361af46266f495b6157c4befd7"
server_key_version: "encryptionkey@0"
I20260809 12:15:20.336691 23010 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskgH5Pjm/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1786277382073938-888-0/minicluster-data/master-0/wal/instance:
uuid: "3810d8e19dcf4b7486e12a7c35adb432"
format_stamp: "Formatted at 2026-08-09 12:15:20 on dist-test-slave-2w3w"
server_key: "5f23976039627e88a949ea32d3693666"
server_key_iv: "3fb5db361af46266f495b6157c4befd7"
server_key_version: "encryptionkey@0"
I20260809 12:15:20.341866 23010 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.000s
I20260809 12:15:20.345388 23027 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260809 12:15:20.346611 23010 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.001s
I20260809 12:15:20.346792 23010 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskgH5Pjm/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1786277382073938-888-0/minicluster-data/master-0/data,/tmp/dist-test-taskgH5Pjm/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1786277382073938-888-0/minicluster-data/master-0/wal
uuid: "3810d8e19dcf4b7486e12a7c35adb432"
format_stamp: "Formatted at 2026-08-09 12:15:20 on dist-test-slave-2w3w"
server_key: "5f23976039627e88a949ea32d3693666"
server_key_iv: "3fb5db361af46266f495b6157c4befd7"
server_key_version: "encryptionkey@0"
I20260809 12:15:20.347023 23010 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskgH5Pjm/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1786277382073938-888-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskgH5Pjm/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1786277382073938-888-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskgH5Pjm/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1786277382073938-888-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260809 12:15:20.363605 23010 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260809 12:15:20.368250 23010 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260809 12:15:20.368543 23010 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260809 12:15:20.387600 23010 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:41177
I20260809 12:15:20.387656 23085 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:41177 every 8 connection(s)
I20260809 12:15:20.389004 23010 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskgH5Pjm/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1786277382073938-888-0/minicluster-data/master-0/data/info.pb
I20260809 12:15:20.390702 888 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskgH5Pjm/build/asan/bin/kudu as pid 23010
I20260809 12:15:20.390903 888 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskgH5Pjm/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1786277382073938-888-0/minicluster-data/master-0/wal/instance
I20260809 12:15:20.391396 888 external_mini_cluster.cc:1428] Setting key 7509bd4a134854a28363c018f9431c4c
I20260809 12:15:20.393277 23086 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260809 12:15:20.406643 23089 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 12:15:20.393270 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48819 (local address 127.0.0.1:41177)
0809 12:15:20.394109 (+ 839us) server_negotiation.cc:206] Beginning negotiation
0809 12:15:20.394125 (+ 16us) server_negotiation.cc:399] Waiting for connection header
0809 12:15:20.394176 (+ 51us) server_negotiation.cc:407] Connection header received
0809 12:15:20.396442 (+ 2266us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 12:15:20.396468 (+ 26us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 12:15:20.396974 (+ 506us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 12:15:20.397444 (+ 470us) server_negotiation.cc:226] Negotiated authn=SASL
0809 12:15:20.397967 (+ 523us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 12:15:20.400062 (+ 2095us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 12:15:20.401458 (+ 1396us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 12:15:20.402447 (+ 989us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 12:15:20.402618 (+ 171us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 12:15:20.403873 (+ 1255us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 12:15:20.403888 (+ 15us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 12:15:20.403904 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0809 12:15:20.404643 (+ 739us) server_negotiation.cc:922] calling sasl_server_start()
0809 12:15:20.404738 (+ 95us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0809 12:15:20.404774 (+ 36us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 12:15:20.404939 (+ 165us) server_negotiation.cc:1035] Waiting for connection context
0809 12:15:20.405197 (+ 258us) server_negotiation.cc:299] Negotiation successful
0809 12:15:20.405385 (+ 188us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":616,"thread_start_us":339,"threads_started":1}
I20260809 12:15:20.408008 23086 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 3810d8e19dcf4b7486e12a7c35adb432: Bootstrap starting.
I20260809 12:15:20.410493 888 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260809 12:15:20.411433 23086 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 3810d8e19dcf4b7486e12a7c35adb432: Neither blocks nor log segments found. Creating new log.
I20260809 12:15:20.412613 23086 log.cc:824] T 00000000000000000000000000000000 P 3810d8e19dcf4b7486e12a7c35adb432: Log is configured to *not* fsync() on all Append() calls
I20260809 12:15:20.416069 23086 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 3810d8e19dcf4b7486e12a7c35adb432: No bootstrap required, opened a new log
I20260809 12:15:20.422676 23089 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 12:15:20.415653 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37257 (local address 127.0.0.1:41177)
0809 12:15:20.415865 (+ 212us) server_negotiation.cc:206] Beginning negotiation
0809 12:15:20.415874 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0809 12:15:20.416390 (+ 516us) server_negotiation.cc:407] Connection header received
0809 12:15:20.416583 (+ 193us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 12:15:20.416587 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 12:15:20.416644 (+ 57us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 12:15:20.416724 (+ 80us) server_negotiation.cc:226] Negotiated authn=SASL
0809 12:15:20.417402 (+ 678us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 12:15:20.418167 (+ 765us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 12:15:20.420137 (+ 1970us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 12:15:20.420588 (+ 451us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 12:15:20.420677 (+ 89us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 12:15:20.421889 (+ 1212us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 12:15:20.421893 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 12:15:20.421896 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0809 12:15:20.421912 (+ 16us) server_negotiation.cc:922] calling sasl_server_start()
0809 12:15:20.421949 (+ 37us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0809 12:15:20.421962 (+ 13us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 12:15:20.422052 (+ 90us) server_negotiation.cc:1035] Waiting for connection context
0809 12:15:20.422251 (+ 199us) server_negotiation.cc:299] Negotiation successful
0809 12:15:20.422293 (+ 42us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":63}
I20260809 12:15:20.425498 23086 raft_consensus.cc:348] T 00000000000000000000000000000000 P 3810d8e19dcf4b7486e12a7c35adb432 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3810d8e19dcf4b7486e12a7c35adb432" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 41177 } }
I20260809 12:15:20.425925 23086 raft_consensus.cc:374] T 00000000000000000000000000000000 P 3810d8e19dcf4b7486e12a7c35adb432 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260809 12:15:20.426045 23086 raft_consensus.cc:729] T 00000000000000000000000000000000 P 3810d8e19dcf4b7486e12a7c35adb432 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3810d8e19dcf4b7486e12a7c35adb432, State: Initialized, Role: FOLLOWER
I20260809 12:15:20.426680 23086 consensus_queue.cc:260] T 00000000000000000000000000000000 P 3810d8e19dcf4b7486e12a7c35adb432 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3810d8e19dcf4b7486e12a7c35adb432" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 41177 } }
I20260809 12:15:20.427011 23086 raft_consensus.cc:388] T 00000000000000000000000000000000 P 3810d8e19dcf4b7486e12a7c35adb432 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260809 12:15:20.427198 23086 raft_consensus.cc:482] T 00000000000000000000000000000000 P 3810d8e19dcf4b7486e12a7c35adb432 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260809 12:15:20.427376 23086 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 3810d8e19dcf4b7486e12a7c35adb432 [term 0 FOLLOWER]: Advancing to term 1
I20260809 12:15:20.429605 23086 raft_consensus.cc:504] T 00000000000000000000000000000000 P 3810d8e19dcf4b7486e12a7c35adb432 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3810d8e19dcf4b7486e12a7c35adb432" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 41177 } }
I20260809 12:15:20.430158 23086 leader_election.cc:302] T 00000000000000000000000000000000 P 3810d8e19dcf4b7486e12a7c35adb432 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 3810d8e19dcf4b7486e12a7c35adb432; no voters:
I20260809 12:15:20.430888 23086 leader_election.cc:288] T 00000000000000000000000000000000 P 3810d8e19dcf4b7486e12a7c35adb432 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260809 12:15:20.431056 23096 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 3810d8e19dcf4b7486e12a7c35adb432 [term 1 FOLLOWER]: Leader election won for term 1
I20260809 12:15:20.432265 23096 raft_consensus.cc:686] T 00000000000000000000000000000000 P 3810d8e19dcf4b7486e12a7c35adb432 [term 1 LEADER]: Becoming Leader. State: Replica: 3810d8e19dcf4b7486e12a7c35adb432, State: Running, Role: LEADER
I20260809 12:15:20.432818 23096 consensus_queue.cc:237] T 00000000000000000000000000000000 P 3810d8e19dcf4b7486e12a7c35adb432 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3810d8e19dcf4b7486e12a7c35adb432" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 41177 } }
I20260809 12:15:20.433365 23086 sys_catalog.cc:564] T 00000000000000000000000000000000 P 3810d8e19dcf4b7486e12a7c35adb432 [sys.catalog]: configured and running, proceeding with master startup.
I20260809 12:15:20.435695 23098 sys_catalog.cc:455] T 00000000000000000000000000000000 P 3810d8e19dcf4b7486e12a7c35adb432 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 3810d8e19dcf4b7486e12a7c35adb432. Latest consensus state: current_term: 1 leader_uuid: "3810d8e19dcf4b7486e12a7c35adb432" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3810d8e19dcf4b7486e12a7c35adb432" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 41177 } } }
I20260809 12:15:20.436157 23098 sys_catalog.cc:458] T 00000000000000000000000000000000 P 3810d8e19dcf4b7486e12a7c35adb432 [sys.catalog]: This master's current role is: LEADER
I20260809 12:15:20.436815 23097 sys_catalog.cc:455] T 00000000000000000000000000000000 P 3810d8e19dcf4b7486e12a7c35adb432 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "3810d8e19dcf4b7486e12a7c35adb432" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3810d8e19dcf4b7486e12a7c35adb432" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 41177 } } }
I20260809 12:15:20.437172 23097 sys_catalog.cc:458] T 00000000000000000000000000000000 P 3810d8e19dcf4b7486e12a7c35adb432 [sys.catalog]: This master's current role is: LEADER
I20260809 12:15:20.438947 23102 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260809 12:15:20.445115 23102 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260809 12:15:20.454432 23102 catalog_manager.cc:1347] Generated new cluster ID: b3e17e4a14fa44bcadf57a5cfa8facae
I20260809 12:15:20.454571 23102 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260809 12:15:20.470494 23102 catalog_manager.cc:1370] Generated new certificate authority record
I20260809 12:15:20.471849 23102 catalog_manager.cc:1504] Loading token signing keys...
I20260809 12:15:20.478721 23102 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 3810d8e19dcf4b7486e12a7c35adb432: Generated new TSK 0
I20260809 12:15:20.479566 23102 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260809 12:15:20.496065 888 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskgH5Pjm/build/asan/bin/kudu pid 23010
W20260809 12:15:20.498099 23080 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='slave'} at 127.0.0.1:48819
W20260809 12:15:20.499032 888 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskgH5Pjm/build/asan/bin/kudu pid 23010: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260809 12:15:20.499269 888 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskgH5Pjm/build/asan/bin/kudu with pid 23010
W20260809 12:15:20.521520 22994 connection.cc:535] client connection to 127.0.0.1:41177 recv error: Network error: failed to read from TLS socket (remote: 127.0.0.1:41177): Connection reset by peer (error 104)
2026-08-09T12:15:20Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/1 (2059 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/2
Loading random data
Initializing database '/tmp/dist-test-taskgH5Pjm/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1786277382073938-888-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 09 12:15:20 dist-test-slave-2w3w krb5kdc[23119](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 09 12:15:20 dist-test-slave-2w3w krb5kdc[23119](info): set up 2 sockets
Aug 09 12:15:20 dist-test-slave-2w3w krb5kdc[23119](info): commencing operation
krb5kdc: starting...
W20260809 12:15:21.671610 888 mini_kdc.cc:121] Time spent starting KDC: real 1.131s user 0.007s sys 0.039s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskgH5Pjm/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1786277382073938-888-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskgH5Pjm/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1786277382073938-888-0/krb5kdc/test-user.keytab.
Aug 09 12:15:21 dist-test-slave-2w3w krb5kdc[23119](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786277721, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-09T12:15:21Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-09T12:15:21Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskgH5Pjm/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1786277382073938-888-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskgH5Pjm/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1786277382073938-888-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260809 12:15:22.008805 888 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskgH5Pjm/build/asan/bin/kudu
/tmp/dist-test-taskgH5Pjm/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskgH5Pjm/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1786277382073938-888-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskgH5Pjm/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1786277382073938-888-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskgH5Pjm/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1786277382073938-888-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskgH5Pjm/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1786277382073938-888-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:38765
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskgH5Pjm/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1786277382073938-888-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:38657
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:38765
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=disabled
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taskgH5Pjm/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1786277382073938-888-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskgH5Pjm/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1786277382073938-888-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskgH5Pjm/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1786277382073938-888-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260809 12:15:22.360734 23133 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260809 12:15:22.361063 23133 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260809 12:15:22.361258 23133 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260809 12:15:22.369155 23133 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260809 12:15:22.369345 23133 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260809 12:15:22.369436 23133 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260809 12:15:22.369557 23133 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260809 12:15:22.369630 23133 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260809 12:15:22.380884 23133 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:38657
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskgH5Pjm/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1786277382073938-888-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskgH5Pjm/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1786277382073938-888-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:38765
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskgH5Pjm/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1786277382073938-888-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:38765
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskgH5Pjm/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1786277382073938-888-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskgH5Pjm/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1786277382073938-888-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 09 Aug 2026 11:06:41 UTC on ccycloud.quasar-bwoogy.root.comops.site
build id 4575991
ASAN enabled
I20260809 12:15:22.382263 23133 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260809 12:15:22.383740 23133 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260809 12:15:22.395964 23142 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 12:15:22.396133 23140 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 12:15:22.396462 23139 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260809 12:15:22.397042 23133 server_base.cc:1034] running on GCE node
I20260809 12:15:22.397872 23133 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260809 12:15:22.399407 23133 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260809 12:15:22.400658 23133 hybrid_clock.cc:648] HybridClock initialized: now 1786277722400614 us; error 38 us; skew 500 ppm
I20260809 12:15:22.401063 23133 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 09 12:15:22 dist-test-slave-2w3w krb5kdc[23119](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786277722, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260809 12:15:22.404716 23133 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260809 12:15:22.406216 23133 webserver.cc:460] Webserver started at http://127.0.0.1:41569/ using document root <none> and password file <none>
I20260809 12:15:22.406890 23133 fs_manager.cc:359] Metadata directory not provided
I20260809 12:15:22.407059 23133 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260809 12:15:22.407364 23133 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260809 12:15:22.410387 23133 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskgH5Pjm/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1786277382073938-888-0/minicluster-data/master-0/data/instance:
uuid: "6e060d29301044b7ba8e366405ac5565"
format_stamp: "Formatted at 2026-08-09 12:15:22 on dist-test-slave-2w3w"
server_key: "5f7a211c191efca8f2168ebbf79cb8fa"
server_key_iv: "c95e415c40f5efc8aed4f97f46e6fa34"
server_key_version: "encryptionkey@0"
I20260809 12:15:22.411206 23133 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskgH5Pjm/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1786277382073938-888-0/minicluster-data/master-0/wal/instance:
uuid: "6e060d29301044b7ba8e366405ac5565"
format_stamp: "Formatted at 2026-08-09 12:15:22 on dist-test-slave-2w3w"
server_key: "5f7a211c191efca8f2168ebbf79cb8fa"
server_key_iv: "c95e415c40f5efc8aed4f97f46e6fa34"
server_key_version: "encryptionkey@0"
I20260809 12:15:22.416501 23133 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.004s sys 0.000s
I20260809 12:15:22.419900 23150 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260809 12:15:22.421281 23133 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260809 12:15:22.421432 23133 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskgH5Pjm/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1786277382073938-888-0/minicluster-data/master-0/data,/tmp/dist-test-taskgH5Pjm/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1786277382073938-888-0/minicluster-data/master-0/wal
uuid: "6e060d29301044b7ba8e366405ac5565"
format_stamp: "Formatted at 2026-08-09 12:15:22 on dist-test-slave-2w3w"
server_key: "5f7a211c191efca8f2168ebbf79cb8fa"
server_key_iv: "c95e415c40f5efc8aed4f97f46e6fa34"
server_key_version: "encryptionkey@0"
I20260809 12:15:22.421614 23133 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskgH5Pjm/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1786277382073938-888-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskgH5Pjm/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1786277382073938-888-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskgH5Pjm/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1786277382073938-888-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260809 12:15:22.427384 23133 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260809 12:15:22.431967 23133 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260809 12:15:22.432243 23133 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260809 12:15:22.450937 23133 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:38765
I20260809 12:15:22.451046 23204 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:38765 every 8 connection(s)
I20260809 12:15:22.452416 23133 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskgH5Pjm/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1786277382073938-888-0/minicluster-data/master-0/data/info.pb
I20260809 12:15:22.456195 23205 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260809 12:15:22.462162 888 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskgH5Pjm/build/asan/bin/kudu as pid 23133
I20260809 12:15:22.462397 888 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskgH5Pjm/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1786277382073938-888-0/minicluster-data/master-0/wal/instance
I20260809 12:15:22.462868 888 external_mini_cluster.cc:1428] Setting key 75500b363334d682d83ca491ddb692d0
I20260809 12:15:22.468091 23205 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 6e060d29301044b7ba8e366405ac5565: Bootstrap starting.
I20260809 12:15:22.472286 23205 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 6e060d29301044b7ba8e366405ac5565: Neither blocks nor log segments found. Creating new log.
I20260809 12:15:22.472460 23209 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 12:15:22.464930 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50073 (local address 127.0.0.1:38765)
0809 12:15:22.465826 (+ 896us) server_negotiation.cc:206] Beginning negotiation
0809 12:15:22.465842 (+ 16us) server_negotiation.cc:399] Waiting for connection header
0809 12:15:22.465891 (+ 49us) server_negotiation.cc:407] Connection header received
0809 12:15:22.468382 (+ 2491us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 12:15:22.468412 (+ 30us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 12:15:22.468857 (+ 445us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 12:15:22.469281 (+ 424us) server_negotiation.cc:226] Negotiated authn=SASL
0809 12:15:22.470378 (+ 1097us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 12:15:22.470387 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 12:15:22.470404 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0809 12:15:22.470434 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0809 12:15:22.470527 (+ 93us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0809 12:15:22.470558 (+ 31us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 12:15:22.470777 (+ 219us) server_negotiation.cc:1035] Waiting for connection context
0809 12:15:22.470906 (+ 129us) server_negotiation.cc:299] Negotiation successful
0809 12:15:22.471137 (+ 231us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":630,"thread_start_us":384,"threads_started":1}
I20260809 12:15:22.473544 23205 log.cc:824] T 00000000000000000000000000000000 P 6e060d29301044b7ba8e366405ac5565: Log is configured to *not* fsync() on all Append() calls
I20260809 12:15:22.476547 888 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260809 12:15:22.477006 23205 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 6e060d29301044b7ba8e366405ac5565: No bootstrap required, opened a new log
I20260809 12:15:22.483837 23209 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 12:15:22.481107 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47523 (local address 127.0.0.1:38765)
0809 12:15:22.481357 (+ 250us) server_negotiation.cc:206] Beginning negotiation
0809 12:15:22.481366 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0809 12:15:22.481803 (+ 437us) server_negotiation.cc:407] Connection header received
0809 12:15:22.481977 (+ 174us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 12:15:22.481981 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 12:15:22.482027 (+ 46us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 12:15:22.482118 (+ 91us) server_negotiation.cc:226] Negotiated authn=SASL
0809 12:15:22.483148 (+ 1030us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 12:15:22.483152 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 12:15:22.483155 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0809 12:15:22.483170 (+ 15us) server_negotiation.cc:922] calling sasl_server_start()
0809 12:15:22.483209 (+ 39us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0809 12:15:22.483224 (+ 15us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 12:15:22.483324 (+ 100us) server_negotiation.cc:1035] Waiting for connection context
0809 12:15:22.483465 (+ 141us) server_negotiation.cc:299] Negotiation successful
0809 12:15:22.483513 (+ 48us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":76}
I20260809 12:15:22.486049 23205 raft_consensus.cc:348] T 00000000000000000000000000000000 P 6e060d29301044b7ba8e366405ac5565 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6e060d29301044b7ba8e366405ac5565" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 38765 } }
I20260809 12:15:22.486466 23205 raft_consensus.cc:374] T 00000000000000000000000000000000 P 6e060d29301044b7ba8e366405ac5565 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260809 12:15:22.486585 23205 raft_consensus.cc:729] T 00000000000000000000000000000000 P 6e060d29301044b7ba8e366405ac5565 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6e060d29301044b7ba8e366405ac5565, State: Initialized, Role: FOLLOWER
I20260809 12:15:22.487346 23205 consensus_queue.cc:260] T 00000000000000000000000000000000 P 6e060d29301044b7ba8e366405ac5565 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6e060d29301044b7ba8e366405ac5565" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 38765 } }
I20260809 12:15:22.487624 23205 raft_consensus.cc:388] T 00000000000000000000000000000000 P 6e060d29301044b7ba8e366405ac5565 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260809 12:15:22.487785 23205 raft_consensus.cc:482] T 00000000000000000000000000000000 P 6e060d29301044b7ba8e366405ac5565 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260809 12:15:22.487967 23205 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 6e060d29301044b7ba8e366405ac5565 [term 0 FOLLOWER]: Advancing to term 1
I20260809 12:15:22.490288 23205 raft_consensus.cc:504] T 00000000000000000000000000000000 P 6e060d29301044b7ba8e366405ac5565 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6e060d29301044b7ba8e366405ac5565" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 38765 } }
I20260809 12:15:22.490809 23205 leader_election.cc:302] T 00000000000000000000000000000000 P 6e060d29301044b7ba8e366405ac5565 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 6e060d29301044b7ba8e366405ac5565; no voters:
I20260809 12:15:22.491423 23205 leader_election.cc:288] T 00000000000000000000000000000000 P 6e060d29301044b7ba8e366405ac5565 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260809 12:15:22.491566 23217 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 6e060d29301044b7ba8e366405ac5565 [term 1 FOLLOWER]: Leader election won for term 1
I20260809 12:15:22.492252 23217 raft_consensus.cc:686] T 00000000000000000000000000000000 P 6e060d29301044b7ba8e366405ac5565 [term 1 LEADER]: Becoming Leader. State: Replica: 6e060d29301044b7ba8e366405ac5565, State: Running, Role: LEADER
I20260809 12:15:22.492789 23217 consensus_queue.cc:237] T 00000000000000000000000000000000 P 6e060d29301044b7ba8e366405ac5565 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6e060d29301044b7ba8e366405ac5565" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 38765 } }
I20260809 12:15:22.493599 23205 sys_catalog.cc:564] T 00000000000000000000000000000000 P 6e060d29301044b7ba8e366405ac5565 [sys.catalog]: configured and running, proceeding with master startup.
I20260809 12:15:22.495903 23218 sys_catalog.cc:455] T 00000000000000000000000000000000 P 6e060d29301044b7ba8e366405ac5565 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "6e060d29301044b7ba8e366405ac5565" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6e060d29301044b7ba8e366405ac5565" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 38765 } } }
I20260809 12:15:22.495877 23219 sys_catalog.cc:455] T 00000000000000000000000000000000 P 6e060d29301044b7ba8e366405ac5565 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 6e060d29301044b7ba8e366405ac5565. Latest consensus state: current_term: 1 leader_uuid: "6e060d29301044b7ba8e366405ac5565" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6e060d29301044b7ba8e366405ac5565" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 38765 } } }
I20260809 12:15:22.496575 23218 sys_catalog.cc:458] T 00000000000000000000000000000000 P 6e060d29301044b7ba8e366405ac5565 [sys.catalog]: This master's current role is: LEADER
I20260809 12:15:22.496706 23219 sys_catalog.cc:458] T 00000000000000000000000000000000 P 6e060d29301044b7ba8e366405ac5565 [sys.catalog]: This master's current role is: LEADER
I20260809 12:15:22.497485 23228 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260809 12:15:22.503099 23228 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260809 12:15:22.512738 23228 catalog_manager.cc:1347] Generated new cluster ID: 6ee0f34915824156877f234a2db1873e
I20260809 12:15:22.512861 23228 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260809 12:15:22.550071 23228 catalog_manager.cc:1370] Generated new certificate authority record
I20260809 12:15:22.550609 23228 catalog_manager.cc:1504] Loading token signing keys...
I20260809 12:15:22.563175 23228 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 6e060d29301044b7ba8e366405ac5565: Generated new TSK 0
I20260809 12:15:22.564215 23228 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260809 12:15:22.574227 888 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskgH5Pjm/build/asan/bin/kudu pid 23133
W20260809 12:15:22.575712 23200 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='slave'} at 127.0.0.1:50073
W20260809 12:15:22.576416 888 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskgH5Pjm/build/asan/bin/kudu pid 23133: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260809 12:15:22.576592 888 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskgH5Pjm/build/asan/bin/kudu with pid 23133
2026-08-09T12:15:22Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/2 (2077 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/3
Loading random data
Initializing database '/tmp/dist-test-taskgH5Pjm/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1786277382073938-888-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 09 12:15:22 dist-test-slave-2w3w krb5kdc[23242](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 09 12:15:22 dist-test-slave-2w3w krb5kdc[23242](info): set up 2 sockets
Aug 09 12:15:22 dist-test-slave-2w3w krb5kdc[23242](info): commencing operation
krb5kdc: starting...
W20260809 12:15:23.756093 888 mini_kdc.cc:121] Time spent starting KDC: real 1.138s user 0.002s sys 0.046s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskgH5Pjm/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1786277382073938-888-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskgH5Pjm/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1786277382073938-888-0/krb5kdc/test-user.keytab.
Aug 09 12:15:23 dist-test-slave-2w3w krb5kdc[23242](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786277723, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-09T12:15:24Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-09T12:15:24Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskgH5Pjm/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1786277382073938-888-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskgH5Pjm/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1786277382073938-888-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260809 12:15:24.088616 888 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskgH5Pjm/build/asan/bin/kudu
/tmp/dist-test-taskgH5Pjm/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskgH5Pjm/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1786277382073938-888-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskgH5Pjm/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1786277382073938-888-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskgH5Pjm/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1786277382073938-888-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskgH5Pjm/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1786277382073938-888-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:33961
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskgH5Pjm/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1786277382073938-888-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:42297
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:33961
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=required
--rpc_encryption=required
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taskgH5Pjm/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1786277382073938-888-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskgH5Pjm/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1786277382073938-888-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskgH5Pjm/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1786277382073938-888-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260809 12:15:24.435423 23255 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260809 12:15:24.435730 23255 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260809 12:15:24.435909 23255 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260809 12:15:24.443394 23255 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260809 12:15:24.443508 23255 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260809 12:15:24.443624 23255 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260809 12:15:24.443735 23255 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260809 12:15:24.443828 23255 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260809 12:15:24.453914 23255 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:42297
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskgH5Pjm/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1786277382073938-888-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskgH5Pjm/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1786277382073938-888-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:33961
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskgH5Pjm/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1786277382073938-888-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:33961
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskgH5Pjm/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1786277382073938-888-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskgH5Pjm/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1786277382073938-888-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 09 Aug 2026 11:06:41 UTC on ccycloud.quasar-bwoogy.root.comops.site
build id 4575991
ASAN enabled
I20260809 12:15:24.455289 23255 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260809 12:15:24.456619 23255 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260809 12:15:24.468777 23262 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 12:15:24.468813 23264 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 12:15:24.468750 23260 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260809 12:15:24.470036 23255 server_base.cc:1034] running on GCE node
I20260809 12:15:24.470870 23255 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260809 12:15:24.472363 23255 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260809 12:15:24.473595 23255 hybrid_clock.cc:648] HybridClock initialized: now 1786277724473564 us; error 53 us; skew 500 ppm
I20260809 12:15:24.473950 23255 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 09 12:15:24 dist-test-slave-2w3w krb5kdc[23242](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786277724, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260809 12:15:24.477121 23255 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260809 12:15:24.478525 23255 webserver.cc:460] Webserver started at http://127.0.0.1:40547/ using document root <none> and password file <none>
I20260809 12:15:24.479238 23255 fs_manager.cc:359] Metadata directory not provided
I20260809 12:15:24.479352 23255 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260809 12:15:24.479619 23255 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260809 12:15:24.482537 23255 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskgH5Pjm/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1786277382073938-888-0/minicluster-data/master-0/data/instance:
uuid: "5c93eb746ca9418194751ffaa1980df3"
format_stamp: "Formatted at 2026-08-09 12:15:24 on dist-test-slave-2w3w"
server_key: "cc8297ad0d7e8363ee202a670c38a6f9"
server_key_iv: "70c61334200815b163451453dc7db168"
server_key_version: "encryptionkey@0"
I20260809 12:15:24.483327 23255 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskgH5Pjm/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1786277382073938-888-0/minicluster-data/master-0/wal/instance:
uuid: "5c93eb746ca9418194751ffaa1980df3"
format_stamp: "Formatted at 2026-08-09 12:15:24 on dist-test-slave-2w3w"
server_key: "cc8297ad0d7e8363ee202a670c38a6f9"
server_key_iv: "70c61334200815b163451453dc7db168"
server_key_version: "encryptionkey@0"
I20260809 12:15:24.488441 23255 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.000s sys 0.004s
I20260809 12:15:24.491770 23271 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260809 12:15:24.493064 23255 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260809 12:15:24.493255 23255 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskgH5Pjm/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1786277382073938-888-0/minicluster-data/master-0/data,/tmp/dist-test-taskgH5Pjm/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1786277382073938-888-0/minicluster-data/master-0/wal
uuid: "5c93eb746ca9418194751ffaa1980df3"
format_stamp: "Formatted at 2026-08-09 12:15:24 on dist-test-slave-2w3w"
server_key: "cc8297ad0d7e8363ee202a670c38a6f9"
server_key_iv: "70c61334200815b163451453dc7db168"
server_key_version: "encryptionkey@0"
I20260809 12:15:24.493460 23255 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskgH5Pjm/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1786277382073938-888-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskgH5Pjm/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1786277382073938-888-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskgH5Pjm/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1786277382073938-888-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260809 12:15:24.515990 23255 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260809 12:15:24.520128 23255 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260809 12:15:24.520439 23255 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260809 12:15:24.539985 23255 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:33961
I20260809 12:15:24.540031 23327 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:33961 every 8 connection(s)
I20260809 12:15:24.541419 23255 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskgH5Pjm/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1786277382073938-888-0/minicluster-data/master-0/data/info.pb
I20260809 12:15:24.545207 23328 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260809 12:15:24.549767 888 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskgH5Pjm/build/asan/bin/kudu as pid 23255
I20260809 12:15:24.549996 888 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskgH5Pjm/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1786277382073938-888-0/minicluster-data/master-0/wal/instance
I20260809 12:15:24.550552 888 external_mini_cluster.cc:1428] Setting key e6a8bd872754a949c40a004d26128cd3
I20260809 12:15:24.557195 23328 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 5c93eb746ca9418194751ffaa1980df3: Bootstrap starting.
I20260809 12:15:24.561007 23328 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 5c93eb746ca9418194751ffaa1980df3: Neither blocks nor log segments found. Creating new log.
I20260809 12:15:24.562211 23328 log.cc:824] T 00000000000000000000000000000000 P 5c93eb746ca9418194751ffaa1980df3: Log is configured to *not* fsync() on all Append() calls
I20260809 12:15:24.565722 23328 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 5c93eb746ca9418194751ffaa1980df3: No bootstrap required, opened a new log
Aug 09 12:15:24 dist-test-slave-2w3w krb5kdc[23242](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786277723, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.0.1@KRBTEST.COM
I20260809 12:15:24.573988 23328 raft_consensus.cc:348] T 00000000000000000000000000000000 P 5c93eb746ca9418194751ffaa1980df3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5c93eb746ca9418194751ffaa1980df3" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 33961 } }
I20260809 12:15:24.574522 23328 raft_consensus.cc:374] T 00000000000000000000000000000000 P 5c93eb746ca9418194751ffaa1980df3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260809 12:15:24.574836 23328 raft_consensus.cc:729] T 00000000000000000000000000000000 P 5c93eb746ca9418194751ffaa1980df3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5c93eb746ca9418194751ffaa1980df3, State: Initialized, Role: FOLLOWER
I20260809 12:15:24.575490 23331 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 12:15:24.552469 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56297 (local address 127.0.0.1:33961)
0809 12:15:24.553237 (+ 768us) server_negotiation.cc:206] Beginning negotiation
0809 12:15:24.553249 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0809 12:15:24.553290 (+ 41us) server_negotiation.cc:407] Connection header received
0809 12:15:24.555608 (+ 2318us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 12:15:24.555634 (+ 26us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 12:15:24.556045 (+ 411us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 12:15:24.556403 (+ 358us) server_negotiation.cc:226] Negotiated authn=SASL
0809 12:15:24.558314 (+ 1911us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 12:15:24.559764 (+ 1450us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 12:15:24.561638 (+ 1874us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 12:15:24.562438 (+ 800us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 12:15:24.562577 (+ 139us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 12:15:24.567654 (+ 5077us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 12:15:24.567662 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 12:15:24.567678 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0809 12:15:24.567719 (+ 41us) server_negotiation.cc:922] calling sasl_server_start()
0809 12:15:24.571168 (+ 3449us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 12:15:24.571904 (+ 736us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 12:15:24.571911 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 12:15:24.571917 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0809 12:15:24.572087 (+ 170us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 12:15:24.572496 (+ 409us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 12:15:24.572501 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 12:15:24.572505 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0809 12:15:24.573056 (+ 551us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 12:15:24.573367 (+ 311us) server_negotiation.cc:1035] Waiting for connection context
0809 12:15:24.573824 (+ 457us) server_negotiation.cc:299] Negotiation successful
0809 12:15:24.574268 (+ 444us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":551,"thread_start_us":345,"threads_started":1}
I20260809 12:15:24.575533 23328 consensus_queue.cc:260] T 00000000000000000000000000000000 P 5c93eb746ca9418194751ffaa1980df3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5c93eb746ca9418194751ffaa1980df3" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 33961 } }
I20260809 12:15:24.575883 23328 raft_consensus.cc:388] T 00000000000000000000000000000000 P 5c93eb746ca9418194751ffaa1980df3 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260809 12:15:24.576046 23328 raft_consensus.cc:482] T 00000000000000000000000000000000 P 5c93eb746ca9418194751ffaa1980df3 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260809 12:15:24.576244 23328 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 5c93eb746ca9418194751ffaa1980df3 [term 0 FOLLOWER]: Advancing to term 1
I20260809 12:15:24.578913 23328 raft_consensus.cc:504] T 00000000000000000000000000000000 P 5c93eb746ca9418194751ffaa1980df3 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5c93eb746ca9418194751ffaa1980df3" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 33961 } }
I20260809 12:15:24.579339 888 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260809 12:15:24.579516 23328 leader_election.cc:302] T 00000000000000000000000000000000 P 5c93eb746ca9418194751ffaa1980df3 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 5c93eb746ca9418194751ffaa1980df3; no voters:
I20260809 12:15:24.580189 23328 leader_election.cc:288] T 00000000000000000000000000000000 P 5c93eb746ca9418194751ffaa1980df3 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260809 12:15:24.580299 23333 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 5c93eb746ca9418194751ffaa1980df3 [term 1 FOLLOWER]: Leader election won for term 1
I20260809 12:15:24.582710 23328 sys_catalog.cc:564] T 00000000000000000000000000000000 P 5c93eb746ca9418194751ffaa1980df3 [sys.catalog]: configured and running, proceeding with master startup.
I20260809 12:15:24.587328 23333 raft_consensus.cc:686] T 00000000000000000000000000000000 P 5c93eb746ca9418194751ffaa1980df3 [term 1 LEADER]: Becoming Leader. State: Replica: 5c93eb746ca9418194751ffaa1980df3, State: Running, Role: LEADER
I20260809 12:15:24.588243 23333 consensus_queue.cc:237] T 00000000000000000000000000000000 P 5c93eb746ca9418194751ffaa1980df3 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5c93eb746ca9418194751ffaa1980df3" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 33961 } }
I20260809 12:15:24.591387 23334 sys_catalog.cc:455] T 00000000000000000000000000000000 P 5c93eb746ca9418194751ffaa1980df3 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "5c93eb746ca9418194751ffaa1980df3" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5c93eb746ca9418194751ffaa1980df3" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 33961 } } }
I20260809 12:15:24.591892 23334 sys_catalog.cc:458] T 00000000000000000000000000000000 P 5c93eb746ca9418194751ffaa1980df3 [sys.catalog]: This master's current role is: LEADER
I20260809 12:15:24.591985 23335 sys_catalog.cc:455] T 00000000000000000000000000000000 P 5c93eb746ca9418194751ffaa1980df3 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 5c93eb746ca9418194751ffaa1980df3. Latest consensus state: current_term: 1 leader_uuid: "5c93eb746ca9418194751ffaa1980df3" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5c93eb746ca9418194751ffaa1980df3" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 33961 } } }
I20260809 12:15:24.592276 23335 sys_catalog.cc:458] T 00000000000000000000000000000000 P 5c93eb746ca9418194751ffaa1980df3 [sys.catalog]: This master's current role is: LEADER
I20260809 12:15:24.605336 23350 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260809 12:15:24.606288 23331 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 12:15:24.584525 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37141 (local address 127.0.0.1:33961)
0809 12:15:24.584722 (+ 197us) server_negotiation.cc:206] Beginning negotiation
0809 12:15:24.584730 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0809 12:15:24.585289 (+ 559us) server_negotiation.cc:407] Connection header received
0809 12:15:24.585590 (+ 301us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 12:15:24.585594 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 12:15:24.585651 (+ 57us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 12:15:24.585754 (+ 103us) server_negotiation.cc:226] Negotiated authn=SASL
0809 12:15:24.587445 (+ 1691us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 12:15:24.589003 (+ 1558us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 12:15:24.590818 (+ 1815us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 12:15:24.591535 (+ 717us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 12:15:24.591670 (+ 135us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 12:15:24.596695 (+ 5025us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 12:15:24.596706 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 12:15:24.596714 (+ 8us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0809 12:15:24.596768 (+ 54us) server_negotiation.cc:922] calling sasl_server_start()
0809 12:15:24.601551 (+ 4783us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 12:15:24.602753 (+ 1202us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 12:15:24.602761 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 12:15:24.602767 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0809 12:15:24.602949 (+ 182us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 12:15:24.603557 (+ 608us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 12:15:24.603565 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 12:15:24.603571 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0809 12:15:24.604081 (+ 510us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 12:15:24.605750 (+ 1669us) server_negotiation.cc:1035] Waiting for connection context
0809 12:15:24.605871 (+ 121us) server_negotiation.cc:299] Negotiation successful
0809 12:15:24.606041 (+ 170us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":61}
I20260809 12:15:24.612201 23350 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260809 12:15:24.621728 23350 catalog_manager.cc:1347] Generated new cluster ID: d372531436324f4299c743d7671db174
I20260809 12:15:24.621860 23350 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260809 12:15:24.630164 23350 catalog_manager.cc:1370] Generated new certificate authority record
I20260809 12:15:24.631508 23350 catalog_manager.cc:1504] Loading token signing keys...
I20260809 12:15:24.638772 23350 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 5c93eb746ca9418194751ffaa1980df3: Generated new TSK 0
I20260809 12:15:24.639588 23350 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260809 12:15:24.649132 888 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskgH5Pjm/build/asan/bin/kudu pid 23255
I20260809 12:15:24.650134 23321 generic_service.cc:196] Checking for leaks (request via RPC)
I20260809 12:15:24.741231 888 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskgH5Pjm/build/asan/bin/kudu with pid 23255
2026-08-09T12:15:24Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/3 (2164 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/4
Loading random data
Initializing database '/tmp/dist-test-taskgH5Pjm/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1786277382073938-888-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 09 12:15:24 dist-test-slave-2w3w krb5kdc[23364](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 09 12:15:24 dist-test-slave-2w3w krb5kdc[23364](info): set up 2 sockets
Aug 09 12:15:24 dist-test-slave-2w3w krb5kdc[23364](info): commencing operation
krb5kdc: starting...
W20260809 12:15:25.926172 888 mini_kdc.cc:121] Time spent starting KDC: real 1.144s user 0.000s sys 0.049s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskgH5Pjm/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1786277382073938-888-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskgH5Pjm/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1786277382073938-888-0/krb5kdc/test-user.keytab.
Aug 09 12:15:26 dist-test-slave-2w3w krb5kdc[23364](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786277726, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-09T12:15:26Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-09T12:15:26Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskgH5Pjm/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1786277382073938-888-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskgH5Pjm/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1786277382073938-888-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260809 12:15:26.263235 888 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskgH5Pjm/build/asan/bin/kudu
/tmp/dist-test-taskgH5Pjm/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskgH5Pjm/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1786277382073938-888-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskgH5Pjm/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1786277382073938-888-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskgH5Pjm/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1786277382073938-888-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskgH5Pjm/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1786277382073938-888-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:38173
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskgH5Pjm/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1786277382073938-888-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:35187
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:38173
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=required
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taskgH5Pjm/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1786277382073938-888-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskgH5Pjm/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1786277382073938-888-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskgH5Pjm/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1786277382073938-888-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260809 12:15:26.621850 23377 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260809 12:15:26.622144 23377 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260809 12:15:26.622325 23377 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260809 12:15:26.629937 23377 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260809 12:15:26.630096 23377 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260809 12:15:26.630204 23377 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260809 12:15:26.630292 23377 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260809 12:15:26.630376 23377 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260809 12:15:26.640627 23377 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:35187
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskgH5Pjm/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1786277382073938-888-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskgH5Pjm/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1786277382073938-888-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:38173
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskgH5Pjm/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1786277382073938-888-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:38173
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskgH5Pjm/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1786277382073938-888-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskgH5Pjm/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1786277382073938-888-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 09 Aug 2026 11:06:41 UTC on ccycloud.quasar-bwoogy.root.comops.site
build id 4575991
ASAN enabled
I20260809 12:15:26.642009 23377 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260809 12:15:26.643446 23377 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260809 12:15:26.656611 23383 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 12:15:26.656548 23384 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 12:15:26.656632 23386 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260809 12:15:26.657446 23377 server_base.cc:1034] running on GCE node
I20260809 12:15:26.658295 23377 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260809 12:15:26.659874 23377 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260809 12:15:26.661126 23377 hybrid_clock.cc:648] HybridClock initialized: now 1786277726661067 us; error 40 us; skew 500 ppm
I20260809 12:15:26.661479 23377 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 09 12:15:26 dist-test-slave-2w3w krb5kdc[23364](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786277726, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260809 12:15:26.668771 23377 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260809 12:15:26.670303 23377 webserver.cc:460] Webserver started at http://127.0.0.1:35033/ using document root <none> and password file <none>
I20260809 12:15:26.671039 23377 fs_manager.cc:359] Metadata directory not provided
I20260809 12:15:26.671162 23377 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260809 12:15:26.671442 23377 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260809 12:15:26.674470 23377 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskgH5Pjm/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1786277382073938-888-0/minicluster-data/master-0/data/instance:
uuid: "8fba726abddd4727b3498623cb8b5bc8"
format_stamp: "Formatted at 2026-08-09 12:15:26 on dist-test-slave-2w3w"
server_key: "8a6f78188fde80432a5c4f232c6bc2ce"
server_key_iv: "9cbec9b3036f576493133bf01000452b"
server_key_version: "encryptionkey@0"
I20260809 12:15:26.675307 23377 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskgH5Pjm/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1786277382073938-888-0/minicluster-data/master-0/wal/instance:
uuid: "8fba726abddd4727b3498623cb8b5bc8"
format_stamp: "Formatted at 2026-08-09 12:15:26 on dist-test-slave-2w3w"
server_key: "8a6f78188fde80432a5c4f232c6bc2ce"
server_key_iv: "9cbec9b3036f576493133bf01000452b"
server_key_version: "encryptionkey@0"
I20260809 12:15:26.680552 23377 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.002s sys 0.003s
I20260809 12:15:26.684296 23393 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260809 12:15:26.685828 23377 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.001s
I20260809 12:15:26.686165 23377 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskgH5Pjm/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1786277382073938-888-0/minicluster-data/master-0/data,/tmp/dist-test-taskgH5Pjm/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1786277382073938-888-0/minicluster-data/master-0/wal
uuid: "8fba726abddd4727b3498623cb8b5bc8"
format_stamp: "Formatted at 2026-08-09 12:15:26 on dist-test-slave-2w3w"
server_key: "8a6f78188fde80432a5c4f232c6bc2ce"
server_key_iv: "9cbec9b3036f576493133bf01000452b"
server_key_version: "encryptionkey@0"
I20260809 12:15:26.686414 23377 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskgH5Pjm/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1786277382073938-888-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskgH5Pjm/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1786277382073938-888-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskgH5Pjm/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1786277382073938-888-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260809 12:15:26.702749 23377 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260809 12:15:26.706876 23377 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260809 12:15:26.707216 23377 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260809 12:15:26.726651 23377 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:38173
I20260809 12:15:26.726684 23447 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:38173 every 8 connection(s)
I20260809 12:15:26.728220 23377 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskgH5Pjm/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1786277382073938-888-0/minicluster-data/master-0/data/info.pb
I20260809 12:15:26.732108 23448 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260809 12:15:26.737838 888 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskgH5Pjm/build/asan/bin/kudu as pid 23377
I20260809 12:15:26.738024 888 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskgH5Pjm/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1786277382073938-888-0/minicluster-data/master-0/wal/instance
I20260809 12:15:26.738511 888 external_mini_cluster.cc:1428] Setting key a0455232a5f4aa69007665090641e8e4
I20260809 12:15:26.743162 23448 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 8fba726abddd4727b3498623cb8b5bc8: Bootstrap starting.
I20260809 12:15:26.746776 23448 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 8fba726abddd4727b3498623cb8b5bc8: Neither blocks nor log segments found. Creating new log.
I20260809 12:15:26.748097 23448 log.cc:824] T 00000000000000000000000000000000 P 8fba726abddd4727b3498623cb8b5bc8: Log is configured to *not* fsync() on all Append() calls
I20260809 12:15:26.751488 23448 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 8fba726abddd4727b3498623cb8b5bc8: No bootstrap required, opened a new log
I20260809 12:15:26.752478 23451 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 12:15:26.740352 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38549 (local address 127.0.0.1:38173)
0809 12:15:26.741219 (+ 867us) server_negotiation.cc:206] Beginning negotiation
0809 12:15:26.741230 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0809 12:15:26.741278 (+ 48us) server_negotiation.cc:407] Connection header received
0809 12:15:26.743094 (+ 1816us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 12:15:26.743127 (+ 33us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 12:15:26.743545 (+ 418us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 12:15:26.743874 (+ 329us) server_negotiation.cc:226] Negotiated authn=SASL
0809 12:15:26.744474 (+ 600us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 12:15:26.745824 (+ 1350us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 12:15:26.747846 (+ 2022us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 12:15:26.748667 (+ 821us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 12:15:26.748894 (+ 227us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 12:15:26.750028 (+ 1134us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 12:15:26.750035 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 12:15:26.750048 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0809 12:15:26.750560 (+ 512us) server_negotiation.cc:922] calling sasl_server_start()
0809 12:15:26.750628 (+ 68us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0809 12:15:26.750655 (+ 27us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 12:15:26.750742 (+ 87us) server_negotiation.cc:1035] Waiting for connection context
0809 12:15:26.751132 (+ 390us) server_negotiation.cc:299] Negotiation successful
0809 12:15:26.751430 (+ 298us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":644,"thread_start_us":431,"threads_started":1}
I20260809 12:15:26.756322 888 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260809 12:15:26.761402 23448 raft_consensus.cc:348] T 00000000000000000000000000000000 P 8fba726abddd4727b3498623cb8b5bc8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8fba726abddd4727b3498623cb8b5bc8" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 38173 } }
I20260809 12:15:26.761933 23448 raft_consensus.cc:374] T 00000000000000000000000000000000 P 8fba726abddd4727b3498623cb8b5bc8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260809 12:15:26.762084 23448 raft_consensus.cc:729] T 00000000000000000000000000000000 P 8fba726abddd4727b3498623cb8b5bc8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8fba726abddd4727b3498623cb8b5bc8, State: Initialized, Role: FOLLOWER
I20260809 12:15:26.762804 23448 consensus_queue.cc:260] T 00000000000000000000000000000000 P 8fba726abddd4727b3498623cb8b5bc8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8fba726abddd4727b3498623cb8b5bc8" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 38173 } }
I20260809 12:15:26.763156 23448 raft_consensus.cc:388] T 00000000000000000000000000000000 P 8fba726abddd4727b3498623cb8b5bc8 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260809 12:15:26.763345 23448 raft_consensus.cc:482] T 00000000000000000000000000000000 P 8fba726abddd4727b3498623cb8b5bc8 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260809 12:15:26.763556 23448 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 8fba726abddd4727b3498623cb8b5bc8 [term 0 FOLLOWER]: Advancing to term 1
I20260809 12:15:26.766794 23448 raft_consensus.cc:504] T 00000000000000000000000000000000 P 8fba726abddd4727b3498623cb8b5bc8 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8fba726abddd4727b3498623cb8b5bc8" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 38173 } }
I20260809 12:15:26.767452 23448 leader_election.cc:302] T 00000000000000000000000000000000 P 8fba726abddd4727b3498623cb8b5bc8 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 8fba726abddd4727b3498623cb8b5bc8; no voters:
I20260809 12:15:26.768229 23448 leader_election.cc:288] T 00000000000000000000000000000000 P 8fba726abddd4727b3498623cb8b5bc8 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260809 12:15:26.768378 23459 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 8fba726abddd4727b3498623cb8b5bc8 [term 1 FOLLOWER]: Leader election won for term 1
I20260809 12:15:26.769232 23451 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 12:15:26.761691 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44417 (local address 127.0.0.1:38173)
0809 12:15:26.761893 (+ 202us) server_negotiation.cc:206] Beginning negotiation
0809 12:15:26.761902 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0809 12:15:26.762488 (+ 586us) server_negotiation.cc:407] Connection header received
0809 12:15:26.762738 (+ 250us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 12:15:26.762753 (+ 15us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 12:15:26.762822 (+ 69us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 12:15:26.762908 (+ 86us) server_negotiation.cc:226] Negotiated authn=SASL
0809 12:15:26.763627 (+ 719us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 12:15:26.764464 (+ 837us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 12:15:26.766371 (+ 1907us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 12:15:26.766829 (+ 458us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 12:15:26.766923 (+ 94us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 12:15:26.768281 (+ 1358us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 12:15:26.768286 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 12:15:26.768291 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0809 12:15:26.768312 (+ 21us) server_negotiation.cc:922] calling sasl_server_start()
0809 12:15:26.768367 (+ 55us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0809 12:15:26.768392 (+ 25us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 12:15:26.768498 (+ 106us) server_negotiation.cc:1035] Waiting for connection context
0809 12:15:26.768769 (+ 271us) server_negotiation.cc:299] Negotiation successful
0809 12:15:26.768877 (+ 108us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":73}
I20260809 12:15:26.770089 23459 raft_consensus.cc:686] T 00000000000000000000000000000000 P 8fba726abddd4727b3498623cb8b5bc8 [term 1 LEADER]: Becoming Leader. State: Replica: 8fba726abddd4727b3498623cb8b5bc8, State: Running, Role: LEADER
I20260809 12:15:26.770725 23448 sys_catalog.cc:564] T 00000000000000000000000000000000 P 8fba726abddd4727b3498623cb8b5bc8 [sys.catalog]: configured and running, proceeding with master startup.
I20260809 12:15:26.770699 23459 consensus_queue.cc:237] T 00000000000000000000000000000000 P 8fba726abddd4727b3498623cb8b5bc8 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8fba726abddd4727b3498623cb8b5bc8" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 38173 } }
I20260809 12:15:26.773934 23460 sys_catalog.cc:455] T 00000000000000000000000000000000 P 8fba726abddd4727b3498623cb8b5bc8 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "8fba726abddd4727b3498623cb8b5bc8" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8fba726abddd4727b3498623cb8b5bc8" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 38173 } } }
I20260809 12:15:26.774443 23460 sys_catalog.cc:458] T 00000000000000000000000000000000 P 8fba726abddd4727b3498623cb8b5bc8 [sys.catalog]: This master's current role is: LEADER
I20260809 12:15:26.774935 23461 sys_catalog.cc:455] T 00000000000000000000000000000000 P 8fba726abddd4727b3498623cb8b5bc8 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 8fba726abddd4727b3498623cb8b5bc8. Latest consensus state: current_term: 1 leader_uuid: "8fba726abddd4727b3498623cb8b5bc8" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8fba726abddd4727b3498623cb8b5bc8" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 38173 } } }
I20260809 12:15:26.775424 23461 sys_catalog.cc:458] T 00000000000000000000000000000000 P 8fba726abddd4727b3498623cb8b5bc8 [sys.catalog]: This master's current role is: LEADER
I20260809 12:15:26.776810 23469 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260809 12:15:26.781311 23469 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260809 12:15:26.790632 23469 catalog_manager.cc:1347] Generated new cluster ID: b5bd4cc633c2487b94f7685730d0e5db
I20260809 12:15:26.790764 23469 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260809 12:15:26.821962 23469 catalog_manager.cc:1370] Generated new certificate authority record
I20260809 12:15:26.823995 23469 catalog_manager.cc:1504] Loading token signing keys...
I20260809 12:15:26.831648 23469 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 8fba726abddd4727b3498623cb8b5bc8: Generated new TSK 0
I20260809 12:15:26.832715 23469 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260809 12:15:26.840276 888 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskgH5Pjm/build/asan/bin/kudu pid 23377
W20260809 12:15:26.841521 23442 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='slave'} at 127.0.0.1:38549
W20260809 12:15:26.842276 888 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskgH5Pjm/build/asan/bin/kudu pid 23377: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260809 12:15:26.842440 888 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskgH5Pjm/build/asan/bin/kudu with pid 23377
2026-08-09T12:15:26Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/4 (2104 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/5
Loading random data
Initializing database '/tmp/dist-test-taskgH5Pjm/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1786277382073938-888-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 09 12:15:26 dist-test-slave-2w3w krb5kdc[23483](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 09 12:15:26 dist-test-slave-2w3w krb5kdc[23483](info): set up 2 sockets
Aug 09 12:15:26 dist-test-slave-2w3w krb5kdc[23483](info): commencing operation
krb5kdc: starting...
W20260809 12:15:28.052418 888 mini_kdc.cc:121] Time spent starting KDC: real 1.166s user 0.006s sys 0.043s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskgH5Pjm/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1786277382073938-888-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskgH5Pjm/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1786277382073938-888-0/krb5kdc/test-user.keytab.
Aug 09 12:15:28 dist-test-slave-2w3w krb5kdc[23483](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786277728, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-09T12:15:28Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-09T12:15:28Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskgH5Pjm/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1786277382073938-888-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskgH5Pjm/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1786277382073938-888-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260809 12:15:28.394270 888 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskgH5Pjm/build/asan/bin/kudu
/tmp/dist-test-taskgH5Pjm/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskgH5Pjm/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1786277382073938-888-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskgH5Pjm/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1786277382073938-888-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskgH5Pjm/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1786277382073938-888-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskgH5Pjm/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1786277382073938-888-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:37879
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskgH5Pjm/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1786277382073938-888-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:34923
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:37879
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=disabled
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taskgH5Pjm/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1786277382073938-888-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskgH5Pjm/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1786277382073938-888-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskgH5Pjm/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1786277382073938-888-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260809 12:15:28.747974 23499 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260809 12:15:28.748329 23499 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260809 12:15:28.748510 23499 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260809 12:15:28.755764 23499 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260809 12:15:28.755926 23499 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260809 12:15:28.756048 23499 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260809 12:15:28.756170 23499 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260809 12:15:28.756254 23499 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260809 12:15:28.766250 23499 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:34923
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskgH5Pjm/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1786277382073938-888-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskgH5Pjm/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1786277382073938-888-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:37879
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskgH5Pjm/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1786277382073938-888-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:37879
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskgH5Pjm/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1786277382073938-888-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskgH5Pjm/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1786277382073938-888-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 09 Aug 2026 11:06:41 UTC on ccycloud.quasar-bwoogy.root.comops.site
build id 4575991
ASAN enabled
I20260809 12:15:28.767725 23499 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260809 12:15:28.769047 23499 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260809 12:15:28.780908 23505 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 12:15:28.780947 23506 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 12:15:28.780987 23509 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260809 12:15:28.782130 23499 server_base.cc:1034] running on GCE node
I20260809 12:15:28.782922 23499 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260809 12:15:28.784359 23499 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260809 12:15:28.785593 23499 hybrid_clock.cc:648] HybridClock initialized: now 1786277728785544 us; error 65 us; skew 500 ppm
I20260809 12:15:28.785944 23499 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 09 12:15:28 dist-test-slave-2w3w krb5kdc[23483](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786277728, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260809 12:15:28.789250 23499 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260809 12:15:28.790685 23499 webserver.cc:460] Webserver started at http://127.0.0.1:35219/ using document root <none> and password file <none>
I20260809 12:15:28.791426 23499 fs_manager.cc:359] Metadata directory not provided
I20260809 12:15:28.791592 23499 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260809 12:15:28.791909 23499 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260809 12:15:28.794916 23499 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskgH5Pjm/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1786277382073938-888-0/minicluster-data/master-0/data/instance:
uuid: "c0f5735177b144a1a63a809c37b531f8"
format_stamp: "Formatted at 2026-08-09 12:15:28 on dist-test-slave-2w3w"
server_key: "2fd42973bcf54718f7fc6449d618238e"
server_key_iv: "56b6c7e59dd88a840a67c2d639b9b7fb"
server_key_version: "encryptionkey@0"
I20260809 12:15:28.795779 23499 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskgH5Pjm/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1786277382073938-888-0/minicluster-data/master-0/wal/instance:
uuid: "c0f5735177b144a1a63a809c37b531f8"
format_stamp: "Formatted at 2026-08-09 12:15:28 on dist-test-slave-2w3w"
server_key: "2fd42973bcf54718f7fc6449d618238e"
server_key_iv: "56b6c7e59dd88a840a67c2d639b9b7fb"
server_key_version: "encryptionkey@0"
I20260809 12:15:28.801229 23499 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.000s
I20260809 12:15:28.804844 23517 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260809 12:15:28.806144 23499 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.000s
I20260809 12:15:28.806336 23499 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskgH5Pjm/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1786277382073938-888-0/minicluster-data/master-0/data,/tmp/dist-test-taskgH5Pjm/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1786277382073938-888-0/minicluster-data/master-0/wal
uuid: "c0f5735177b144a1a63a809c37b531f8"
format_stamp: "Formatted at 2026-08-09 12:15:28 on dist-test-slave-2w3w"
server_key: "2fd42973bcf54718f7fc6449d618238e"
server_key_iv: "56b6c7e59dd88a840a67c2d639b9b7fb"
server_key_version: "encryptionkey@0"
I20260809 12:15:28.806540 23499 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskgH5Pjm/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1786277382073938-888-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskgH5Pjm/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1786277382073938-888-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskgH5Pjm/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1786277382073938-888-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260809 12:15:28.812469 23499 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260809 12:15:28.816836 23499 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260809 12:15:28.817142 23499 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260809 12:15:28.836277 23499 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:37879
I20260809 12:15:28.836364 23573 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:37879 every 8 connection(s)
I20260809 12:15:28.837710 23499 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskgH5Pjm/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1786277382073938-888-0/minicluster-data/master-0/data/info.pb
I20260809 12:15:28.839089 888 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskgH5Pjm/build/asan/bin/kudu as pid 23499
I20260809 12:15:28.839280 888 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskgH5Pjm/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1786277382073938-888-0/minicluster-data/master-0/wal/instance
I20260809 12:15:28.839857 888 external_mini_cluster.cc:1428] Setting key 05fe035996df6d32ddd64e63fc3209a4
I20260809 12:15:28.842514 23574 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260809 12:15:28.847784 23577 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 12:15:28.841759 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50895 (local address 127.0.0.1:37879)
0809 12:15:28.842549 (+ 790us) server_negotiation.cc:206] Beginning negotiation
0809 12:15:28.842585 (+ 36us) server_negotiation.cc:399] Waiting for connection header
0809 12:15:28.842638 (+ 53us) server_negotiation.cc:407] Connection header received
0809 12:15:28.844342 (+ 1704us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 12:15:28.844363 (+ 21us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 12:15:28.844675 (+ 312us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 12:15:28.845067 (+ 392us) server_negotiation.cc:226] Negotiated authn=SASL
0809 12:15:28.846123 (+ 1056us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 12:15:28.846129 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 12:15:28.846140 (+ 11us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0809 12:15:28.846154 (+ 14us) server_negotiation.cc:922] calling sasl_server_start()
0809 12:15:28.846233 (+ 79us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0809 12:15:28.846251 (+ 18us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 12:15:28.846499 (+ 248us) server_negotiation.cc:1035] Waiting for connection context
0809 12:15:28.846631 (+ 132us) server_negotiation.cc:299] Negotiation successful
0809 12:15:28.846806 (+ 175us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":569,"thread_start_us":334,"threads_started":1}
I20260809 12:15:28.851395 888 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260809 12:15:28.853793 23574 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P c0f5735177b144a1a63a809c37b531f8: Bootstrap starting.
I20260809 12:15:28.857525 23574 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P c0f5735177b144a1a63a809c37b531f8: Neither blocks nor log segments found. Creating new log.
I20260809 12:15:28.858345 23577 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 12:15:28.855694 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46537 (local address 127.0.0.1:37879)
0809 12:15:28.855851 (+ 157us) server_negotiation.cc:206] Beginning negotiation
0809 12:15:28.855855 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0809 12:15:28.856317 (+ 462us) server_negotiation.cc:407] Connection header received
0809 12:15:28.856516 (+ 199us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 12:15:28.856520 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 12:15:28.856564 (+ 44us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 12:15:28.856658 (+ 94us) server_negotiation.cc:226] Negotiated authn=SASL
0809 12:15:28.857618 (+ 960us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 12:15:28.857623 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 12:15:28.857628 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0809 12:15:28.857650 (+ 22us) server_negotiation.cc:922] calling sasl_server_start()
0809 12:15:28.857697 (+ 47us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0809 12:15:28.857715 (+ 18us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 12:15:28.857845 (+ 130us) server_negotiation.cc:1035] Waiting for connection context
0809 12:15:28.858017 (+ 172us) server_negotiation.cc:299] Negotiation successful
0809 12:15:28.858077 (+ 60us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":51}
I20260809 12:15:28.858785 23574 log.cc:824] T 00000000000000000000000000000000 P c0f5735177b144a1a63a809c37b531f8: Log is configured to *not* fsync() on all Append() calls
I20260809 12:15:28.861454 23574 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P c0f5735177b144a1a63a809c37b531f8: No bootstrap required, opened a new log
I20260809 12:15:28.868710 23574 raft_consensus.cc:348] T 00000000000000000000000000000000 P c0f5735177b144a1a63a809c37b531f8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c0f5735177b144a1a63a809c37b531f8" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 37879 } }
I20260809 12:15:28.869071 23574 raft_consensus.cc:374] T 00000000000000000000000000000000 P c0f5735177b144a1a63a809c37b531f8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260809 12:15:28.869179 23574 raft_consensus.cc:729] T 00000000000000000000000000000000 P c0f5735177b144a1a63a809c37b531f8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c0f5735177b144a1a63a809c37b531f8, State: Initialized, Role: FOLLOWER
I20260809 12:15:28.869810 23574 consensus_queue.cc:260] T 00000000000000000000000000000000 P c0f5735177b144a1a63a809c37b531f8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c0f5735177b144a1a63a809c37b531f8" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 37879 } }
I20260809 12:15:28.870062 23574 raft_consensus.cc:388] T 00000000000000000000000000000000 P c0f5735177b144a1a63a809c37b531f8 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260809 12:15:28.870229 23574 raft_consensus.cc:482] T 00000000000000000000000000000000 P c0f5735177b144a1a63a809c37b531f8 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260809 12:15:28.870411 23574 raft_consensus.cc:3037] T 00000000000000000000000000000000 P c0f5735177b144a1a63a809c37b531f8 [term 0 FOLLOWER]: Advancing to term 1
I20260809 12:15:28.872644 23574 raft_consensus.cc:504] T 00000000000000000000000000000000 P c0f5735177b144a1a63a809c37b531f8 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c0f5735177b144a1a63a809c37b531f8" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 37879 } }
I20260809 12:15:28.873152 23574 leader_election.cc:302] T 00000000000000000000000000000000 P c0f5735177b144a1a63a809c37b531f8 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: c0f5735177b144a1a63a809c37b531f8; no voters:
I20260809 12:15:28.873826 23574 leader_election.cc:288] T 00000000000000000000000000000000 P c0f5735177b144a1a63a809c37b531f8 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260809 12:15:28.873965 23585 raft_consensus.cc:2781] T 00000000000000000000000000000000 P c0f5735177b144a1a63a809c37b531f8 [term 1 FOLLOWER]: Leader election won for term 1
I20260809 12:15:28.874696 23585 raft_consensus.cc:686] T 00000000000000000000000000000000 P c0f5735177b144a1a63a809c37b531f8 [term 1 LEADER]: Becoming Leader. State: Replica: c0f5735177b144a1a63a809c37b531f8, State: Running, Role: LEADER
I20260809 12:15:28.875272 23585 consensus_queue.cc:237] T 00000000000000000000000000000000 P c0f5735177b144a1a63a809c37b531f8 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c0f5735177b144a1a63a809c37b531f8" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 37879 } }
I20260809 12:15:28.876161 23574 sys_catalog.cc:564] T 00000000000000000000000000000000 P c0f5735177b144a1a63a809c37b531f8 [sys.catalog]: configured and running, proceeding with master startup.
I20260809 12:15:28.878227 23586 sys_catalog.cc:455] T 00000000000000000000000000000000 P c0f5735177b144a1a63a809c37b531f8 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "c0f5735177b144a1a63a809c37b531f8" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c0f5735177b144a1a63a809c37b531f8" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 37879 } } }
I20260809 12:15:28.878798 23586 sys_catalog.cc:458] T 00000000000000000000000000000000 P c0f5735177b144a1a63a809c37b531f8 [sys.catalog]: This master's current role is: LEADER
I20260809 12:15:28.878304 23587 sys_catalog.cc:455] T 00000000000000000000000000000000 P c0f5735177b144a1a63a809c37b531f8 [sys.catalog]: SysCatalogTable state changed. Reason: New leader c0f5735177b144a1a63a809c37b531f8. Latest consensus state: current_term: 1 leader_uuid: "c0f5735177b144a1a63a809c37b531f8" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c0f5735177b144a1a63a809c37b531f8" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 37879 } } }
I20260809 12:15:28.879391 23587 sys_catalog.cc:458] T 00000000000000000000000000000000 P c0f5735177b144a1a63a809c37b531f8 [sys.catalog]: This master's current role is: LEADER
I20260809 12:15:28.880647 23592 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260809 12:15:28.886272 23592 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260809 12:15:28.894608 23592 catalog_manager.cc:1347] Generated new cluster ID: 19b01caecaae4b749525ac23784db263
I20260809 12:15:28.894727 23592 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260809 12:15:28.925055 23592 catalog_manager.cc:1370] Generated new certificate authority record
I20260809 12:15:28.925544 23592 catalog_manager.cc:1504] Loading token signing keys...
I20260809 12:15:28.934597 23592 catalog_manager.cc:5868] T 00000000000000000000000000000000 P c0f5735177b144a1a63a809c37b531f8: Generated new TSK 0
I20260809 12:15:28.935582 23592 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260809 12:15:28.941934 888 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskgH5Pjm/build/asan/bin/kudu pid 23499
W20260809 12:15:28.943269 23568 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='slave'} at 127.0.0.1:50895
W20260809 12:15:28.943971 888 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskgH5Pjm/build/asan/bin/kudu pid 23499: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260809 12:15:28.944136 888 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskgH5Pjm/build/asan/bin/kudu with pid 23499
2026-08-09T12:15:28Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/5 (2099 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/6
Loading random data
Initializing database '/tmp/dist-test-taskgH5Pjm/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1786277382073938-888-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 09 12:15:29 dist-test-slave-2w3w krb5kdc[23611](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 09 12:15:29 dist-test-slave-2w3w krb5kdc[23611](info): set up 2 sockets
Aug 09 12:15:29 dist-test-slave-2w3w krb5kdc[23611](info): commencing operation
krb5kdc: starting...
W20260809 12:15:30.124635 888 mini_kdc.cc:121] Time spent starting KDC: real 1.138s user 0.000s sys 0.048s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskgH5Pjm/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1786277382073938-888-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskgH5Pjm/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1786277382073938-888-0/krb5kdc/test-user.keytab.
Aug 09 12:15:30 dist-test-slave-2w3w krb5kdc[23611](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786277730, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-09T12:15:30Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-09T12:15:30Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskgH5Pjm/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1786277382073938-888-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskgH5Pjm/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1786277382073938-888-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260809 12:15:30.462546 888 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskgH5Pjm/build/asan/bin/kudu
/tmp/dist-test-taskgH5Pjm/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskgH5Pjm/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1786277382073938-888-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskgH5Pjm/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1786277382073938-888-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskgH5Pjm/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1786277382073938-888-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskgH5Pjm/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1786277382073938-888-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:42207
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskgH5Pjm/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1786277382073938-888-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:40873
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:42207
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=required
--rpc_encryption=required
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taskgH5Pjm/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1786277382073938-888-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskgH5Pjm/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1786277382073938-888-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskgH5Pjm/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1786277382073938-888-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260809 12:15:30.809324 23625 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260809 12:15:30.809590 23625 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260809 12:15:30.809684 23625 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260809 12:15:30.817047 23625 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260809 12:15:30.817154 23625 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260809 12:15:30.817202 23625 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260809 12:15:30.817265 23625 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260809 12:15:30.817314 23625 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260809 12:15:30.826977 23625 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:40873
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskgH5Pjm/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1786277382073938-888-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskgH5Pjm/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1786277382073938-888-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:42207
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskgH5Pjm/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1786277382073938-888-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:42207
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskgH5Pjm/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1786277382073938-888-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskgH5Pjm/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1786277382073938-888-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 09 Aug 2026 11:06:41 UTC on ccycloud.quasar-bwoogy.root.comops.site
build id 4575991
ASAN enabled
I20260809 12:15:30.828173 23625 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260809 12:15:30.829392 23625 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260809 12:15:30.841295 23630 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 12:15:30.841589 23631 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 12:15:30.841609 23634 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260809 12:15:30.842862 23625 server_base.cc:1034] running on GCE node
I20260809 12:15:30.843726 23625 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260809 12:15:30.845113 23625 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260809 12:15:30.846365 23625 hybrid_clock.cc:648] HybridClock initialized: now 1786277730846321 us; error 57 us; skew 500 ppm
I20260809 12:15:30.846727 23625 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 09 12:15:30 dist-test-slave-2w3w krb5kdc[23611](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786277730, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260809 12:15:30.849927 23625 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260809 12:15:30.851395 23625 webserver.cc:460] Webserver started at http://127.0.0.1:45599/ using document root <none> and password file <none>
I20260809 12:15:30.852020 23625 fs_manager.cc:359] Metadata directory not provided
I20260809 12:15:30.852123 23625 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260809 12:15:30.852388 23625 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260809 12:15:30.855495 23625 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskgH5Pjm/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1786277382073938-888-0/minicluster-data/master-0/data/instance:
uuid: "93dc867750004007ae5d736b93302ef0"
format_stamp: "Formatted at 2026-08-09 12:15:30 on dist-test-slave-2w3w"
server_key: "eb5512afa24ca04b8cf7feb3a2af69c7"
server_key_iv: "67e25d6c1998b2db39d2e7006d7fbabe"
server_key_version: "encryptionkey@0"
I20260809 12:15:30.856328 23625 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskgH5Pjm/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1786277382073938-888-0/minicluster-data/master-0/wal/instance:
uuid: "93dc867750004007ae5d736b93302ef0"
format_stamp: "Formatted at 2026-08-09 12:15:30 on dist-test-slave-2w3w"
server_key: "eb5512afa24ca04b8cf7feb3a2af69c7"
server_key_iv: "67e25d6c1998b2db39d2e7006d7fbabe"
server_key_version: "encryptionkey@0"
I20260809 12:15:30.861745 23625 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.000s
I20260809 12:15:30.865360 23641 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260809 12:15:30.866878 23625 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.001s
I20260809 12:15:30.867111 23625 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskgH5Pjm/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1786277382073938-888-0/minicluster-data/master-0/data,/tmp/dist-test-taskgH5Pjm/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1786277382073938-888-0/minicluster-data/master-0/wal
uuid: "93dc867750004007ae5d736b93302ef0"
format_stamp: "Formatted at 2026-08-09 12:15:30 on dist-test-slave-2w3w"
server_key: "eb5512afa24ca04b8cf7feb3a2af69c7"
server_key_iv: "67e25d6c1998b2db39d2e7006d7fbabe"
server_key_version: "encryptionkey@0"
I20260809 12:15:30.867311 23625 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskgH5Pjm/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1786277382073938-888-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskgH5Pjm/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1786277382073938-888-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskgH5Pjm/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1786277382073938-888-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260809 12:15:30.878410 23625 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260809 12:15:30.882290 23625 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260809 12:15:30.882571 23625 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260809 12:15:30.902105 23625 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:42207
I20260809 12:15:30.902146 23698 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:42207 every 8 connection(s)
I20260809 12:15:30.903580 23625 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskgH5Pjm/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1786277382073938-888-0/minicluster-data/master-0/data/info.pb
I20260809 12:15:30.907274 888 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskgH5Pjm/build/asan/bin/kudu as pid 23625
I20260809 12:15:30.907478 888 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskgH5Pjm/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1786277382073938-888-0/minicluster-data/master-0/wal/instance
I20260809 12:15:30.907706 23699 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260809 12:15:30.907977 888 external_mini_cluster.cc:1428] Setting key c17f388588668a61a6ddd499888543ed
I20260809 12:15:30.922925 23699 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 93dc867750004007ae5d736b93302ef0: Bootstrap starting.
Aug 09 12:15:30 dist-test-slave-2w3w krb5kdc[23611](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786277730, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.0.1@KRBTEST.COM
I20260809 12:15:30.926545 23699 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 93dc867750004007ae5d736b93302ef0: Neither blocks nor log segments found. Creating new log.
I20260809 12:15:30.927723 23699 log.cc:824] T 00000000000000000000000000000000 P 93dc867750004007ae5d736b93302ef0: Log is configured to *not* fsync() on all Append() calls
I20260809 12:15:30.930559 23699 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 93dc867750004007ae5d736b93302ef0: No bootstrap required, opened a new log
I20260809 12:15:30.932482 23702 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 12:15:30.909867 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.84:58815 (local address 127.0.0.1:42207)
0809 12:15:30.910720 (+ 853us) server_negotiation.cc:206] Beginning negotiation
0809 12:15:30.910731 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0809 12:15:30.910770 (+ 39us) server_negotiation.cc:407] Connection header received
0809 12:15:30.912488 (+ 1718us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 12:15:30.912509 (+ 21us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 12:15:30.912891 (+ 382us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 12:15:30.913221 (+ 330us) server_negotiation.cc:226] Negotiated authn=SASL
0809 12:15:30.915177 (+ 1956us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 12:15:30.916476 (+ 1299us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 12:15:30.918347 (+ 1871us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 12:15:30.919149 (+ 802us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 12:15:30.919295 (+ 146us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 12:15:30.924191 (+ 4896us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 12:15:30.924199 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 12:15:30.924213 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0809 12:15:30.924251 (+ 38us) server_negotiation.cc:922] calling sasl_server_start()
0809 12:15:30.927818 (+ 3567us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 12:15:30.928781 (+ 963us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 12:15:30.928792 (+ 11us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 12:15:30.928799 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0809 12:15:30.928986 (+ 187us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 12:15:30.929519 (+ 533us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 12:15:30.929525 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 12:15:30.929527 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0809 12:15:30.930055 (+ 528us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 12:15:30.930431 (+ 376us) server_negotiation.cc:1035] Waiting for connection context
0809 12:15:30.931022 (+ 591us) server_negotiation.cc:299] Negotiation successful
0809 12:15:30.931403 (+ 381us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":638,"thread_start_us":372,"threads_started":1}
I20260809 12:15:30.936789 888 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260809 12:15:30.939136 23699 raft_consensus.cc:348] T 00000000000000000000000000000000 P 93dc867750004007ae5d736b93302ef0 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "93dc867750004007ae5d736b93302ef0" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 42207 } }
I20260809 12:15:30.939541 23699 raft_consensus.cc:374] T 00000000000000000000000000000000 P 93dc867750004007ae5d736b93302ef0 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260809 12:15:30.939668 23699 raft_consensus.cc:729] T 00000000000000000000000000000000 P 93dc867750004007ae5d736b93302ef0 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 93dc867750004007ae5d736b93302ef0, State: Initialized, Role: FOLLOWER
I20260809 12:15:30.940292 23699 consensus_queue.cc:260] T 00000000000000000000000000000000 P 93dc867750004007ae5d736b93302ef0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "93dc867750004007ae5d736b93302ef0" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 42207 } }
I20260809 12:15:30.940613 23699 raft_consensus.cc:388] T 00000000000000000000000000000000 P 93dc867750004007ae5d736b93302ef0 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260809 12:15:30.940817 23699 raft_consensus.cc:482] T 00000000000000000000000000000000 P 93dc867750004007ae5d736b93302ef0 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260809 12:15:30.940999 23699 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 93dc867750004007ae5d736b93302ef0 [term 0 FOLLOWER]: Advancing to term 1
I20260809 12:15:30.944476 23699 raft_consensus.cc:504] T 00000000000000000000000000000000 P 93dc867750004007ae5d736b93302ef0 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "93dc867750004007ae5d736b93302ef0" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 42207 } }
I20260809 12:15:30.945142 23699 leader_election.cc:302] T 00000000000000000000000000000000 P 93dc867750004007ae5d736b93302ef0 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 93dc867750004007ae5d736b93302ef0; no voters:
I20260809 12:15:30.945858 23699 leader_election.cc:288] T 00000000000000000000000000000000 P 93dc867750004007ae5d736b93302ef0 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260809 12:15:30.946010 23709 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 93dc867750004007ae5d736b93302ef0 [term 1 FOLLOWER]: Leader election won for term 1
I20260809 12:15:30.948699 23699 sys_catalog.cc:564] T 00000000000000000000000000000000 P 93dc867750004007ae5d736b93302ef0 [sys.catalog]: configured and running, proceeding with master startup.
I20260809 12:15:30.952960 23709 raft_consensus.cc:686] T 00000000000000000000000000000000 P 93dc867750004007ae5d736b93302ef0 [term 1 LEADER]: Becoming Leader. State: Replica: 93dc867750004007ae5d736b93302ef0, State: Running, Role: LEADER
I20260809 12:15:30.953732 23709 consensus_queue.cc:237] T 00000000000000000000000000000000 P 93dc867750004007ae5d736b93302ef0 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "93dc867750004007ae5d736b93302ef0" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 42207 } }
I20260809 12:15:30.958935 23702 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 12:15:30.941905 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.84:53953 (local address 127.0.0.1:42207)
0809 12:15:30.942167 (+ 262us) server_negotiation.cc:206] Beginning negotiation
0809 12:15:30.942176 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0809 12:15:30.942510 (+ 334us) server_negotiation.cc:407] Connection header received
0809 12:15:30.942694 (+ 184us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 12:15:30.942698 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 12:15:30.942821 (+ 123us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 12:15:30.942915 (+ 94us) server_negotiation.cc:226] Negotiated authn=SASL
0809 12:15:30.944979 (+ 2064us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 12:15:30.946026 (+ 1047us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 12:15:30.947815 (+ 1789us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 12:15:30.948401 (+ 586us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 12:15:30.948553 (+ 152us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 12:15:30.951617 (+ 3064us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 12:15:30.951622 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 12:15:30.951626 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0809 12:15:30.951659 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0809 12:15:30.955468 (+ 3809us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 12:15:30.956289 (+ 821us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 12:15:30.956296 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 12:15:30.956300 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0809 12:15:30.956436 (+ 136us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 12:15:30.957046 (+ 610us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 12:15:30.957051 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 12:15:30.957054 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0809 12:15:30.957528 (+ 474us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 12:15:30.957899 (+ 371us) server_negotiation.cc:1035] Waiting for connection context
0809 12:15:30.958327 (+ 428us) server_negotiation.cc:299] Negotiation successful
0809 12:15:30.958523 (+ 196us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":77}
I20260809 12:15:30.960716 23711 sys_catalog.cc:455] T 00000000000000000000000000000000 P 93dc867750004007ae5d736b93302ef0 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 93dc867750004007ae5d736b93302ef0. Latest consensus state: current_term: 1 leader_uuid: "93dc867750004007ae5d736b93302ef0" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "93dc867750004007ae5d736b93302ef0" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 42207 } } }
I20260809 12:15:30.961200 23711 sys_catalog.cc:458] T 00000000000000000000000000000000 P 93dc867750004007ae5d736b93302ef0 [sys.catalog]: This master's current role is: LEADER
I20260809 12:15:30.961953 23710 sys_catalog.cc:455] T 00000000000000000000000000000000 P 93dc867750004007ae5d736b93302ef0 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "93dc867750004007ae5d736b93302ef0" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "93dc867750004007ae5d736b93302ef0" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 42207 } } }
I20260809 12:15:30.962359 23718 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260809 12:15:30.962447 23710 sys_catalog.cc:458] T 00000000000000000000000000000000 P 93dc867750004007ae5d736b93302ef0 [sys.catalog]: This master's current role is: LEADER
I20260809 12:15:30.976387 23718 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260809 12:15:30.986614 23718 catalog_manager.cc:1347] Generated new cluster ID: 0526d8e86a324d1098153117e9eaa5e0
I20260809 12:15:30.986770 23718 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260809 12:15:31.014760 23718 catalog_manager.cc:1370] Generated new certificate authority record
I20260809 12:15:31.016197 23718 catalog_manager.cc:1504] Loading token signing keys...
I20260809 12:15:31.023017 23718 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 93dc867750004007ae5d736b93302ef0: Generated new TSK 0
I20260809 12:15:31.023859 23718 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260809 12:15:31.034495 888 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskgH5Pjm/build/asan/bin/kudu pid 23625
I20260809 12:15:31.035719 23692 generic_service.cc:196] Checking for leaks (request via RPC)
I20260809 12:15:31.136848 888 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskgH5Pjm/build/asan/bin/kudu with pid 23625
W20260809 12:15:31.158502 23609 connection.cc:535] client connection to 127.0.0.1:42207 recv error: Network error: failed to read from TLS socket (remote: 127.0.0.1:42207): Connection reset by peer (error 104)
2026-08-09T12:15:31Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/6 (2192 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/7
Loading random data
Initializing database '/tmp/dist-test-taskgH5Pjm/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1786277382073938-888-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 09 12:15:31 dist-test-slave-2w3w krb5kdc[23734](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 09 12:15:31 dist-test-slave-2w3w krb5kdc[23734](info): set up 2 sockets
Aug 09 12:15:31 dist-test-slave-2w3w krb5kdc[23734](info): commencing operation
krb5kdc: starting...
W20260809 12:15:32.368422 888 mini_kdc.cc:121] Time spent starting KDC: real 1.190s user 0.000s sys 0.048s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskgH5Pjm/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1786277382073938-888-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskgH5Pjm/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1786277382073938-888-0/krb5kdc/test-user.keytab.
Aug 09 12:15:32 dist-test-slave-2w3w krb5kdc[23734](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786277732, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-09T12:15:32Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-09T12:15:32Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskgH5Pjm/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1786277382073938-888-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskgH5Pjm/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1786277382073938-888-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260809 12:15:32.710537 888 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskgH5Pjm/build/asan/bin/kudu
/tmp/dist-test-taskgH5Pjm/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskgH5Pjm/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1786277382073938-888-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskgH5Pjm/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1786277382073938-888-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskgH5Pjm/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1786277382073938-888-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskgH5Pjm/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1786277382073938-888-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:33633
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskgH5Pjm/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1786277382073938-888-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:35291
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:33633
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=required
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taskgH5Pjm/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1786277382073938-888-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskgH5Pjm/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1786277382073938-888-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskgH5Pjm/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1786277382073938-888-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260809 12:15:33.068606 23750 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260809 12:15:33.068886 23750 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260809 12:15:33.069099 23750 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260809 12:15:33.076493 23750 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260809 12:15:33.076601 23750 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260809 12:15:33.076738 23750 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260809 12:15:33.076849 23750 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260809 12:15:33.076936 23750 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260809 12:15:33.086892 23750 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:35291
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskgH5Pjm/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1786277382073938-888-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskgH5Pjm/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1786277382073938-888-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:33633
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskgH5Pjm/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1786277382073938-888-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:33633
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskgH5Pjm/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1786277382073938-888-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskgH5Pjm/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1786277382073938-888-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 09 Aug 2026 11:06:41 UTC on ccycloud.quasar-bwoogy.root.comops.site
build id 4575991
ASAN enabled
I20260809 12:15:33.088250 23750 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260809 12:15:33.089576 23750 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260809 12:15:33.101421 23755 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 12:15:33.101631 23756 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 12:15:33.101828 23758 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260809 12:15:33.102489 23750 server_base.cc:1034] running on GCE node
I20260809 12:15:33.103361 23750 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260809 12:15:33.104825 23750 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260809 12:15:33.106058 23750 hybrid_clock.cc:648] HybridClock initialized: now 1786277733106020 us; error 49 us; skew 500 ppm
I20260809 12:15:33.106413 23750 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 09 12:15:33 dist-test-slave-2w3w krb5kdc[23734](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786277733, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260809 12:15:33.113160 23750 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260809 12:15:33.114765 23750 webserver.cc:460] Webserver started at http://127.0.0.1:46427/ using document root <none> and password file <none>
I20260809 12:15:33.115486 23750 fs_manager.cc:359] Metadata directory not provided
I20260809 12:15:33.115593 23750 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260809 12:15:33.115859 23750 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260809 12:15:33.118741 23750 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskgH5Pjm/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1786277382073938-888-0/minicluster-data/master-0/data/instance:
uuid: "8f8ce704b822412395d51071e2246009"
format_stamp: "Formatted at 2026-08-09 12:15:33 on dist-test-slave-2w3w"
server_key: "7d97acfbfe8463d3640a5420aa7311c3"
server_key_iv: "d7e5582fddad10e6032f3e100f963f2e"
server_key_version: "encryptionkey@0"
I20260809 12:15:33.119515 23750 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskgH5Pjm/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1786277382073938-888-0/minicluster-data/master-0/wal/instance:
uuid: "8f8ce704b822412395d51071e2246009"
format_stamp: "Formatted at 2026-08-09 12:15:33 on dist-test-slave-2w3w"
server_key: "7d97acfbfe8463d3640a5420aa7311c3"
server_key_iv: "d7e5582fddad10e6032f3e100f963f2e"
server_key_version: "encryptionkey@0"
I20260809 12:15:33.124555 23750 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.000s sys 0.004s
I20260809 12:15:33.127954 23765 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260809 12:15:33.129282 23750 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.001s
I20260809 12:15:33.129468 23750 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskgH5Pjm/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1786277382073938-888-0/minicluster-data/master-0/data,/tmp/dist-test-taskgH5Pjm/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1786277382073938-888-0/minicluster-data/master-0/wal
uuid: "8f8ce704b822412395d51071e2246009"
format_stamp: "Formatted at 2026-08-09 12:15:33 on dist-test-slave-2w3w"
server_key: "7d97acfbfe8463d3640a5420aa7311c3"
server_key_iv: "d7e5582fddad10e6032f3e100f963f2e"
server_key_version: "encryptionkey@0"
I20260809 12:15:33.129662 23750 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskgH5Pjm/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1786277382073938-888-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskgH5Pjm/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1786277382073938-888-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskgH5Pjm/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1786277382073938-888-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260809 12:15:33.153923 23750 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260809 12:15:33.158061 23750 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260809 12:15:33.158376 23750 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260809 12:15:33.177328 23750 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:33633
I20260809 12:15:33.177366 23821 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:33633 every 8 connection(s)
I20260809 12:15:33.178694 23750 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskgH5Pjm/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1786277382073938-888-0/minicluster-data/master-0/data/info.pb
I20260809 12:15:33.182336 23822 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260809 12:15:33.185686 888 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskgH5Pjm/build/asan/bin/kudu as pid 23750
I20260809 12:15:33.185869 888 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskgH5Pjm/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1786277382073938-888-0/minicluster-data/master-0/wal/instance
I20260809 12:15:33.186437 888 external_mini_cluster.cc:1428] Setting key 57bd86d1d4ae49f94e207e0a80593be9
I20260809 12:15:33.194821 23822 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 8f8ce704b822412395d51071e2246009: Bootstrap starting.
I20260809 12:15:33.198546 23822 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 8f8ce704b822412395d51071e2246009: Neither blocks nor log segments found. Creating new log.
I20260809 12:15:33.199743 23822 log.cc:824] T 00000000000000000000000000000000 P 8f8ce704b822412395d51071e2246009: Log is configured to *not* fsync() on all Append() calls
I20260809 12:15:33.200133 23826 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 12:15:33.188367 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.84:47197 (local address 127.0.0.1:33633)
0809 12:15:33.189139 (+ 772us) server_negotiation.cc:206] Beginning negotiation
0809 12:15:33.189150 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0809 12:15:33.189192 (+ 42us) server_negotiation.cc:407] Connection header received
0809 12:15:33.190867 (+ 1675us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 12:15:33.190889 (+ 22us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 12:15:33.191357 (+ 468us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 12:15:33.191676 (+ 319us) server_negotiation.cc:226] Negotiated authn=SASL
0809 12:15:33.192284 (+ 608us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 12:15:33.193605 (+ 1321us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 12:15:33.195518 (+ 1913us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 12:15:33.196153 (+ 635us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 12:15:33.196282 (+ 129us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 12:15:33.197666 (+ 1384us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 12:15:33.197673 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 12:15:33.197686 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0809 12:15:33.198212 (+ 526us) server_negotiation.cc:922] calling sasl_server_start()
0809 12:15:33.198289 (+ 77us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0809 12:15:33.198314 (+ 25us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 12:15:33.198442 (+ 128us) server_negotiation.cc:1035] Waiting for connection context
0809 12:15:33.198878 (+ 436us) server_negotiation.cc:299] Negotiation successful
0809 12:15:33.199102 (+ 224us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":579,"thread_start_us":371,"threads_started":1}
I20260809 12:15:33.202703 23822 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 8f8ce704b822412395d51071e2246009: No bootstrap required, opened a new log
I20260809 12:15:33.204173 888 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260809 12:15:33.211122 23822 raft_consensus.cc:348] T 00000000000000000000000000000000 P 8f8ce704b822412395d51071e2246009 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8f8ce704b822412395d51071e2246009" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 33633 } }
I20260809 12:15:33.211566 23822 raft_consensus.cc:374] T 00000000000000000000000000000000 P 8f8ce704b822412395d51071e2246009 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260809 12:15:33.211678 23822 raft_consensus.cc:729] T 00000000000000000000000000000000 P 8f8ce704b822412395d51071e2246009 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8f8ce704b822412395d51071e2246009, State: Initialized, Role: FOLLOWER
I20260809 12:15:33.212273 23822 consensus_queue.cc:260] T 00000000000000000000000000000000 P 8f8ce704b822412395d51071e2246009 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8f8ce704b822412395d51071e2246009" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 33633 } }
I20260809 12:15:33.212545 23822 raft_consensus.cc:388] T 00000000000000000000000000000000 P 8f8ce704b822412395d51071e2246009 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260809 12:15:33.212700 23822 raft_consensus.cc:482] T 00000000000000000000000000000000 P 8f8ce704b822412395d51071e2246009 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260809 12:15:33.212877 23822 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 8f8ce704b822412395d51071e2246009 [term 0 FOLLOWER]: Advancing to term 1
I20260809 12:15:33.215663 23826 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 12:15:33.208654 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.84:57641 (local address 127.0.0.1:33633)
0809 12:15:33.208884 (+ 230us) server_negotiation.cc:206] Beginning negotiation
0809 12:15:33.208897 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0809 12:15:33.209270 (+ 373us) server_negotiation.cc:407] Connection header received
0809 12:15:33.209502 (+ 232us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 12:15:33.209507 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 12:15:33.209583 (+ 76us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 12:15:33.209698 (+ 115us) server_negotiation.cc:226] Negotiated authn=SASL
0809 12:15:33.210227 (+ 529us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 12:15:33.211209 (+ 982us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 12:15:33.213021 (+ 1812us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 12:15:33.213481 (+ 460us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 12:15:33.213639 (+ 158us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 12:15:33.214871 (+ 1232us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 12:15:33.214876 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 12:15:33.214879 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0809 12:15:33.214897 (+ 18us) server_negotiation.cc:922] calling sasl_server_start()
0809 12:15:33.214935 (+ 38us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0809 12:15:33.214950 (+ 15us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 12:15:33.215129 (+ 179us) server_negotiation.cc:1035] Waiting for connection context
0809 12:15:33.215329 (+ 200us) server_negotiation.cc:299] Negotiation successful
0809 12:15:33.215379 (+ 50us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":52}
I20260809 12:15:33.215772 23822 raft_consensus.cc:504] T 00000000000000000000000000000000 P 8f8ce704b822412395d51071e2246009 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8f8ce704b822412395d51071e2246009" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 33633 } }
I20260809 12:15:33.216331 23822 leader_election.cc:302] T 00000000000000000000000000000000 P 8f8ce704b822412395d51071e2246009 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 8f8ce704b822412395d51071e2246009; no voters:
I20260809 12:15:33.217051 23822 leader_election.cc:288] T 00000000000000000000000000000000 P 8f8ce704b822412395d51071e2246009 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260809 12:15:33.217175 23837 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 8f8ce704b822412395d51071e2246009 [term 1 FOLLOWER]: Leader election won for term 1
I20260809 12:15:33.217965 23837 raft_consensus.cc:686] T 00000000000000000000000000000000 P 8f8ce704b822412395d51071e2246009 [term 1 LEADER]: Becoming Leader. State: Replica: 8f8ce704b822412395d51071e2246009, State: Running, Role: LEADER
I20260809 12:15:33.218561 23837 consensus_queue.cc:237] T 00000000000000000000000000000000 P 8f8ce704b822412395d51071e2246009 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8f8ce704b822412395d51071e2246009" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 33633 } }
I20260809 12:15:33.219674 23822 sys_catalog.cc:564] T 00000000000000000000000000000000 P 8f8ce704b822412395d51071e2246009 [sys.catalog]: configured and running, proceeding with master startup.
I20260809 12:15:33.221604 23838 sys_catalog.cc:455] T 00000000000000000000000000000000 P 8f8ce704b822412395d51071e2246009 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "8f8ce704b822412395d51071e2246009" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8f8ce704b822412395d51071e2246009" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 33633 } } }
I20260809 12:15:33.221695 23839 sys_catalog.cc:455] T 00000000000000000000000000000000 P 8f8ce704b822412395d51071e2246009 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 8f8ce704b822412395d51071e2246009. Latest consensus state: current_term: 1 leader_uuid: "8f8ce704b822412395d51071e2246009" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8f8ce704b822412395d51071e2246009" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 33633 } } }
I20260809 12:15:33.222057 23838 sys_catalog.cc:458] T 00000000000000000000000000000000 P 8f8ce704b822412395d51071e2246009 [sys.catalog]: This master's current role is: LEADER
I20260809 12:15:33.222134 23839 sys_catalog.cc:458] T 00000000000000000000000000000000 P 8f8ce704b822412395d51071e2246009 [sys.catalog]: This master's current role is: LEADER
I20260809 12:15:33.230865 23842 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260809 12:15:33.236274 23842 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260809 12:15:33.245908 23842 catalog_manager.cc:1347] Generated new cluster ID: ab4187ac344a437c86075d0f724da057
I20260809 12:15:33.246038 23842 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260809 12:15:33.260672 23842 catalog_manager.cc:1370] Generated new certificate authority record
I20260809 12:15:33.262481 23842 catalog_manager.cc:1504] Loading token signing keys...
I20260809 12:15:33.277415 23842 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 8f8ce704b822412395d51071e2246009: Generated new TSK 0
I20260809 12:15:33.278311 23842 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260809 12:15:33.287317 888 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskgH5Pjm/build/asan/bin/kudu pid 23750
W20260809 12:15:33.289014 23817 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='slave'} at 10.240.0.84:47197
W20260809 12:15:33.289781 888 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskgH5Pjm/build/asan/bin/kudu pid 23750: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260809 12:15:33.289943 888 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskgH5Pjm/build/asan/bin/kudu with pid 23750
W20260809 12:15:33.312441 23731 connection.cc:535] client connection to 127.0.0.1:33633 recv error: Network error: failed to read from TLS socket (remote: 127.0.0.1:33633): Connection reset by peer (error 104)
2026-08-09T12:15:33Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/7 (2154 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/8
Loading random data
Initializing database '/tmp/dist-test-taskgH5Pjm/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1786277382073938-888-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 09 12:15:33 dist-test-slave-2w3w krb5kdc[23859](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 09 12:15:33 dist-test-slave-2w3w krb5kdc[23859](info): set up 2 sockets
Aug 09 12:15:33 dist-test-slave-2w3w krb5kdc[23859](info): commencing operation
krb5kdc: starting...
W20260809 12:15:34.484334 888 mini_kdc.cc:121] Time spent starting KDC: real 1.152s user 0.004s sys 0.046s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskgH5Pjm/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1786277382073938-888-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskgH5Pjm/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1786277382073938-888-0/krb5kdc/test-user.keytab.
Aug 09 12:15:34 dist-test-slave-2w3w krb5kdc[23859](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786277734, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-09T12:15:34Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-09T12:15:34Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskgH5Pjm/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1786277382073938-888-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskgH5Pjm/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1786277382073938-888-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260809 12:15:34.822794 888 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskgH5Pjm/build/asan/bin/kudu
/tmp/dist-test-taskgH5Pjm/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskgH5Pjm/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1786277382073938-888-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskgH5Pjm/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1786277382073938-888-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskgH5Pjm/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1786277382073938-888-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskgH5Pjm/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1786277382073938-888-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:40599
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskgH5Pjm/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1786277382073938-888-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:34885
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:40599
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=disabled
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taskgH5Pjm/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1786277382073938-888-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskgH5Pjm/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1786277382073938-888-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskgH5Pjm/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1786277382073938-888-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260809 12:15:35.171617 23874 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260809 12:15:35.171918 23874 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260809 12:15:35.172111 23874 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260809 12:15:35.179569 23874 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260809 12:15:35.179682 23874 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260809 12:15:35.179798 23874 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260809 12:15:35.179912 23874 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260809 12:15:35.179998 23874 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260809 12:15:35.190191 23874 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:34885
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskgH5Pjm/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1786277382073938-888-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskgH5Pjm/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1786277382073938-888-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:40599
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskgH5Pjm/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1786277382073938-888-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:40599
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskgH5Pjm/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1786277382073938-888-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskgH5Pjm/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1786277382073938-888-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 09 Aug 2026 11:06:41 UTC on ccycloud.quasar-bwoogy.root.comops.site
build id 4575991
ASAN enabled
I20260809 12:15:35.191608 23874 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260809 12:15:35.192968 23874 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260809 12:15:35.205049 23882 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 12:15:35.205097 23879 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 12:15:35.205122 23880 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260809 12:15:35.205981 23874 server_base.cc:1034] running on GCE node
I20260809 12:15:35.206766 23874 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260809 12:15:35.208247 23874 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260809 12:15:35.209502 23874 hybrid_clock.cc:648] HybridClock initialized: now 1786277735209455 us; error 40 us; skew 500 ppm
I20260809 12:15:35.209894 23874 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 09 12:15:35 dist-test-slave-2w3w krb5kdc[23859](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786277735, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260809 12:15:35.213218 23874 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260809 12:15:35.214919 23874 webserver.cc:460] Webserver started at http://127.0.0.1:36877/ using document root <none> and password file <none>
I20260809 12:15:35.215700 23874 fs_manager.cc:359] Metadata directory not provided
I20260809 12:15:35.215864 23874 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260809 12:15:35.216197 23874 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260809 12:15:35.219448 23874 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskgH5Pjm/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1786277382073938-888-0/minicluster-data/master-0/data/instance:
uuid: "41a1ee0c5e354eecbc4c18835b0455de"
format_stamp: "Formatted at 2026-08-09 12:15:35 on dist-test-slave-2w3w"
server_key: "800622fef7af7878631bd374a264953f"
server_key_iv: "99fd10baa4eb0e9a3943b24caf8d9bd3"
server_key_version: "encryptionkey@0"
I20260809 12:15:35.220237 23874 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskgH5Pjm/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1786277382073938-888-0/minicluster-data/master-0/wal/instance:
uuid: "41a1ee0c5e354eecbc4c18835b0455de"
format_stamp: "Formatted at 2026-08-09 12:15:35 on dist-test-slave-2w3w"
server_key: "800622fef7af7878631bd374a264953f"
server_key_iv: "99fd10baa4eb0e9a3943b24caf8d9bd3"
server_key_version: "encryptionkey@0"
I20260809 12:15:35.225593 23874 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.000s
I20260809 12:15:35.229146 23889 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260809 12:15:35.230501 23874 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260809 12:15:35.230695 23874 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskgH5Pjm/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1786277382073938-888-0/minicluster-data/master-0/data,/tmp/dist-test-taskgH5Pjm/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1786277382073938-888-0/minicluster-data/master-0/wal
uuid: "41a1ee0c5e354eecbc4c18835b0455de"
format_stamp: "Formatted at 2026-08-09 12:15:35 on dist-test-slave-2w3w"
server_key: "800622fef7af7878631bd374a264953f"
server_key_iv: "99fd10baa4eb0e9a3943b24caf8d9bd3"
server_key_version: "encryptionkey@0"
I20260809 12:15:35.230918 23874 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskgH5Pjm/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1786277382073938-888-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskgH5Pjm/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1786277382073938-888-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskgH5Pjm/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1786277382073938-888-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260809 12:15:35.237165 23874 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260809 12:15:35.241694 23874 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260809 12:15:35.242038 23874 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260809 12:15:35.261431 23874 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:40599
I20260809 12:15:35.261482 23945 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:40599 every 8 connection(s)
I20260809 12:15:35.262861 23874 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskgH5Pjm/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1786277382073938-888-0/minicluster-data/master-0/data/info.pb
I20260809 12:15:35.265077 888 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskgH5Pjm/build/asan/bin/kudu as pid 23874
I20260809 12:15:35.265290 888 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskgH5Pjm/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1786277382073938-888-0/minicluster-data/master-0/wal/instance
I20260809 12:15:35.265825 888 external_mini_cluster.cc:1428] Setting key aa2c08d4dd8552524931f95e884ebf15
I20260809 12:15:35.267439 23946 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260809 12:15:35.273571 23950 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 12:15:35.267712 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.84:35323 (local address 127.0.0.1:40599)
0809 12:15:35.268523 (+ 811us) server_negotiation.cc:206] Beginning negotiation
0809 12:15:35.268534 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0809 12:15:35.268577 (+ 43us) server_negotiation.cc:407] Connection header received
0809 12:15:35.270204 (+ 1627us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 12:15:35.270225 (+ 21us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 12:15:35.270551 (+ 326us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 12:15:35.270883 (+ 332us) server_negotiation.cc:226] Negotiated authn=SASL
0809 12:15:35.272005 (+ 1122us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 12:15:35.272011 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 12:15:35.272022 (+ 11us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0809 12:15:35.272036 (+ 14us) server_negotiation.cc:922] calling sasl_server_start()
0809 12:15:35.272113 (+ 77us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0809 12:15:35.272132 (+ 19us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 12:15:35.272301 (+ 169us) server_negotiation.cc:1035] Waiting for connection context
0809 12:15:35.272425 (+ 124us) server_negotiation.cc:299] Negotiation successful
0809 12:15:35.272632 (+ 207us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":594,"thread_start_us":369,"threads_started":1}
I20260809 12:15:35.277235 888 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260809 12:15:35.281047 23946 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 41a1ee0c5e354eecbc4c18835b0455de: Bootstrap starting.
I20260809 12:15:35.285199 23946 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 41a1ee0c5e354eecbc4c18835b0455de: Neither blocks nor log segments found. Creating new log.
I20260809 12:15:35.285459 23950 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 12:15:35.282853 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.84:49475 (local address 127.0.0.1:40599)
0809 12:15:35.283095 (+ 242us) server_negotiation.cc:206] Beginning negotiation
0809 12:15:35.283102 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0809 12:15:35.283475 (+ 373us) server_negotiation.cc:407] Connection header received
0809 12:15:35.283616 (+ 141us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 12:15:35.283619 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 12:15:35.283668 (+ 49us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 12:15:35.283752 (+ 84us) server_negotiation.cc:226] Negotiated authn=SASL
0809 12:15:35.284683 (+ 931us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 12:15:35.284687 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 12:15:35.284690 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0809 12:15:35.284704 (+ 14us) server_negotiation.cc:922] calling sasl_server_start()
0809 12:15:35.284741 (+ 37us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0809 12:15:35.284755 (+ 14us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 12:15:35.284843 (+ 88us) server_negotiation.cc:1035] Waiting for connection context
0809 12:15:35.285004 (+ 161us) server_negotiation.cc:299] Negotiation successful
0809 12:15:35.285051 (+ 47us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":115}
I20260809 12:15:35.286401 23946 log.cc:824] T 00000000000000000000000000000000 P 41a1ee0c5e354eecbc4c18835b0455de: Log is configured to *not* fsync() on all Append() calls
I20260809 12:15:35.289471 23946 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 41a1ee0c5e354eecbc4c18835b0455de: No bootstrap required, opened a new log
I20260809 12:15:35.297156 23946 raft_consensus.cc:348] T 00000000000000000000000000000000 P 41a1ee0c5e354eecbc4c18835b0455de [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "41a1ee0c5e354eecbc4c18835b0455de" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 40599 } }
I20260809 12:15:35.297520 23946 raft_consensus.cc:374] T 00000000000000000000000000000000 P 41a1ee0c5e354eecbc4c18835b0455de [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260809 12:15:35.297619 23946 raft_consensus.cc:729] T 00000000000000000000000000000000 P 41a1ee0c5e354eecbc4c18835b0455de [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 41a1ee0c5e354eecbc4c18835b0455de, State: Initialized, Role: FOLLOWER
I20260809 12:15:35.298190 23946 consensus_queue.cc:260] T 00000000000000000000000000000000 P 41a1ee0c5e354eecbc4c18835b0455de [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "41a1ee0c5e354eecbc4c18835b0455de" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 40599 } }
I20260809 12:15:35.298441 23946 raft_consensus.cc:388] T 00000000000000000000000000000000 P 41a1ee0c5e354eecbc4c18835b0455de [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260809 12:15:35.298600 23946 raft_consensus.cc:482] T 00000000000000000000000000000000 P 41a1ee0c5e354eecbc4c18835b0455de [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260809 12:15:35.298782 23946 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 41a1ee0c5e354eecbc4c18835b0455de [term 0 FOLLOWER]: Advancing to term 1
I20260809 12:15:35.300961 23946 raft_consensus.cc:504] T 00000000000000000000000000000000 P 41a1ee0c5e354eecbc4c18835b0455de [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "41a1ee0c5e354eecbc4c18835b0455de" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 40599 } }
I20260809 12:15:35.301450 23946 leader_election.cc:302] T 00000000000000000000000000000000 P 41a1ee0c5e354eecbc4c18835b0455de [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 41a1ee0c5e354eecbc4c18835b0455de; no voters:
I20260809 12:15:35.302083 23946 leader_election.cc:288] T 00000000000000000000000000000000 P 41a1ee0c5e354eecbc4c18835b0455de [CANDIDATE]: Term 1 election: Requested vote from peers
I20260809 12:15:35.302254 23957 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 41a1ee0c5e354eecbc4c18835b0455de [term 1 FOLLOWER]: Leader election won for term 1
I20260809 12:15:35.303613 23957 raft_consensus.cc:686] T 00000000000000000000000000000000 P 41a1ee0c5e354eecbc4c18835b0455de [term 1 LEADER]: Becoming Leader. State: Replica: 41a1ee0c5e354eecbc4c18835b0455de, State: Running, Role: LEADER
I20260809 12:15:35.304200 23957 consensus_queue.cc:237] T 00000000000000000000000000000000 P 41a1ee0c5e354eecbc4c18835b0455de [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "41a1ee0c5e354eecbc4c18835b0455de" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 40599 } }
I20260809 12:15:35.304443 23946 sys_catalog.cc:564] T 00000000000000000000000000000000 P 41a1ee0c5e354eecbc4c18835b0455de [sys.catalog]: configured and running, proceeding with master startup.
I20260809 12:15:35.307286 23958 sys_catalog.cc:455] T 00000000000000000000000000000000 P 41a1ee0c5e354eecbc4c18835b0455de [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "41a1ee0c5e354eecbc4c18835b0455de" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "41a1ee0c5e354eecbc4c18835b0455de" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 40599 } } }
I20260809 12:15:35.307684 23958 sys_catalog.cc:458] T 00000000000000000000000000000000 P 41a1ee0c5e354eecbc4c18835b0455de [sys.catalog]: This master's current role is: LEADER
I20260809 12:15:35.306980 23959 sys_catalog.cc:455] T 00000000000000000000000000000000 P 41a1ee0c5e354eecbc4c18835b0455de [sys.catalog]: SysCatalogTable state changed. Reason: New leader 41a1ee0c5e354eecbc4c18835b0455de. Latest consensus state: current_term: 1 leader_uuid: "41a1ee0c5e354eecbc4c18835b0455de" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "41a1ee0c5e354eecbc4c18835b0455de" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 40599 } } }
I20260809 12:15:35.308324 23959 sys_catalog.cc:458] T 00000000000000000000000000000000 P 41a1ee0c5e354eecbc4c18835b0455de [sys.catalog]: This master's current role is: LEADER
I20260809 12:15:35.309970 23966 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260809 12:15:35.319015 23966 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260809 12:15:35.328811 23966 catalog_manager.cc:1347] Generated new cluster ID: a79d226f2bf64872a8f7aa36e9a7d6d2
I20260809 12:15:35.328936 23966 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260809 12:15:35.349267 23966 catalog_manager.cc:1370] Generated new certificate authority record
I20260809 12:15:35.349752 23966 catalog_manager.cc:1504] Loading token signing keys...
I20260809 12:15:35.358806 23966 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 41a1ee0c5e354eecbc4c18835b0455de: Generated new TSK 0
I20260809 12:15:35.359843 23966 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260809 12:15:35.365106 888 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskgH5Pjm/build/asan/bin/kudu pid 23874
W20260809 12:15:35.366509 23941 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='slave'} at 10.240.0.84:35323
W20260809 12:15:35.367233 888 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskgH5Pjm/build/asan/bin/kudu pid 23874: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260809 12:15:35.367379 888 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskgH5Pjm/build/asan/bin/kudu with pid 23874
2026-08-09T12:15:35Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/8 (2076 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/9
Loading random data
Initializing database '/tmp/dist-test-taskgH5Pjm/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1786277382073938-888-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 09 12:15:35 dist-test-slave-2w3w krb5kdc[23980](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 09 12:15:35 dist-test-slave-2w3w krb5kdc[23980](info): set up 2 sockets
Aug 09 12:15:35 dist-test-slave-2w3w krb5kdc[23980](info): commencing operation
krb5kdc: starting...
W20260809 12:15:36.552163 888 mini_kdc.cc:121] Time spent starting KDC: real 1.143s user 0.000s sys 0.049s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskgH5Pjm/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1786277382073938-888-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskgH5Pjm/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1786277382073938-888-0/krb5kdc/test-user.keytab.
Aug 09 12:15:36 dist-test-slave-2w3w krb5kdc[23980](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786277736, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-09T12:15:36Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-09T12:15:36Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskgH5Pjm/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1786277382073938-888-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskgH5Pjm/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1786277382073938-888-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260809 12:15:36.896695 888 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskgH5Pjm/build/asan/bin/kudu
/tmp/dist-test-taskgH5Pjm/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskgH5Pjm/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1786277382073938-888-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskgH5Pjm/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1786277382073938-888-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskgH5Pjm/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1786277382073938-888-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskgH5Pjm/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1786277382073938-888-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:37935
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskgH5Pjm/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1786277382073938-888-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:32903
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:37935
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=required
--rpc_encryption=required
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taskgH5Pjm/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1786277382073938-888-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskgH5Pjm/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1786277382073938-888-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskgH5Pjm/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1786277382073938-888-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260809 12:15:37.254521 23996 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260809 12:15:37.255012 23996 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260809 12:15:37.255195 23996 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260809 12:15:37.267350 23996 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260809 12:15:37.267510 23996 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260809 12:15:37.267634 23996 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260809 12:15:37.267747 23996 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260809 12:15:37.267838 23996 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260809 12:15:37.286125 23996 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:32903
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskgH5Pjm/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1786277382073938-888-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskgH5Pjm/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1786277382073938-888-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:37935
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskgH5Pjm/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1786277382073938-888-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:37935
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskgH5Pjm/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1786277382073938-888-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskgH5Pjm/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1786277382073938-888-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 09 Aug 2026 11:06:41 UTC on ccycloud.quasar-bwoogy.root.comops.site
build id 4575991
ASAN enabled
I20260809 12:15:37.288125 23996 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260809 12:15:37.290135 23996 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260809 12:15:37.304569 24003 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 12:15:37.304713 24006 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 12:15:37.304996 24002 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260809 12:15:37.305451 23996 server_base.cc:1034] running on GCE node
I20260809 12:15:37.306267 23996 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260809 12:15:37.307781 23996 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260809 12:15:37.308990 23996 hybrid_clock.cc:648] HybridClock initialized: now 1786277737308950 us; error 27 us; skew 500 ppm
I20260809 12:15:37.309382 23996 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 09 12:15:37 dist-test-slave-2w3w krb5kdc[23980](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786277737, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260809 12:15:37.312713 23996 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260809 12:15:37.314267 23996 webserver.cc:460] Webserver started at http://127.0.0.1:43871/ using document root <none> and password file <none>
I20260809 12:15:37.315039 23996 fs_manager.cc:359] Metadata directory not provided
I20260809 12:15:37.315196 23996 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260809 12:15:37.315527 23996 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260809 12:15:37.318667 23996 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskgH5Pjm/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1786277382073938-888-0/minicluster-data/master-0/data/instance:
uuid: "3e2538062c47474f859ad7b957ebcdb5"
format_stamp: "Formatted at 2026-08-09 12:15:37 on dist-test-slave-2w3w"
server_key: "da9e84e2e4b1eb67cd435b48634b6427"
server_key_iv: "ea3cd5a1b8f2cc5ce8f6708c53e7cbb3"
server_key_version: "encryptionkey@0"
I20260809 12:15:37.319552 23996 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskgH5Pjm/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1786277382073938-888-0/minicluster-data/master-0/wal/instance:
uuid: "3e2538062c47474f859ad7b957ebcdb5"
format_stamp: "Formatted at 2026-08-09 12:15:37 on dist-test-slave-2w3w"
server_key: "da9e84e2e4b1eb67cd435b48634b6427"
server_key_iv: "ea3cd5a1b8f2cc5ce8f6708c53e7cbb3"
server_key_version: "encryptionkey@0"
I20260809 12:15:37.324860 23996 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.001s sys 0.004s
I20260809 12:15:37.328392 24014 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260809 12:15:37.329635 23996 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260809 12:15:37.329824 23996 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskgH5Pjm/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1786277382073938-888-0/minicluster-data/master-0/data,/tmp/dist-test-taskgH5Pjm/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1786277382073938-888-0/minicluster-data/master-0/wal
uuid: "3e2538062c47474f859ad7b957ebcdb5"
format_stamp: "Formatted at 2026-08-09 12:15:37 on dist-test-slave-2w3w"
server_key: "da9e84e2e4b1eb67cd435b48634b6427"
server_key_iv: "ea3cd5a1b8f2cc5ce8f6708c53e7cbb3"
server_key_version: "encryptionkey@0"
I20260809 12:15:37.330035 23996 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskgH5Pjm/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1786277382073938-888-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskgH5Pjm/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1786277382073938-888-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskgH5Pjm/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1786277382073938-888-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260809 12:15:37.361804 23996 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260809 12:15:37.366540 23996 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260809 12:15:37.366855 23996 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260809 12:15:37.386368 23996 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:37935
I20260809 12:15:37.386405 24069 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:37935 every 8 connection(s)
I20260809 12:15:37.387794 23996 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskgH5Pjm/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1786277382073938-888-0/minicluster-data/master-0/data/info.pb
I20260809 12:15:37.391644 24070 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260809 12:15:37.391661 888 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskgH5Pjm/build/asan/bin/kudu as pid 23996
I20260809 12:15:37.391875 888 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskgH5Pjm/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1786277382073938-888-0/minicluster-data/master-0/wal/instance
I20260809 12:15:37.392323 888 external_mini_cluster.cc:1428] Setting key f0b4aec8ce9bc14de769716249614e0d
I20260809 12:15:37.406868 24070 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 3e2538062c47474f859ad7b957ebcdb5: Bootstrap starting.
Aug 09 12:15:37 dist-test-slave-2w3w krb5kdc[23980](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786277736, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.0.1@KRBTEST.COM
I20260809 12:15:37.411144 24070 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 3e2538062c47474f859ad7b957ebcdb5: Neither blocks nor log segments found. Creating new log.
I20260809 12:15:37.412598 24070 log.cc:824] T 00000000000000000000000000000000 P 3e2538062c47474f859ad7b957ebcdb5: Log is configured to *not* fsync() on all Append() calls
I20260809 12:15:37.416360 24070 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 3e2538062c47474f859ad7b957ebcdb5: No bootstrap required, opened a new log
I20260809 12:15:37.417879 24073 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 12:15:37.394282 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.84:40917 (local address 127.0.0.1:37935)
0809 12:15:37.395180 (+ 898us) server_negotiation.cc:206] Beginning negotiation
0809 12:15:37.395197 (+ 17us) server_negotiation.cc:399] Waiting for connection header
0809 12:15:37.395247 (+ 50us) server_negotiation.cc:407] Connection header received
0809 12:15:37.396884 (+ 1637us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 12:15:37.396914 (+ 30us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 12:15:37.397482 (+ 568us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 12:15:37.397947 (+ 465us) server_negotiation.cc:226] Negotiated authn=SASL
0809 12:15:37.399299 (+ 1352us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 12:15:37.401641 (+ 2342us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 12:15:37.402993 (+ 1352us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 12:15:37.404129 (+ 1136us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 12:15:37.404330 (+ 201us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 12:15:37.408587 (+ 4257us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 12:15:37.408600 (+ 13us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 12:15:37.408623 (+ 23us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0809 12:15:37.408689 (+ 66us) server_negotiation.cc:922] calling sasl_server_start()
0809 12:15:37.412880 (+ 4191us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 12:15:37.413907 (+ 1027us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 12:15:37.413914 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 12:15:37.413921 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0809 12:15:37.414127 (+ 206us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 12:15:37.414612 (+ 485us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 12:15:37.414618 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 12:15:37.414624 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0809 12:15:37.415354 (+ 730us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 12:15:37.415709 (+ 355us) server_negotiation.cc:1035] Waiting for connection context
0809 12:15:37.416126 (+ 417us) server_negotiation.cc:299] Negotiation successful
0809 12:15:37.416473 (+ 347us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":625,"thread_start_us":368,"threads_started":1}
I20260809 12:15:37.422061 888 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260809 12:15:37.425660 24070 raft_consensus.cc:348] T 00000000000000000000000000000000 P 3e2538062c47474f859ad7b957ebcdb5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3e2538062c47474f859ad7b957ebcdb5" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 37935 } }
I20260809 12:15:37.426180 24070 raft_consensus.cc:374] T 00000000000000000000000000000000 P 3e2538062c47474f859ad7b957ebcdb5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260809 12:15:37.426306 24070 raft_consensus.cc:729] T 00000000000000000000000000000000 P 3e2538062c47474f859ad7b957ebcdb5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3e2538062c47474f859ad7b957ebcdb5, State: Initialized, Role: FOLLOWER
I20260809 12:15:37.427183 24070 consensus_queue.cc:260] T 00000000000000000000000000000000 P 3e2538062c47474f859ad7b957ebcdb5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3e2538062c47474f859ad7b957ebcdb5" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 37935 } }
I20260809 12:15:37.427527 24070 raft_consensus.cc:388] T 00000000000000000000000000000000 P 3e2538062c47474f859ad7b957ebcdb5 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260809 12:15:37.427721 24070 raft_consensus.cc:482] T 00000000000000000000000000000000 P 3e2538062c47474f859ad7b957ebcdb5 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260809 12:15:37.427944 24070 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 3e2538062c47474f859ad7b957ebcdb5 [term 0 FOLLOWER]: Advancing to term 1
I20260809 12:15:37.430665 24070 raft_consensus.cc:504] T 00000000000000000000000000000000 P 3e2538062c47474f859ad7b957ebcdb5 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3e2538062c47474f859ad7b957ebcdb5" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 37935 } }
I20260809 12:15:37.431342 24070 leader_election.cc:302] T 00000000000000000000000000000000 P 3e2538062c47474f859ad7b957ebcdb5 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 3e2538062c47474f859ad7b957ebcdb5; no voters:
I20260809 12:15:37.432144 24070 leader_election.cc:288] T 00000000000000000000000000000000 P 3e2538062c47474f859ad7b957ebcdb5 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260809 12:15:37.432324 24080 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 3e2538062c47474f859ad7b957ebcdb5 [term 1 FOLLOWER]: Leader election won for term 1
I20260809 12:15:37.434918 24070 sys_catalog.cc:564] T 00000000000000000000000000000000 P 3e2538062c47474f859ad7b957ebcdb5 [sys.catalog]: configured and running, proceeding with master startup.
I20260809 12:15:37.439653 24080 raft_consensus.cc:686] T 00000000000000000000000000000000 P 3e2538062c47474f859ad7b957ebcdb5 [term 1 LEADER]: Becoming Leader. State: Replica: 3e2538062c47474f859ad7b957ebcdb5, State: Running, Role: LEADER
I20260809 12:15:37.440506 24080 consensus_queue.cc:237] T 00000000000000000000000000000000 P 3e2538062c47474f859ad7b957ebcdb5 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3e2538062c47474f859ad7b957ebcdb5" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 37935 } }
I20260809 12:15:37.444213 24081 sys_catalog.cc:455] T 00000000000000000000000000000000 P 3e2538062c47474f859ad7b957ebcdb5 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "3e2538062c47474f859ad7b957ebcdb5" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3e2538062c47474f859ad7b957ebcdb5" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 37935 } } }
I20260809 12:15:37.444708 24081 sys_catalog.cc:458] T 00000000000000000000000000000000 P 3e2538062c47474f859ad7b957ebcdb5 [sys.catalog]: This master's current role is: LEADER
I20260809 12:15:37.445626 24082 sys_catalog.cc:455] T 00000000000000000000000000000000 P 3e2538062c47474f859ad7b957ebcdb5 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 3e2538062c47474f859ad7b957ebcdb5. Latest consensus state: current_term: 1 leader_uuid: "3e2538062c47474f859ad7b957ebcdb5" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3e2538062c47474f859ad7b957ebcdb5" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 37935 } } }
I20260809 12:15:37.446040 24082 sys_catalog.cc:458] T 00000000000000000000000000000000 P 3e2538062c47474f859ad7b957ebcdb5 [sys.catalog]: This master's current role is: LEADER
I20260809 12:15:37.446127 24073 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 12:15:37.427937 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.84:60641 (local address 127.0.0.1:37935)
0809 12:15:37.428327 (+ 390us) server_negotiation.cc:206] Beginning negotiation
0809 12:15:37.428333 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0809 12:15:37.429086 (+ 753us) server_negotiation.cc:407] Connection header received
0809 12:15:37.429312 (+ 226us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 12:15:37.429318 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 12:15:37.429402 (+ 84us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 12:15:37.429524 (+ 122us) server_negotiation.cc:226] Negotiated authn=SASL
0809 12:15:37.431210 (+ 1686us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 12:15:37.432117 (+ 907us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 12:15:37.433917 (+ 1800us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 12:15:37.434588 (+ 671us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 12:15:37.434770 (+ 182us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 12:15:37.438109 (+ 3339us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 12:15:37.438114 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 12:15:37.438118 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0809 12:15:37.438151 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0809 12:15:37.441410 (+ 3259us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 12:15:37.442381 (+ 971us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 12:15:37.442385 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 12:15:37.442387 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0809 12:15:37.442513 (+ 126us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 12:15:37.443220 (+ 707us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 12:15:37.443225 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 12:15:37.443229 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0809 12:15:37.443732 (+ 503us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 12:15:37.444058 (+ 326us) server_negotiation.cc:1035] Waiting for connection context
0809 12:15:37.445276 (+ 1218us) server_negotiation.cc:299] Negotiation successful
0809 12:15:37.445517 (+ 241us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":238}
I20260809 12:15:37.455541 24089 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260809 12:15:37.460579 24089 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260809 12:15:37.469307 24089 catalog_manager.cc:1347] Generated new cluster ID: 344eebe506f14757a0874f59fda9b4db
I20260809 12:15:37.469434 24089 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260809 12:15:37.515997 24089 catalog_manager.cc:1370] Generated new certificate authority record
I20260809 12:15:37.517380 24089 catalog_manager.cc:1504] Loading token signing keys...
I20260809 12:15:37.524322 24089 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 3e2538062c47474f859ad7b957ebcdb5: Generated new TSK 0
I20260809 12:15:37.525250 24089 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260809 12:15:37.540144 888 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskgH5Pjm/build/asan/bin/kudu pid 23996
I20260809 12:15:37.542173 24065 generic_service.cc:196] Checking for leaks (request via RPC)
I20260809 12:15:37.641901 888 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskgH5Pjm/build/asan/bin/kudu with pid 23996
W20260809 12:15:37.664551 23978 connection.cc:535] client connection to 127.0.0.1:37935 recv error: Network error: failed to read from TLS socket (remote: 127.0.0.1:37935): Connection reset by peer (error 104)
2026-08-09T12:15:37Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/9 (2280 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/10
Loading random data
Initializing database '/tmp/dist-test-taskgH5Pjm/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1786277382073938-888-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 09 12:15:37 dist-test-slave-2w3w krb5kdc[24105](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 09 12:15:37 dist-test-slave-2w3w krb5kdc[24105](info): set up 2 sockets
Aug 09 12:15:37 dist-test-slave-2w3w krb5kdc[24105](info): commencing operation
krb5kdc: starting...
W20260809 12:15:38.832754 888 mini_kdc.cc:121] Time spent starting KDC: real 1.144s user 0.000s sys 0.050s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskgH5Pjm/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1786277382073938-888-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskgH5Pjm/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1786277382073938-888-0/krb5kdc/test-user.keytab.
Aug 09 12:15:39 dist-test-slave-2w3w krb5kdc[24105](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786277739, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-09T12:15:39Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-09T12:15:39Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskgH5Pjm/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1786277382073938-888-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskgH5Pjm/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1786277382073938-888-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260809 12:15:39.175643 888 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskgH5Pjm/build/asan/bin/kudu
/tmp/dist-test-taskgH5Pjm/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskgH5Pjm/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1786277382073938-888-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskgH5Pjm/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1786277382073938-888-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskgH5Pjm/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1786277382073938-888-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskgH5Pjm/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1786277382073938-888-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:35175
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskgH5Pjm/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1786277382073938-888-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:43285
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:35175
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=required
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taskgH5Pjm/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1786277382073938-888-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskgH5Pjm/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1786277382073938-888-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskgH5Pjm/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1786277382073938-888-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260809 12:15:39.529788 24119 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260809 12:15:39.530082 24119 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260809 12:15:39.530256 24119 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260809 12:15:39.537488 24119 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260809 12:15:39.537633 24119 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260809 12:15:39.537747 24119 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260809 12:15:39.537855 24119 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260809 12:15:39.537945 24119 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260809 12:15:39.548228 24119 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:43285
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskgH5Pjm/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1786277382073938-888-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskgH5Pjm/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1786277382073938-888-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:35175
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskgH5Pjm/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1786277382073938-888-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:35175
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskgH5Pjm/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1786277382073938-888-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskgH5Pjm/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1786277382073938-888-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 09 Aug 2026 11:06:41 UTC on ccycloud.quasar-bwoogy.root.comops.site
build id 4575991
ASAN enabled
I20260809 12:15:39.549582 24119 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260809 12:15:39.550900 24119 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260809 12:15:39.563081 24125 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 12:15:39.562704 24126 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 12:15:39.563122 24128 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260809 12:15:39.564378 24119 server_base.cc:1034] running on GCE node
I20260809 12:15:39.565191 24119 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260809 12:15:39.566540 24119 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260809 12:15:39.567741 24119 hybrid_clock.cc:648] HybridClock initialized: now 1786277739567700 us; error 44 us; skew 500 ppm
I20260809 12:15:39.568111 24119 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 09 12:15:39 dist-test-slave-2w3w krb5kdc[24105](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786277739, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260809 12:15:39.576812 24119 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260809 12:15:39.578437 24119 webserver.cc:460] Webserver started at http://127.0.0.1:34741/ using document root <none> and password file <none>
I20260809 12:15:39.579216 24119 fs_manager.cc:359] Metadata directory not provided
I20260809 12:15:39.579365 24119 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260809 12:15:39.579684 24119 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260809 12:15:39.582731 24119 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskgH5Pjm/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1786277382073938-888-0/minicluster-data/master-0/data/instance:
uuid: "8712e61d371242c8b8b830b0837f853b"
format_stamp: "Formatted at 2026-08-09 12:15:39 on dist-test-slave-2w3w"
server_key: "1eac60b53bc94440944746253fa2b017"
server_key_iv: "f8b8fc0470b50fe0fe4be055a02b7edd"
server_key_version: "encryptionkey@0"
I20260809 12:15:39.583603 24119 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskgH5Pjm/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1786277382073938-888-0/minicluster-data/master-0/wal/instance:
uuid: "8712e61d371242c8b8b830b0837f853b"
format_stamp: "Formatted at 2026-08-09 12:15:39 on dist-test-slave-2w3w"
server_key: "1eac60b53bc94440944746253fa2b017"
server_key_iv: "f8b8fc0470b50fe0fe4be055a02b7edd"
server_key_version: "encryptionkey@0"
I20260809 12:15:39.589437 24119 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.001s sys 0.004s
I20260809 12:15:39.593139 24136 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260809 12:15:39.594563 24119 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.000s sys 0.002s
I20260809 12:15:39.594781 24119 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskgH5Pjm/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1786277382073938-888-0/minicluster-data/master-0/data,/tmp/dist-test-taskgH5Pjm/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1786277382073938-888-0/minicluster-data/master-0/wal
uuid: "8712e61d371242c8b8b830b0837f853b"
format_stamp: "Formatted at 2026-08-09 12:15:39 on dist-test-slave-2w3w"
server_key: "1eac60b53bc94440944746253fa2b017"
server_key_iv: "f8b8fc0470b50fe0fe4be055a02b7edd"
server_key_version: "encryptionkey@0"
I20260809 12:15:39.595063 24119 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskgH5Pjm/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1786277382073938-888-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskgH5Pjm/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1786277382073938-888-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskgH5Pjm/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1786277382073938-888-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260809 12:15:39.616767 24119 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260809 12:15:39.621256 24119 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260809 12:15:39.621567 24119 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260809 12:15:39.641319 24119 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:35175
I20260809 12:15:39.641360 24195 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:35175 every 8 connection(s)
I20260809 12:15:39.642758 24119 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskgH5Pjm/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1786277382073938-888-0/minicluster-data/master-0/data/info.pb
I20260809 12:15:39.646847 24196 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260809 12:15:39.649541 888 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskgH5Pjm/build/asan/bin/kudu as pid 24119
I20260809 12:15:39.649750 888 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskgH5Pjm/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1786277382073938-888-0/minicluster-data/master-0/wal/instance
I20260809 12:15:39.650226 888 external_mini_cluster.cc:1428] Setting key 34864a9f11e36e6abe6d6c0f15889a3d
I20260809 12:15:39.658516 24196 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 8712e61d371242c8b8b830b0837f853b: Bootstrap starting.
I20260809 12:15:39.661967 24196 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 8712e61d371242c8b8b830b0837f853b: Neither blocks nor log segments found. Creating new log.
I20260809 12:15:39.663321 24196 log.cc:824] T 00000000000000000000000000000000 P 8712e61d371242c8b8b830b0837f853b: Log is configured to *not* fsync() on all Append() calls
I20260809 12:15:39.664431 24199 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 12:15:39.652304 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.84:33965 (local address 127.0.0.1:35175)
0809 12:15:39.653114 (+ 810us) server_negotiation.cc:206] Beginning negotiation
0809 12:15:39.653126 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0809 12:15:39.653165 (+ 39us) server_negotiation.cc:407] Connection header received
0809 12:15:39.654894 (+ 1729us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 12:15:39.654918 (+ 24us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 12:15:39.655468 (+ 550us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 12:15:39.655887 (+ 419us) server_negotiation.cc:226] Negotiated authn=SASL
0809 12:15:39.656461 (+ 574us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 12:15:39.657810 (+ 1349us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 12:15:39.659308 (+ 1498us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 12:15:39.660007 (+ 699us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 12:15:39.660215 (+ 208us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 12:15:39.661650 (+ 1435us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 12:15:39.661658 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 12:15:39.661672 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0809 12:15:39.662308 (+ 636us) server_negotiation.cc:922] calling sasl_server_start()
0809 12:15:39.662401 (+ 93us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0809 12:15:39.662431 (+ 30us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 12:15:39.662622 (+ 191us) server_negotiation.cc:1035] Waiting for connection context
0809 12:15:39.662994 (+ 372us) server_negotiation.cc:299] Negotiation successful
0809 12:15:39.663161 (+ 167us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":587,"thread_start_us":329,"threads_started":1}
I20260809 12:15:39.666591 24196 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 8712e61d371242c8b8b830b0837f853b: No bootstrap required, opened a new log
I20260809 12:15:39.668560 888 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260809 12:15:39.675205 24196 raft_consensus.cc:348] T 00000000000000000000000000000000 P 8712e61d371242c8b8b830b0837f853b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8712e61d371242c8b8b830b0837f853b" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 35175 } }
I20260809 12:15:39.675712 24196 raft_consensus.cc:374] T 00000000000000000000000000000000 P 8712e61d371242c8b8b830b0837f853b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260809 12:15:39.675827 24196 raft_consensus.cc:729] T 00000000000000000000000000000000 P 8712e61d371242c8b8b830b0837f853b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8712e61d371242c8b8b830b0837f853b, State: Initialized, Role: FOLLOWER
I20260809 12:15:39.676506 24196 consensus_queue.cc:260] T 00000000000000000000000000000000 P 8712e61d371242c8b8b830b0837f853b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8712e61d371242c8b8b830b0837f853b" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 35175 } }
I20260809 12:15:39.676774 24196 raft_consensus.cc:388] T 00000000000000000000000000000000 P 8712e61d371242c8b8b830b0837f853b [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260809 12:15:39.676932 24196 raft_consensus.cc:482] T 00000000000000000000000000000000 P 8712e61d371242c8b8b830b0837f853b [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260809 12:15:39.677105 24196 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 8712e61d371242c8b8b830b0837f853b [term 0 FOLLOWER]: Advancing to term 1
I20260809 12:15:39.680270 24199 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 12:15:39.673270 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.84:60775 (local address 127.0.0.1:35175)
0809 12:15:39.673497 (+ 227us) server_negotiation.cc:206] Beginning negotiation
0809 12:15:39.673504 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0809 12:15:39.673763 (+ 259us) server_negotiation.cc:407] Connection header received
0809 12:15:39.673969 (+ 206us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 12:15:39.673975 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 12:15:39.674061 (+ 86us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 12:15:39.674177 (+ 116us) server_negotiation.cc:226] Negotiated authn=SASL
0809 12:15:39.674734 (+ 557us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 12:15:39.675916 (+ 1182us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 12:15:39.677523 (+ 1607us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 12:15:39.678230 (+ 707us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 12:15:39.678384 (+ 154us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 12:15:39.679503 (+ 1119us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 12:15:39.679510 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 12:15:39.679515 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0809 12:15:39.679536 (+ 21us) server_negotiation.cc:922] calling sasl_server_start()
0809 12:15:39.679601 (+ 65us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0809 12:15:39.679621 (+ 20us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 12:15:39.679731 (+ 110us) server_negotiation.cc:1035] Waiting for connection context
0809 12:15:39.679897 (+ 166us) server_negotiation.cc:299] Negotiation successful
0809 12:15:39.679951 (+ 54us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":57}
I20260809 12:15:39.680546 24196 raft_consensus.cc:504] T 00000000000000000000000000000000 P 8712e61d371242c8b8b830b0837f853b [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8712e61d371242c8b8b830b0837f853b" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 35175 } }
I20260809 12:15:39.681157 24196 leader_election.cc:302] T 00000000000000000000000000000000 P 8712e61d371242c8b8b830b0837f853b [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 8712e61d371242c8b8b830b0837f853b; no voters:
I20260809 12:15:39.681841 24196 leader_election.cc:288] T 00000000000000000000000000000000 P 8712e61d371242c8b8b830b0837f853b [CANDIDATE]: Term 1 election: Requested vote from peers
I20260809 12:15:39.681962 24207 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 8712e61d371242c8b8b830b0837f853b [term 1 FOLLOWER]: Leader election won for term 1
I20260809 12:15:39.684047 24207 raft_consensus.cc:686] T 00000000000000000000000000000000 P 8712e61d371242c8b8b830b0837f853b [term 1 LEADER]: Becoming Leader. State: Replica: 8712e61d371242c8b8b830b0837f853b, State: Running, Role: LEADER
I20260809 12:15:39.684301 24196 sys_catalog.cc:564] T 00000000000000000000000000000000 P 8712e61d371242c8b8b830b0837f853b [sys.catalog]: configured and running, proceeding with master startup.
I20260809 12:15:39.684742 24207 consensus_queue.cc:237] T 00000000000000000000000000000000 P 8712e61d371242c8b8b830b0837f853b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8712e61d371242c8b8b830b0837f853b" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 35175 } }
I20260809 12:15:39.688844 24210 sys_catalog.cc:455] T 00000000000000000000000000000000 P 8712e61d371242c8b8b830b0837f853b [sys.catalog]: SysCatalogTable state changed. Reason: New leader 8712e61d371242c8b8b830b0837f853b. Latest consensus state: current_term: 1 leader_uuid: "8712e61d371242c8b8b830b0837f853b" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8712e61d371242c8b8b830b0837f853b" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 35175 } } }
I20260809 12:15:39.688879 24208 sys_catalog.cc:455] T 00000000000000000000000000000000 P 8712e61d371242c8b8b830b0837f853b [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "8712e61d371242c8b8b830b0837f853b" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8712e61d371242c8b8b830b0837f853b" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 35175 } } }
I20260809 12:15:39.689424 24210 sys_catalog.cc:458] T 00000000000000000000000000000000 P 8712e61d371242c8b8b830b0837f853b [sys.catalog]: This master's current role is: LEADER
I20260809 12:15:39.689421 24208 sys_catalog.cc:458] T 00000000000000000000000000000000 P 8712e61d371242c8b8b830b0837f853b [sys.catalog]: This master's current role is: LEADER
W20260809 12:15:39.699131 24225 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 8712e61d371242c8b8b830b0837f853b: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260809 12:15:39.699244 24225 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260809 12:15:39.699383 24218 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260809 12:15:39.701869 24218 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260809 12:15:39.711134 24218 catalog_manager.cc:1347] Generated new cluster ID: c79539d77ec5465f89769bae0cc6b21a
I20260809 12:15:39.711277 24218 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260809 12:15:39.735386 24218 catalog_manager.cc:1370] Generated new certificate authority record
I20260809 12:15:39.736714 24218 catalog_manager.cc:1504] Loading token signing keys...
I20260809 12:15:39.748283 24218 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 8712e61d371242c8b8b830b0837f853b: Generated new TSK 0
I20260809 12:15:39.749150 24218 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260809 12:15:39.760603 888 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskgH5Pjm/build/asan/bin/kudu pid 24119
W20260809 12:15:39.761830 24190 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='slave'} at 10.240.0.84:33965
W20260809 12:15:39.762631 888 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskgH5Pjm/build/asan/bin/kudu pid 24119: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260809 12:15:39.762786 888 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskgH5Pjm/build/asan/bin/kudu with pid 24119
W20260809 12:15:39.784179 24103 connection.cc:535] client connection to 127.0.0.1:35175 recv error: Network error: failed to read from TLS socket (remote: 127.0.0.1:35175): Connection reset by peer (error 104)
2026-08-09T12:15:39Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/10 (2114 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/11
Loading random data
Initializing database '/tmp/dist-test-taskgH5Pjm/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1786277382073938-888-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 09 12:15:39 dist-test-slave-2w3w krb5kdc[24230](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 09 12:15:39 dist-test-slave-2w3w krb5kdc[24230](info): set up 2 sockets
Aug 09 12:15:39 dist-test-slave-2w3w krb5kdc[24230](info): commencing operation
krb5kdc: starting...
W20260809 12:15:40.949064 888 mini_kdc.cc:121] Time spent starting KDC: real 1.146s user 0.000s sys 0.050s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskgH5Pjm/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1786277382073938-888-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskgH5Pjm/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1786277382073938-888-0/krb5kdc/test-user.keytab.
Aug 09 12:15:41 dist-test-slave-2w3w krb5kdc[24230](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786277741, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-09T12:15:41Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-09T12:15:41Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskgH5Pjm/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1786277382073938-888-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskgH5Pjm/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1786277382073938-888-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260809 12:15:41.290681 888 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskgH5Pjm/build/asan/bin/kudu
/tmp/dist-test-taskgH5Pjm/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskgH5Pjm/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1786277382073938-888-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskgH5Pjm/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1786277382073938-888-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskgH5Pjm/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1786277382073938-888-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskgH5Pjm/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1786277382073938-888-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:42277
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskgH5Pjm/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1786277382073938-888-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:38717
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:42277
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=disabled
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taskgH5Pjm/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1786277382073938-888-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskgH5Pjm/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1786277382073938-888-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskgH5Pjm/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1786277382073938-888-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260809 12:15:41.648720 24244 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260809 12:15:41.649076 24244 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260809 12:15:41.649272 24244 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260809 12:15:41.656728 24244 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260809 12:15:41.656883 24244 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260809 12:15:41.656996 24244 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260809 12:15:41.657104 24244 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260809 12:15:41.657193 24244 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260809 12:15:41.667413 24244 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:38717
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskgH5Pjm/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1786277382073938-888-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskgH5Pjm/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1786277382073938-888-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:42277
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskgH5Pjm/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1786277382073938-888-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:42277
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskgH5Pjm/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1786277382073938-888-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskgH5Pjm/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1786277382073938-888-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 09 Aug 2026 11:06:41 UTC on ccycloud.quasar-bwoogy.root.comops.site
build id 4575991
ASAN enabled
I20260809 12:15:41.668756 24244 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260809 12:15:41.670102 24244 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260809 12:15:41.682384 24249 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 12:15:41.682449 24250 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 12:15:41.682394 24252 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260809 12:15:41.683621 24244 server_base.cc:1034] running on GCE node
I20260809 12:15:41.684515 24244 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260809 12:15:41.685994 24244 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260809 12:15:41.687211 24244 hybrid_clock.cc:648] HybridClock initialized: now 1786277741687171 us; error 27 us; skew 500 ppm
I20260809 12:15:41.687552 24244 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 09 12:15:41 dist-test-slave-2w3w krb5kdc[24230](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786277741, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260809 12:15:41.690753 24244 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260809 12:15:41.692356 24244 webserver.cc:460] Webserver started at http://127.0.0.1:38249/ using document root <none> and password file <none>
I20260809 12:15:41.693060 24244 fs_manager.cc:359] Metadata directory not provided
I20260809 12:15:41.693198 24244 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260809 12:15:41.693528 24244 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260809 12:15:41.696612 24244 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskgH5Pjm/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1786277382073938-888-0/minicluster-data/master-0/data/instance:
uuid: "04bf1437519447f584bb7cc36db17e8a"
format_stamp: "Formatted at 2026-08-09 12:15:41 on dist-test-slave-2w3w"
server_key: "4449ec83d552efd2c7d2be2f0534be31"
server_key_iv: "f55a8cb34ee1103f9d07ed1d96b1a6c2"
server_key_version: "encryptionkey@0"
I20260809 12:15:41.697393 24244 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskgH5Pjm/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1786277382073938-888-0/minicluster-data/master-0/wal/instance:
uuid: "04bf1437519447f584bb7cc36db17e8a"
format_stamp: "Formatted at 2026-08-09 12:15:41 on dist-test-slave-2w3w"
server_key: "4449ec83d552efd2c7d2be2f0534be31"
server_key_iv: "f55a8cb34ee1103f9d07ed1d96b1a6c2"
server_key_version: "encryptionkey@0"
I20260809 12:15:41.702770 24244 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.002s sys 0.005s
I20260809 12:15:41.706660 24260 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260809 12:15:41.708029 24244 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260809 12:15:41.708213 24244 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskgH5Pjm/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1786277382073938-888-0/minicluster-data/master-0/data,/tmp/dist-test-taskgH5Pjm/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1786277382073938-888-0/minicluster-data/master-0/wal
uuid: "04bf1437519447f584bb7cc36db17e8a"
format_stamp: "Formatted at 2026-08-09 12:15:41 on dist-test-slave-2w3w"
server_key: "4449ec83d552efd2c7d2be2f0534be31"
server_key_iv: "f55a8cb34ee1103f9d07ed1d96b1a6c2"
server_key_version: "encryptionkey@0"
I20260809 12:15:41.708415 24244 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskgH5Pjm/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1786277382073938-888-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskgH5Pjm/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1786277382073938-888-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskgH5Pjm/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1786277382073938-888-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260809 12:15:41.714231 24244 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260809 12:15:41.718236 24244 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260809 12:15:41.718539 24244 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260809 12:15:41.737387 24244 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:42277
I20260809 12:15:41.737433 24314 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:42277 every 8 connection(s)
I20260809 12:15:41.738880 24244 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskgH5Pjm/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1786277382073938-888-0/minicluster-data/master-0/data/info.pb
I20260809 12:15:41.742643 24315 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260809 12:15:41.743238 888 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskgH5Pjm/build/asan/bin/kudu as pid 24244
I20260809 12:15:41.743420 888 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskgH5Pjm/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1786277382073938-888-0/minicluster-data/master-0/wal/instance
I20260809 12:15:41.744009 888 external_mini_cluster.cc:1428] Setting key 6e63c6a9ff78c5f8edf894052f1e941b
I20260809 12:15:41.752588 24318 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 12:15:41.745806 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.84:48361 (local address 127.0.0.1:42277)
0809 12:15:41.746655 (+ 849us) server_negotiation.cc:206] Beginning negotiation
0809 12:15:41.746666 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0809 12:15:41.746714 (+ 48us) server_negotiation.cc:407] Connection header received
0809 12:15:41.749079 (+ 2365us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 12:15:41.749102 (+ 23us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 12:15:41.749445 (+ 343us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 12:15:41.749778 (+ 333us) server_negotiation.cc:226] Negotiated authn=SASL
0809 12:15:41.750919 (+ 1141us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 12:15:41.750925 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 12:15:41.750938 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0809 12:15:41.750952 (+ 14us) server_negotiation.cc:922] calling sasl_server_start()
0809 12:15:41.751073 (+ 121us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0809 12:15:41.751095 (+ 22us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 12:15:41.751373 (+ 278us) server_negotiation.cc:1035] Waiting for connection context
0809 12:15:41.751521 (+ 148us) server_negotiation.cc:299] Negotiation successful
0809 12:15:41.751725 (+ 204us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":611,"thread_start_us":387,"threads_started":1}
I20260809 12:15:41.756089 24315 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 04bf1437519447f584bb7cc36db17e8a: Bootstrap starting.
I20260809 12:15:41.756317 888 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260809 12:15:41.759296 24315 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 04bf1437519447f584bb7cc36db17e8a: Neither blocks nor log segments found. Creating new log.
I20260809 12:15:41.760454 24315 log.cc:824] T 00000000000000000000000000000000 P 04bf1437519447f584bb7cc36db17e8a: Log is configured to *not* fsync() on all Append() calls
I20260809 12:15:41.763374 24318 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 12:15:41.760864 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.84:35069 (local address 127.0.0.1:42277)
0809 12:15:41.761066 (+ 202us) server_negotiation.cc:206] Beginning negotiation
0809 12:15:41.761073 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0809 12:15:41.761344 (+ 271us) server_negotiation.cc:407] Connection header received
0809 12:15:41.761552 (+ 208us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 12:15:41.761557 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 12:15:41.761614 (+ 57us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 12:15:41.761724 (+ 110us) server_negotiation.cc:226] Negotiated authn=SASL
0809 12:15:41.762642 (+ 918us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 12:15:41.762646 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 12:15:41.762649 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0809 12:15:41.762665 (+ 16us) server_negotiation.cc:922] calling sasl_server_start()
0809 12:15:41.762703 (+ 38us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0809 12:15:41.762717 (+ 14us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 12:15:41.762811 (+ 94us) server_negotiation.cc:1035] Waiting for connection context
0809 12:15:41.762951 (+ 140us) server_negotiation.cc:299] Negotiation successful
0809 12:15:41.763066 (+ 115us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":75}
I20260809 12:15:41.764341 24315 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 04bf1437519447f584bb7cc36db17e8a: No bootstrap required, opened a new log
I20260809 12:15:41.772356 24315 raft_consensus.cc:348] T 00000000000000000000000000000000 P 04bf1437519447f584bb7cc36db17e8a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "04bf1437519447f584bb7cc36db17e8a" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 42277 } }
I20260809 12:15:41.772706 24315 raft_consensus.cc:374] T 00000000000000000000000000000000 P 04bf1437519447f584bb7cc36db17e8a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260809 12:15:41.772778 24315 raft_consensus.cc:729] T 00000000000000000000000000000000 P 04bf1437519447f584bb7cc36db17e8a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 04bf1437519447f584bb7cc36db17e8a, State: Initialized, Role: FOLLOWER
I20260809 12:15:41.773291 24315 consensus_queue.cc:260] T 00000000000000000000000000000000 P 04bf1437519447f584bb7cc36db17e8a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "04bf1437519447f584bb7cc36db17e8a" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 42277 } }
I20260809 12:15:41.773502 24315 raft_consensus.cc:388] T 00000000000000000000000000000000 P 04bf1437519447f584bb7cc36db17e8a [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260809 12:15:41.773598 24315 raft_consensus.cc:482] T 00000000000000000000000000000000 P 04bf1437519447f584bb7cc36db17e8a [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260809 12:15:41.773712 24315 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 04bf1437519447f584bb7cc36db17e8a [term 0 FOLLOWER]: Advancing to term 1
I20260809 12:15:41.776407 24315 raft_consensus.cc:504] T 00000000000000000000000000000000 P 04bf1437519447f584bb7cc36db17e8a [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "04bf1437519447f584bb7cc36db17e8a" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 42277 } }
I20260809 12:15:41.776882 24315 leader_election.cc:302] T 00000000000000000000000000000000 P 04bf1437519447f584bb7cc36db17e8a [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 04bf1437519447f584bb7cc36db17e8a; no voters:
I20260809 12:15:41.777478 24315 leader_election.cc:288] T 00000000000000000000000000000000 P 04bf1437519447f584bb7cc36db17e8a [CANDIDATE]: Term 1 election: Requested vote from peers
I20260809 12:15:41.777608 24326 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 04bf1437519447f584bb7cc36db17e8a [term 1 FOLLOWER]: Leader election won for term 1
I20260809 12:15:41.778787 24326 raft_consensus.cc:686] T 00000000000000000000000000000000 P 04bf1437519447f584bb7cc36db17e8a [term 1 LEADER]: Becoming Leader. State: Replica: 04bf1437519447f584bb7cc36db17e8a, State: Running, Role: LEADER
I20260809 12:15:41.779335 24326 consensus_queue.cc:237] T 00000000000000000000000000000000 P 04bf1437519447f584bb7cc36db17e8a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "04bf1437519447f584bb7cc36db17e8a" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 42277 } }
I20260809 12:15:41.779533 24315 sys_catalog.cc:564] T 00000000000000000000000000000000 P 04bf1437519447f584bb7cc36db17e8a [sys.catalog]: configured and running, proceeding with master startup.
I20260809 12:15:41.782385 24328 sys_catalog.cc:455] T 00000000000000000000000000000000 P 04bf1437519447f584bb7cc36db17e8a [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "04bf1437519447f584bb7cc36db17e8a" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "04bf1437519447f584bb7cc36db17e8a" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 42277 } } }
I20260809 12:15:41.782351 24329 sys_catalog.cc:455] T 00000000000000000000000000000000 P 04bf1437519447f584bb7cc36db17e8a [sys.catalog]: SysCatalogTable state changed. Reason: New leader 04bf1437519447f584bb7cc36db17e8a. Latest consensus state: current_term: 1 leader_uuid: "04bf1437519447f584bb7cc36db17e8a" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "04bf1437519447f584bb7cc36db17e8a" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 42277 } } }
I20260809 12:15:41.782827 24328 sys_catalog.cc:458] T 00000000000000000000000000000000 P 04bf1437519447f584bb7cc36db17e8a [sys.catalog]: This master's current role is: LEADER
I20260809 12:15:41.782864 24329 sys_catalog.cc:458] T 00000000000000000000000000000000 P 04bf1437519447f584bb7cc36db17e8a [sys.catalog]: This master's current role is: LEADER
I20260809 12:15:41.786206 24333 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260809 12:15:41.793301 24333 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260809 12:15:41.803000 24333 catalog_manager.cc:1347] Generated new cluster ID: 528f88a6d3cc45ac80578aada5885639
I20260809 12:15:41.803138 24333 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260809 12:15:41.835531 24333 catalog_manager.cc:1370] Generated new certificate authority record
I20260809 12:15:41.836032 24333 catalog_manager.cc:1504] Loading token signing keys...
I20260809 12:15:41.842999 24333 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 04bf1437519447f584bb7cc36db17e8a: Generated new TSK 0
I20260809 12:15:41.843811 24333 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260809 12:15:41.855239 888 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskgH5Pjm/build/asan/bin/kudu pid 24244
W20260809 12:15:41.856510 24310 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='slave'} at 10.240.0.84:48361
W20260809 12:15:41.857283 888 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskgH5Pjm/build/asan/bin/kudu pid 24244: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260809 12:15:41.857457 888 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskgH5Pjm/build/asan/bin/kudu with pid 24244
2026-08-09T12:15:41Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/11 (2110 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/12
Loading random data
Initializing database '/tmp/dist-test-taskgH5Pjm/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1786277382073938-888-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 09 12:15:42 dist-test-slave-2w3w krb5kdc[24351](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 09 12:15:42 dist-test-slave-2w3w krb5kdc[24351](info): set up 2 sockets
Aug 09 12:15:42 dist-test-slave-2w3w krb5kdc[24351](info): commencing operation
krb5kdc: starting...
W20260809 12:15:43.070727 888 mini_kdc.cc:121] Time spent starting KDC: real 1.157s user 0.004s sys 0.045s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskgH5Pjm/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1786277382073938-888-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskgH5Pjm/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1786277382073938-888-0/krb5kdc/test-user.keytab.
Aug 09 12:15:43 dist-test-slave-2w3w krb5kdc[24351](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786277743, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-09T12:15:43Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-09T12:15:43Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.222.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.222.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.222.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskgH5Pjm/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1786277382073938-888-0/krb5kdc/kudu_127.0.222.62.keytab.
Entry for principal kudu/127.0.222.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskgH5Pjm/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1786277382073938-888-0/krb5kdc/kudu_127.0.222.62.keytab.
I20260809 12:15:43.402859 888 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskgH5Pjm/build/asan/bin/kudu
/tmp/dist-test-taskgH5Pjm/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskgH5Pjm/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1786277382073938-888-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskgH5Pjm/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1786277382073938-888-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskgH5Pjm/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1786277382073938-888-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskgH5Pjm/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1786277382073938-888-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.222.62:43965
--webserver_interface=127.0.222.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskgH5Pjm/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1786277382073938-888-0/krb5kdc/kudu_127.0.222.62.keytab
--principal=kudu/127.0.222.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.222.20:36327
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.222.62:43965
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=required
--rpc_encryption=required
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taskgH5Pjm/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1786277382073938-888-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskgH5Pjm/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1786277382073938-888-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskgH5Pjm/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1786277382073938-888-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260809 12:15:43.757882 24364 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260809 12:15:43.758221 24364 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260809 12:15:43.758406 24364 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260809 12:15:43.766016 24364 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260809 12:15:43.766120 24364 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260809 12:15:43.766167 24364 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260809 12:15:43.766242 24364 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260809 12:15:43.766287 24364 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260809 12:15:43.776322 24364 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.222.20:36327
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskgH5Pjm/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1786277382073938-888-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskgH5Pjm/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1786277382073938-888-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.222.62:43965
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskgH5Pjm/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1786277382073938-888-0/krb5kdc/kudu_127.0.222.62.keytab
--principal=kudu/127.0.222.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.222.62:43965
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskgH5Pjm/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1786277382073938-888-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.222.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskgH5Pjm/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1786277382073938-888-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 09 Aug 2026 11:06:41 UTC on ccycloud.quasar-bwoogy.root.comops.site
build id 4575991
ASAN enabled
I20260809 12:15:43.777590 24364 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260809 12:15:43.778837 24364 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260809 12:15:43.791041 24370 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 12:15:43.791007 24372 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 12:15:43.791226 24369 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260809 12:15:43.792212 24364 server_base.cc:1034] running on GCE node
I20260809 12:15:43.793005 24364 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260809 12:15:43.794421 24364 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260809 12:15:43.795667 24364 hybrid_clock.cc:648] HybridClock initialized: now 1786277743795621 us; error 33 us; skew 500 ppm
I20260809 12:15:43.796021 24364 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 09 12:15:43 dist-test-slave-2w3w krb5kdc[24351](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786277743, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.222.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260809 12:15:43.799377 24364 init.cc:377] Logged in from keytab as kudu/127.0.222.62@KRBTEST.COM (short username kudu)
I20260809 12:15:43.800889 24364 webserver.cc:460] Webserver started at http://127.0.222.62:38161/ using document root <none> and password file <none>
I20260809 12:15:43.801641 24364 fs_manager.cc:359] Metadata directory not provided
I20260809 12:15:43.801774 24364 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260809 12:15:43.802104 24364 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260809 12:15:43.805222 24364 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskgH5Pjm/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1786277382073938-888-0/minicluster-data/master-0/data/instance:
uuid: "cbb8e258757843b2bcd54a415f7dd460"
format_stamp: "Formatted at 2026-08-09 12:15:43 on dist-test-slave-2w3w"
server_key: "ce8f46945d03f541a2bef61aae19e7b3"
server_key_iv: "0bc8da94480eb79543ddd2ed56409834"
server_key_version: "encryptionkey@0"
I20260809 12:15:43.806027 24364 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskgH5Pjm/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1786277382073938-888-0/minicluster-data/master-0/wal/instance:
uuid: "cbb8e258757843b2bcd54a415f7dd460"
format_stamp: "Formatted at 2026-08-09 12:15:43 on dist-test-slave-2w3w"
server_key: "ce8f46945d03f541a2bef61aae19e7b3"
server_key_iv: "0bc8da94480eb79543ddd2ed56409834"
server_key_version: "encryptionkey@0"
I20260809 12:15:43.811338 24364 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.007s sys 0.000s
I20260809 12:15:43.814844 24379 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260809 12:15:43.816200 24364 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260809 12:15:43.816419 24364 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskgH5Pjm/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1786277382073938-888-0/minicluster-data/master-0/data,/tmp/dist-test-taskgH5Pjm/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1786277382073938-888-0/minicluster-data/master-0/wal
uuid: "cbb8e258757843b2bcd54a415f7dd460"
format_stamp: "Formatted at 2026-08-09 12:15:43 on dist-test-slave-2w3w"
server_key: "ce8f46945d03f541a2bef61aae19e7b3"
server_key_iv: "0bc8da94480eb79543ddd2ed56409834"
server_key_version: "encryptionkey@0"
I20260809 12:15:43.816612 24364 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskgH5Pjm/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1786277382073938-888-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskgH5Pjm/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1786277382073938-888-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskgH5Pjm/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1786277382073938-888-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260809 12:15:43.848254 24364 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260809 12:15:43.852563 24364 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260809 12:15:43.852877 24364 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260809 12:15:43.872843 24364 rpc_server.cc:307] RPC server started. Bound to: 127.0.222.62:43965
I20260809 12:15:43.872889 24433 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.222.62:43965 every 8 connection(s)
I20260809 12:15:43.874318 24364 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskgH5Pjm/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1786277382073938-888-0/minicluster-data/master-0/data/info.pb
I20260809 12:15:43.877224 888 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskgH5Pjm/build/asan/bin/kudu as pid 24364
I20260809 12:15:43.877460 888 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskgH5Pjm/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1786277382073938-888-0/minicluster-data/master-0/wal/instance
I20260809 12:15:43.877978 888 external_mini_cluster.cc:1428] Setting key e4a56cbe7729df6b8894dc308433cd99
I20260809 12:15:43.878345 24434 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260809 12:15:43.893126 24434 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P cbb8e258757843b2bcd54a415f7dd460: Bootstrap starting.
Aug 09 12:15:43 dist-test-slave-2w3w krb5kdc[24351](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786277743, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.222.62@KRBTEST.COM
I20260809 12:15:43.897219 24434 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P cbb8e258757843b2bcd54a415f7dd460: Neither blocks nor log segments found. Creating new log.
I20260809 12:15:43.898658 24434 log.cc:824] T 00000000000000000000000000000000 P cbb8e258757843b2bcd54a415f7dd460: Log is configured to *not* fsync() on all Append() calls
I20260809 12:15:43.902407 24434 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P cbb8e258757843b2bcd54a415f7dd460: No bootstrap required, opened a new log
I20260809 12:15:43.904199 24437 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 12:15:43.880051 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38105 (local address 127.0.222.62:43965)
0809 12:15:43.880885 (+ 834us) server_negotiation.cc:206] Beginning negotiation
0809 12:15:43.880906 (+ 21us) server_negotiation.cc:399] Waiting for connection header
0809 12:15:43.880955 (+ 49us) server_negotiation.cc:407] Connection header received
0809 12:15:43.883293 (+ 2338us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 12:15:43.883324 (+ 31us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 12:15:43.883838 (+ 514us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 12:15:43.884297 (+ 459us) server_negotiation.cc:226] Negotiated authn=SASL
0809 12:15:43.885759 (+ 1462us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 12:15:43.887778 (+ 2019us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 12:15:43.889205 (+ 1427us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 12:15:43.890313 (+ 1108us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 12:15:43.890514 (+ 201us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 12:15:43.894859 (+ 4345us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 12:15:43.894872 (+ 13us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 12:15:43.894890 (+ 18us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0809 12:15:43.894954 (+ 64us) server_negotiation.cc:922] calling sasl_server_start()
0809 12:15:43.899076 (+ 4122us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 12:15:43.900169 (+ 1093us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 12:15:43.900176 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 12:15:43.900182 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0809 12:15:43.900442 (+ 260us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 12:15:43.900914 (+ 472us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 12:15:43.900919 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 12:15:43.900922 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0809 12:15:43.901646 (+ 724us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 12:15:43.901997 (+ 351us) server_negotiation.cc:1035] Waiting for connection context
0809 12:15:43.902441 (+ 444us) server_negotiation.cc:299] Negotiation successful
0809 12:15:43.902790 (+ 349us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":597,"thread_start_us":360,"threads_started":1}
I20260809 12:15:43.908278 888 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260809 12:15:43.912467 24434 raft_consensus.cc:348] T 00000000000000000000000000000000 P cbb8e258757843b2bcd54a415f7dd460 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "cbb8e258757843b2bcd54a415f7dd460" member_type: VOTER last_known_addr { host: "127.0.222.62" port: 43965 } }
I20260809 12:15:43.912971 24434 raft_consensus.cc:374] T 00000000000000000000000000000000 P cbb8e258757843b2bcd54a415f7dd460 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260809 12:15:43.913091 24434 raft_consensus.cc:729] T 00000000000000000000000000000000 P cbb8e258757843b2bcd54a415f7dd460 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: cbb8e258757843b2bcd54a415f7dd460, State: Initialized, Role: FOLLOWER
I20260809 12:15:43.913793 24434 consensus_queue.cc:260] T 00000000000000000000000000000000 P cbb8e258757843b2bcd54a415f7dd460 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "cbb8e258757843b2bcd54a415f7dd460" member_type: VOTER last_known_addr { host: "127.0.222.62" port: 43965 } }
I20260809 12:15:43.914186 24434 raft_consensus.cc:388] T 00000000000000000000000000000000 P cbb8e258757843b2bcd54a415f7dd460 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260809 12:15:43.914361 24434 raft_consensus.cc:482] T 00000000000000000000000000000000 P cbb8e258757843b2bcd54a415f7dd460 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260809 12:15:43.914553 24434 raft_consensus.cc:3037] T 00000000000000000000000000000000 P cbb8e258757843b2bcd54a415f7dd460 [term 0 FOLLOWER]: Advancing to term 1
I20260809 12:15:43.917426 24434 raft_consensus.cc:504] T 00000000000000000000000000000000 P cbb8e258757843b2bcd54a415f7dd460 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "cbb8e258757843b2bcd54a415f7dd460" member_type: VOTER last_known_addr { host: "127.0.222.62" port: 43965 } }
I20260809 12:15:43.917999 24434 leader_election.cc:302] T 00000000000000000000000000000000 P cbb8e258757843b2bcd54a415f7dd460 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: cbb8e258757843b2bcd54a415f7dd460; no voters:
I20260809 12:15:43.918632 24434 leader_election.cc:288] T 00000000000000000000000000000000 P cbb8e258757843b2bcd54a415f7dd460 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260809 12:15:43.918784 24445 raft_consensus.cc:2781] T 00000000000000000000000000000000 P cbb8e258757843b2bcd54a415f7dd460 [term 1 FOLLOWER]: Leader election won for term 1
I20260809 12:15:43.921025 24434 sys_catalog.cc:564] T 00000000000000000000000000000000 P cbb8e258757843b2bcd54a415f7dd460 [sys.catalog]: configured and running, proceeding with master startup.
I20260809 12:15:43.922086 24445 raft_consensus.cc:686] T 00000000000000000000000000000000 P cbb8e258757843b2bcd54a415f7dd460 [term 1 LEADER]: Becoming Leader. State: Replica: cbb8e258757843b2bcd54a415f7dd460, State: Running, Role: LEADER
I20260809 12:15:43.922746 24445 consensus_queue.cc:237] T 00000000000000000000000000000000 P cbb8e258757843b2bcd54a415f7dd460 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "cbb8e258757843b2bcd54a415f7dd460" member_type: VOTER last_known_addr { host: "127.0.222.62" port: 43965 } }
I20260809 12:15:43.926153 24446 sys_catalog.cc:455] T 00000000000000000000000000000000 P cbb8e258757843b2bcd54a415f7dd460 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "cbb8e258757843b2bcd54a415f7dd460" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "cbb8e258757843b2bcd54a415f7dd460" member_type: VOTER last_known_addr { host: "127.0.222.62" port: 43965 } } }
I20260809 12:15:43.926577 24446 sys_catalog.cc:458] T 00000000000000000000000000000000 P cbb8e258757843b2bcd54a415f7dd460 [sys.catalog]: This master's current role is: LEADER
I20260809 12:15:43.927181 24449 sys_catalog.cc:455] T 00000000000000000000000000000000 P cbb8e258757843b2bcd54a415f7dd460 [sys.catalog]: SysCatalogTable state changed. Reason: New leader cbb8e258757843b2bcd54a415f7dd460. Latest consensus state: current_term: 1 leader_uuid: "cbb8e258757843b2bcd54a415f7dd460" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "cbb8e258757843b2bcd54a415f7dd460" member_type: VOTER last_known_addr { host: "127.0.222.62" port: 43965 } } }
I20260809 12:15:43.927521 24449 sys_catalog.cc:458] T 00000000000000000000000000000000 P cbb8e258757843b2bcd54a415f7dd460 [sys.catalog]: This master's current role is: LEADER
I20260809 12:15:43.935532 24437 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 12:15:43.913830 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45131 (local address 127.0.222.62:43965)
0809 12:15:43.914053 (+ 223us) server_negotiation.cc:206] Beginning negotiation
0809 12:15:43.914062 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0809 12:15:43.914606 (+ 544us) server_negotiation.cc:407] Connection header received
0809 12:15:43.914788 (+ 182us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 12:15:43.914793 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 12:15:43.914877 (+ 84us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 12:15:43.915053 (+ 176us) server_negotiation.cc:226] Negotiated authn=SASL
0809 12:15:43.916426 (+ 1373us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 12:15:43.917539 (+ 1113us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 12:15:43.919372 (+ 1833us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 12:15:43.920360 (+ 988us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 12:15:43.920492 (+ 132us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 12:15:43.924529 (+ 4037us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 12:15:43.924541 (+ 12us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 12:15:43.924549 (+ 8us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0809 12:15:43.924606 (+ 57us) server_negotiation.cc:922] calling sasl_server_start()
0809 12:15:43.929668 (+ 5062us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 12:15:43.931131 (+ 1463us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 12:15:43.931140 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 12:15:43.931147 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0809 12:15:43.931419 (+ 272us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 12:15:43.932206 (+ 787us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 12:15:43.932215 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 12:15:43.932220 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0809 12:15:43.932785 (+ 565us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 12:15:43.934421 (+ 1636us) server_negotiation.cc:1035] Waiting for connection context
0809 12:15:43.934674 (+ 253us) server_negotiation.cc:299] Negotiation successful
0809 12:15:43.934947 (+ 273us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":72}
I20260809 12:15:43.935626 24455 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260809 12:15:43.941298 24455 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260809 12:15:43.950390 24455 catalog_manager.cc:1347] Generated new cluster ID: 17ae278cd22441d3aa3fa3c54f165241
I20260809 12:15:43.950516 24455 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260809 12:15:43.977648 24455 catalog_manager.cc:1370] Generated new certificate authority record
I20260809 12:15:43.979671 24455 catalog_manager.cc:1504] Loading token signing keys...
I20260809 12:15:43.987166 24455 catalog_manager.cc:5868] T 00000000000000000000000000000000 P cbb8e258757843b2bcd54a415f7dd460: Generated new TSK 0
I20260809 12:15:43.988233 24455 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260809 12:15:43.995772 888 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskgH5Pjm/build/asan/bin/kudu pid 24364
I20260809 12:15:43.997323 24428 generic_service.cc:196] Checking for leaks (request via RPC)
I20260809 12:15:44.091207 888 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskgH5Pjm/build/asan/bin/kudu with pid 24364
W20260809 12:15:44.114100 24349 connection.cc:535] client connection to 127.0.222.62:43965 recv error: Network error: failed to read from TLS socket (remote: 127.0.222.62:43965): Connection reset by peer (error 104)
2026-08-09T12:15:44Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/12 (2219 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/13
Loading random data
Initializing database '/tmp/dist-test-taskgH5Pjm/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1786277382073938-888-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 09 12:15:44 dist-test-slave-2w3w krb5kdc[24471](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 09 12:15:44 dist-test-slave-2w3w krb5kdc[24471](info): set up 2 sockets
Aug 09 12:15:44 dist-test-slave-2w3w krb5kdc[24471](info): commencing operation
krb5kdc: starting...
W20260809 12:15:45.276610 888 mini_kdc.cc:121] Time spent starting KDC: real 1.143s user 0.001s sys 0.051s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskgH5Pjm/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1786277382073938-888-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskgH5Pjm/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1786277382073938-888-0/krb5kdc/test-user.keytab.
Aug 09 12:15:45 dist-test-slave-2w3w krb5kdc[24471](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786277745, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-09T12:15:45Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-09T12:15:45Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.222.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.222.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.222.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskgH5Pjm/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1786277382073938-888-0/krb5kdc/kudu_127.0.222.62.keytab.
Entry for principal kudu/127.0.222.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskgH5Pjm/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1786277382073938-888-0/krb5kdc/kudu_127.0.222.62.keytab.
I20260809 12:15:45.624239 888 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskgH5Pjm/build/asan/bin/kudu
/tmp/dist-test-taskgH5Pjm/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskgH5Pjm/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1786277382073938-888-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskgH5Pjm/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1786277382073938-888-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskgH5Pjm/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1786277382073938-888-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskgH5Pjm/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1786277382073938-888-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.222.62:38211
--webserver_interface=127.0.222.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskgH5Pjm/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1786277382073938-888-0/krb5kdc/kudu_127.0.222.62.keytab
--principal=kudu/127.0.222.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.222.20:44359
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.222.62:38211
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=required
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taskgH5Pjm/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1786277382073938-888-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskgH5Pjm/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1786277382073938-888-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskgH5Pjm/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1786277382073938-888-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260809 12:15:45.980441 24485 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260809 12:15:45.980744 24485 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260809 12:15:45.980928 24485 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260809 12:15:45.988327 24485 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260809 12:15:45.988442 24485 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260809 12:15:45.988559 24485 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260809 12:15:45.988667 24485 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260809 12:15:45.988751 24485 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260809 12:15:45.998690 24485 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.222.20:44359
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskgH5Pjm/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1786277382073938-888-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskgH5Pjm/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1786277382073938-888-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.222.62:38211
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskgH5Pjm/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1786277382073938-888-0/krb5kdc/kudu_127.0.222.62.keytab
--principal=kudu/127.0.222.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.222.62:38211
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskgH5Pjm/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1786277382073938-888-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.222.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskgH5Pjm/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1786277382073938-888-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 09 Aug 2026 11:06:41 UTC on ccycloud.quasar-bwoogy.root.comops.site
build id 4575991
ASAN enabled
I20260809 12:15:46.000078 24485 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260809 12:15:46.001415 24485 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260809 12:15:46.013518 24491 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 12:15:46.013597 24494 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 12:15:46.013619 24492 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260809 12:15:46.014616 24485 server_base.cc:1034] running on GCE node
I20260809 12:15:46.015499 24485 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260809 12:15:46.017062 24485 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260809 12:15:46.018337 24485 hybrid_clock.cc:648] HybridClock initialized: now 1786277746018286 us; error 35 us; skew 500 ppm
I20260809 12:15:46.018730 24485 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 09 12:15:46 dist-test-slave-2w3w krb5kdc[24471](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786277746, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.222.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260809 12:15:46.025507 24485 init.cc:377] Logged in from keytab as kudu/127.0.222.62@KRBTEST.COM (short username kudu)
I20260809 12:15:46.027064 24485 webserver.cc:460] Webserver started at http://127.0.222.62:37771/ using document root <none> and password file <none>
I20260809 12:15:46.027745 24485 fs_manager.cc:359] Metadata directory not provided
I20260809 12:15:46.027851 24485 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260809 12:15:46.028113 24485 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260809 12:15:46.031311 24485 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskgH5Pjm/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1786277382073938-888-0/minicluster-data/master-0/data/instance:
uuid: "bf0fbdc116dc4855adcadee07241ce55"
format_stamp: "Formatted at 2026-08-09 12:15:46 on dist-test-slave-2w3w"
server_key: "2795599f5cc1755665c620dc977ccc82"
server_key_iv: "2152848d569c1780344756df2ee88b37"
server_key_version: "encryptionkey@0"
I20260809 12:15:46.032033 24485 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskgH5Pjm/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1786277382073938-888-0/minicluster-data/master-0/wal/instance:
uuid: "bf0fbdc116dc4855adcadee07241ce55"
format_stamp: "Formatted at 2026-08-09 12:15:46 on dist-test-slave-2w3w"
server_key: "2795599f5cc1755665c620dc977ccc82"
server_key_iv: "2152848d569c1780344756df2ee88b37"
server_key_version: "encryptionkey@0"
I20260809 12:15:46.037478 24485 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.000s
I20260809 12:15:46.041043 24501 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260809 12:15:46.042425 24485 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260809 12:15:46.042577 24485 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskgH5Pjm/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1786277382073938-888-0/minicluster-data/master-0/data,/tmp/dist-test-taskgH5Pjm/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1786277382073938-888-0/minicluster-data/master-0/wal
uuid: "bf0fbdc116dc4855adcadee07241ce55"
format_stamp: "Formatted at 2026-08-09 12:15:46 on dist-test-slave-2w3w"
server_key: "2795599f5cc1755665c620dc977ccc82"
server_key_iv: "2152848d569c1780344756df2ee88b37"
server_key_version: "encryptionkey@0"
I20260809 12:15:46.042742 24485 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskgH5Pjm/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1786277382073938-888-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskgH5Pjm/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1786277382073938-888-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskgH5Pjm/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1786277382073938-888-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260809 12:15:46.061949 24485 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260809 12:15:46.066107 24485 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260809 12:15:46.066377 24485 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260809 12:15:46.085942 24485 rpc_server.cc:307] RPC server started. Bound to: 127.0.222.62:38211
I20260809 12:15:46.085985 24556 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.222.62:38211 every 8 connection(s)
I20260809 12:15:46.087502 24485 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskgH5Pjm/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1786277382073938-888-0/minicluster-data/master-0/data/info.pb
I20260809 12:15:46.089669 888 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskgH5Pjm/build/asan/bin/kudu as pid 24485
I20260809 12:15:46.089887 888 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskgH5Pjm/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1786277382073938-888-0/minicluster-data/master-0/wal/instance
I20260809 12:15:46.090369 888 external_mini_cluster.cc:1428] Setting key 0dbf73b576eb5f7c4fec0af6bd56e6a8
I20260809 12:15:46.091902 24557 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260809 12:15:46.105746 24560 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 12:15:46.092284 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43075 (local address 127.0.222.62:38211)
0809 12:15:46.093142 (+ 858us) server_negotiation.cc:206] Beginning negotiation
0809 12:15:46.093157 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0809 12:15:46.093207 (+ 50us) server_negotiation.cc:407] Connection header received
0809 12:15:46.095290 (+ 2083us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 12:15:46.095318 (+ 28us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 12:15:46.095853 (+ 535us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 12:15:46.096323 (+ 470us) server_negotiation.cc:226] Negotiated authn=SASL
0809 12:15:46.096835 (+ 512us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 12:15:46.098884 (+ 2049us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 12:15:46.100242 (+ 1358us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 12:15:46.101275 (+ 1033us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 12:15:46.101472 (+ 197us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 12:15:46.102725 (+ 1253us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 12:15:46.102739 (+ 14us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 12:15:46.102758 (+ 19us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0809 12:15:46.103562 (+ 804us) server_negotiation.cc:922] calling sasl_server_start()
0809 12:15:46.103672 (+ 110us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0809 12:15:46.103708 (+ 36us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 12:15:46.103883 (+ 175us) server_negotiation.cc:1035] Waiting for connection context
0809 12:15:46.104179 (+ 296us) server_negotiation.cc:299] Negotiation successful
0809 12:15:46.104370 (+ 191us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":610,"thread_start_us":338,"threads_started":1}
I20260809 12:15:46.107338 24557 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P bf0fbdc116dc4855adcadee07241ce55: Bootstrap starting.
I20260809 12:15:46.109817 888 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260809 12:15:46.110806 24557 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P bf0fbdc116dc4855adcadee07241ce55: Neither blocks nor log segments found. Creating new log.
I20260809 12:15:46.112084 24557 log.cc:824] T 00000000000000000000000000000000 P bf0fbdc116dc4855adcadee07241ce55: Log is configured to *not* fsync() on all Append() calls
I20260809 12:15:46.115541 24557 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P bf0fbdc116dc4855adcadee07241ce55: No bootstrap required, opened a new log
I20260809 12:15:46.122455 24560 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 12:15:46.115465 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58019 (local address 127.0.222.62:38211)
0809 12:15:46.115666 (+ 201us) server_negotiation.cc:206] Beginning negotiation
0809 12:15:46.115673 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0809 12:15:46.116096 (+ 423us) server_negotiation.cc:407] Connection header received
0809 12:15:46.116266 (+ 170us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 12:15:46.116271 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 12:15:46.116356 (+ 85us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 12:15:46.116488 (+ 132us) server_negotiation.cc:226] Negotiated authn=SASL
0809 12:15:46.117012 (+ 524us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 12:15:46.118217 (+ 1205us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 12:15:46.119637 (+ 1420us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 12:15:46.120331 (+ 694us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 12:15:46.120464 (+ 133us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 12:15:46.121596 (+ 1132us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 12:15:46.121602 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 12:15:46.121609 (+ 7us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0809 12:15:46.121631 (+ 22us) server_negotiation.cc:922] calling sasl_server_start()
0809 12:15:46.121698 (+ 67us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0809 12:15:46.121718 (+ 20us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 12:15:46.121844 (+ 126us) server_negotiation.cc:1035] Waiting for connection context
0809 12:15:46.122010 (+ 166us) server_negotiation.cc:299] Negotiation successful
0809 12:15:46.122072 (+ 62us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":71}
I20260809 12:15:46.125806 24557 raft_consensus.cc:348] T 00000000000000000000000000000000 P bf0fbdc116dc4855adcadee07241ce55 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "bf0fbdc116dc4855adcadee07241ce55" member_type: VOTER last_known_addr { host: "127.0.222.62" port: 38211 } }
I20260809 12:15:46.126186 24557 raft_consensus.cc:374] T 00000000000000000000000000000000 P bf0fbdc116dc4855adcadee07241ce55 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260809 12:15:46.126271 24557 raft_consensus.cc:729] T 00000000000000000000000000000000 P bf0fbdc116dc4855adcadee07241ce55 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: bf0fbdc116dc4855adcadee07241ce55, State: Initialized, Role: FOLLOWER
I20260809 12:15:46.126858 24557 consensus_queue.cc:260] T 00000000000000000000000000000000 P bf0fbdc116dc4855adcadee07241ce55 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "bf0fbdc116dc4855adcadee07241ce55" member_type: VOTER last_known_addr { host: "127.0.222.62" port: 38211 } }
I20260809 12:15:46.127146 24557 raft_consensus.cc:388] T 00000000000000000000000000000000 P bf0fbdc116dc4855adcadee07241ce55 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260809 12:15:46.127255 24557 raft_consensus.cc:482] T 00000000000000000000000000000000 P bf0fbdc116dc4855adcadee07241ce55 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260809 12:15:46.127384 24557 raft_consensus.cc:3037] T 00000000000000000000000000000000 P bf0fbdc116dc4855adcadee07241ce55 [term 0 FOLLOWER]: Advancing to term 1
I20260809 12:15:46.130100 24557 raft_consensus.cc:504] T 00000000000000000000000000000000 P bf0fbdc116dc4855adcadee07241ce55 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "bf0fbdc116dc4855adcadee07241ce55" member_type: VOTER last_known_addr { host: "127.0.222.62" port: 38211 } }
I20260809 12:15:46.130568 24557 leader_election.cc:302] T 00000000000000000000000000000000 P bf0fbdc116dc4855adcadee07241ce55 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: bf0fbdc116dc4855adcadee07241ce55; no voters:
I20260809 12:15:46.131274 24557 leader_election.cc:288] T 00000000000000000000000000000000 P bf0fbdc116dc4855adcadee07241ce55 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260809 12:15:46.131377 24567 raft_consensus.cc:2781] T 00000000000000000000000000000000 P bf0fbdc116dc4855adcadee07241ce55 [term 1 FOLLOWER]: Leader election won for term 1
I20260809 12:15:46.133423 24567 raft_consensus.cc:686] T 00000000000000000000000000000000 P bf0fbdc116dc4855adcadee07241ce55 [term 1 LEADER]: Becoming Leader. State: Replica: bf0fbdc116dc4855adcadee07241ce55, State: Running, Role: LEADER
I20260809 12:15:46.134020 24557 sys_catalog.cc:564] T 00000000000000000000000000000000 P bf0fbdc116dc4855adcadee07241ce55 [sys.catalog]: configured and running, proceeding with master startup.
I20260809 12:15:46.133989 24567 consensus_queue.cc:237] T 00000000000000000000000000000000 P bf0fbdc116dc4855adcadee07241ce55 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "bf0fbdc116dc4855adcadee07241ce55" member_type: VOTER last_known_addr { host: "127.0.222.62" port: 38211 } }
I20260809 12:15:46.136965 24569 sys_catalog.cc:455] T 00000000000000000000000000000000 P bf0fbdc116dc4855adcadee07241ce55 [sys.catalog]: SysCatalogTable state changed. Reason: New leader bf0fbdc116dc4855adcadee07241ce55. Latest consensus state: current_term: 1 leader_uuid: "bf0fbdc116dc4855adcadee07241ce55" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "bf0fbdc116dc4855adcadee07241ce55" member_type: VOTER last_known_addr { host: "127.0.222.62" port: 38211 } } }
I20260809 12:15:46.137316 24569 sys_catalog.cc:458] T 00000000000000000000000000000000 P bf0fbdc116dc4855adcadee07241ce55 [sys.catalog]: This master's current role is: LEADER
I20260809 12:15:46.137984 24568 sys_catalog.cc:455] T 00000000000000000000000000000000 P bf0fbdc116dc4855adcadee07241ce55 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "bf0fbdc116dc4855adcadee07241ce55" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "bf0fbdc116dc4855adcadee07241ce55" member_type: VOTER last_known_addr { host: "127.0.222.62" port: 38211 } } }
I20260809 12:15:46.138373 24568 sys_catalog.cc:458] T 00000000000000000000000000000000 P bf0fbdc116dc4855adcadee07241ce55 [sys.catalog]: This master's current role is: LEADER
I20260809 12:15:46.140530 24578 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260809 12:15:46.145951 24578 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260809 12:15:46.156170 24578 catalog_manager.cc:1347] Generated new cluster ID: bd110cb0fd47406dbeabaa1c6d3e6ecc
I20260809 12:15:46.156303 24578 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260809 12:15:46.175820 24578 catalog_manager.cc:1370] Generated new certificate authority record
I20260809 12:15:46.177155 24578 catalog_manager.cc:1504] Loading token signing keys...
I20260809 12:15:46.185123 24578 catalog_manager.cc:5868] T 00000000000000000000000000000000 P bf0fbdc116dc4855adcadee07241ce55: Generated new TSK 0
I20260809 12:15:46.185945 24578 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260809 12:15:46.199018 888 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskgH5Pjm/build/asan/bin/kudu pid 24485
W20260809 12:15:46.200224 24552 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='slave'} at 127.0.0.1:43075
W20260809 12:15:46.201097 888 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskgH5Pjm/build/asan/bin/kudu pid 24485: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260809 12:15:46.201256 888 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskgH5Pjm/build/asan/bin/kudu with pid 24485
W20260809 12:15:46.223279 24469 connection.cc:535] client connection to 127.0.222.62:38211 recv error: Network error: failed to read from TLS socket (remote: 127.0.222.62:38211): Connection reset by peer (error 104)
2026-08-09T12:15:46Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/13 (2109 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/14
Loading random data
Initializing database '/tmp/dist-test-taskgH5Pjm/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1786277382073938-888-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 09 12:15:46 dist-test-slave-2w3w krb5kdc[24591](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 09 12:15:46 dist-test-slave-2w3w krb5kdc[24591](info): set up 2 sockets
Aug 09 12:15:46 dist-test-slave-2w3w krb5kdc[24591](info): commencing operation
krb5kdc: starting...
W20260809 12:15:47.393172 888 mini_kdc.cc:121] Time spent starting KDC: real 1.150s user 0.001s sys 0.047s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskgH5Pjm/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1786277382073938-888-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskgH5Pjm/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1786277382073938-888-0/krb5kdc/test-user.keytab.
Aug 09 12:15:47 dist-test-slave-2w3w krb5kdc[24591](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786277747, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-09T12:15:47Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-09T12:15:47Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.222.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.222.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.222.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskgH5Pjm/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1786277382073938-888-0/krb5kdc/kudu_127.0.222.62.keytab.
Entry for principal kudu/127.0.222.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskgH5Pjm/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1786277382073938-888-0/krb5kdc/kudu_127.0.222.62.keytab.
I20260809 12:15:47.734791 888 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskgH5Pjm/build/asan/bin/kudu
/tmp/dist-test-taskgH5Pjm/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskgH5Pjm/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1786277382073938-888-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskgH5Pjm/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1786277382073938-888-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskgH5Pjm/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1786277382073938-888-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskgH5Pjm/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1786277382073938-888-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.222.62:39411
--webserver_interface=127.0.222.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskgH5Pjm/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1786277382073938-888-0/krb5kdc/kudu_127.0.222.62.keytab
--principal=kudu/127.0.222.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.222.20:34013
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.222.62:39411
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=disabled
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taskgH5Pjm/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1786277382073938-888-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskgH5Pjm/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1786277382073938-888-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskgH5Pjm/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1786277382073938-888-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260809 12:15:48.083774 24607 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260809 12:15:48.084075 24607 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260809 12:15:48.084254 24607 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260809 12:15:48.091565 24607 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260809 12:15:48.091678 24607 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260809 12:15:48.091774 24607 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260809 12:15:48.091907 24607 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260809 12:15:48.091988 24607 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260809 12:15:48.101971 24607 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.222.20:34013
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskgH5Pjm/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1786277382073938-888-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskgH5Pjm/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1786277382073938-888-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.222.62:39411
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskgH5Pjm/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1786277382073938-888-0/krb5kdc/kudu_127.0.222.62.keytab
--principal=kudu/127.0.222.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.222.62:39411
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskgH5Pjm/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1786277382073938-888-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.222.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskgH5Pjm/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1786277382073938-888-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 09 Aug 2026 11:06:41 UTC on ccycloud.quasar-bwoogy.root.comops.site
build id 4575991
ASAN enabled
I20260809 12:15:48.103458 24607 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260809 12:15:48.104817 24607 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260809 12:15:48.117122 24612 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 12:15:48.117201 24617 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 12:15:48.117162 24613 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260809 12:15:48.118053 24607 server_base.cc:1034] running on GCE node
I20260809 12:15:48.118920 24607 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260809 12:15:48.120427 24607 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260809 12:15:48.121667 24607 hybrid_clock.cc:648] HybridClock initialized: now 1786277748121621 us; error 39 us; skew 500 ppm
I20260809 12:15:48.122048 24607 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 09 12:15:48 dist-test-slave-2w3w krb5kdc[24591](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786277748, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.222.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260809 12:15:48.125306 24607 init.cc:377] Logged in from keytab as kudu/127.0.222.62@KRBTEST.COM (short username kudu)
I20260809 12:15:48.126791 24607 webserver.cc:460] Webserver started at http://127.0.222.62:34695/ using document root <none> and password file <none>
I20260809 12:15:48.127525 24607 fs_manager.cc:359] Metadata directory not provided
I20260809 12:15:48.127630 24607 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260809 12:15:48.127887 24607 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260809 12:15:48.130822 24607 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskgH5Pjm/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1786277382073938-888-0/minicluster-data/master-0/data/instance:
uuid: "f47dcf6a6c474639acbeae175cb13848"
format_stamp: "Formatted at 2026-08-09 12:15:48 on dist-test-slave-2w3w"
server_key: "de22ac0862f4d414d680148a79affa10"
server_key_iv: "80ebf8021befdc310ecfb98c67366f6d"
server_key_version: "encryptionkey@0"
I20260809 12:15:48.131623 24607 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskgH5Pjm/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1786277382073938-888-0/minicluster-data/master-0/wal/instance:
uuid: "f47dcf6a6c474639acbeae175cb13848"
format_stamp: "Formatted at 2026-08-09 12:15:48 on dist-test-slave-2w3w"
server_key: "de22ac0862f4d414d680148a79affa10"
server_key_iv: "80ebf8021befdc310ecfb98c67366f6d"
server_key_version: "encryptionkey@0"
I20260809 12:15:48.137173 24607 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.000s
I20260809 12:15:48.140542 24625 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260809 12:15:48.142014 24607 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.000s
I20260809 12:15:48.142160 24607 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskgH5Pjm/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1786277382073938-888-0/minicluster-data/master-0/data,/tmp/dist-test-taskgH5Pjm/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1786277382073938-888-0/minicluster-data/master-0/wal
uuid: "f47dcf6a6c474639acbeae175cb13848"
format_stamp: "Formatted at 2026-08-09 12:15:48 on dist-test-slave-2w3w"
server_key: "de22ac0862f4d414d680148a79affa10"
server_key_iv: "80ebf8021befdc310ecfb98c67366f6d"
server_key_version: "encryptionkey@0"
I20260809 12:15:48.142330 24607 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskgH5Pjm/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1786277382073938-888-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskgH5Pjm/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1786277382073938-888-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskgH5Pjm/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1786277382073938-888-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260809 12:15:48.148088 24607 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260809 12:15:48.152390 24607 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260809 12:15:48.152649 24607 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260809 12:15:48.170785 24607 rpc_server.cc:307] RPC server started. Bound to: 127.0.222.62:39411
I20260809 12:15:48.170825 24684 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.222.62:39411 every 8 connection(s)
I20260809 12:15:48.172278 24607 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskgH5Pjm/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1786277382073938-888-0/minicluster-data/master-0/data/info.pb
I20260809 12:15:48.176115 24685 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260809 12:15:48.178278 888 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskgH5Pjm/build/asan/bin/kudu as pid 24607
I20260809 12:15:48.178460 888 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskgH5Pjm/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1786277382073938-888-0/minicluster-data/master-0/wal/instance
I20260809 12:15:48.178929 888 external_mini_cluster.cc:1428] Setting key f408862248defe3efcaa3ea05385d03a
I20260809 12:15:48.187943 24689 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 12:15:48.180714 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60543 (local address 127.0.222.62:39411)
0809 12:15:48.181529 (+ 815us) server_negotiation.cc:206] Beginning negotiation
0809 12:15:48.181544 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0809 12:15:48.181594 (+ 50us) server_negotiation.cc:407] Connection header received
0809 12:15:48.184002 (+ 2408us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 12:15:48.184030 (+ 28us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 12:15:48.184445 (+ 415us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 12:15:48.184899 (+ 454us) server_negotiation.cc:226] Negotiated authn=SASL
0809 12:15:48.185891 (+ 992us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 12:15:48.185899 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 12:15:48.185921 (+ 22us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0809 12:15:48.185939 (+ 18us) server_negotiation.cc:922] calling sasl_server_start()
0809 12:15:48.186011 (+ 72us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0809 12:15:48.186048 (+ 37us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 12:15:48.186276 (+ 228us) server_negotiation.cc:1035] Waiting for connection context
0809 12:15:48.186409 (+ 133us) server_negotiation.cc:299] Negotiation successful
0809 12:15:48.186608 (+ 199us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":564,"thread_start_us":340,"threads_started":1}
I20260809 12:15:48.189925 24685 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P f47dcf6a6c474639acbeae175cb13848: Bootstrap starting.
I20260809 12:15:48.191888 888 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260809 12:15:48.193890 24685 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P f47dcf6a6c474639acbeae175cb13848: Neither blocks nor log segments found. Creating new log.
I20260809 12:15:48.195248 24685 log.cc:824] T 00000000000000000000000000000000 P f47dcf6a6c474639acbeae175cb13848: Log is configured to *not* fsync() on all Append() calls
I20260809 12:15:48.198572 24685 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P f47dcf6a6c474639acbeae175cb13848: No bootstrap required, opened a new log
I20260809 12:15:48.200088 24689 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 12:15:48.197436 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48269 (local address 127.0.222.62:39411)
0809 12:15:48.197619 (+ 183us) server_negotiation.cc:206] Beginning negotiation
0809 12:15:48.197625 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0809 12:15:48.198156 (+ 531us) server_negotiation.cc:407] Connection header received
0809 12:15:48.198260 (+ 104us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 12:15:48.198270 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 12:15:48.198327 (+ 57us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 12:15:48.198433 (+ 106us) server_negotiation.cc:226] Negotiated authn=SASL
0809 12:15:48.199438 (+ 1005us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 12:15:48.199444 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 12:15:48.199449 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0809 12:15:48.199474 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0809 12:15:48.199518 (+ 44us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0809 12:15:48.199538 (+ 20us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 12:15:48.199652 (+ 114us) server_negotiation.cc:1035] Waiting for connection context
0809 12:15:48.199733 (+ 81us) server_negotiation.cc:299] Negotiation successful
0809 12:15:48.199787 (+ 54us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":59}
I20260809 12:15:48.206764 24685 raft_consensus.cc:348] T 00000000000000000000000000000000 P f47dcf6a6c474639acbeae175cb13848 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f47dcf6a6c474639acbeae175cb13848" member_type: VOTER last_known_addr { host: "127.0.222.62" port: 39411 } }
I20260809 12:15:48.207178 24685 raft_consensus.cc:374] T 00000000000000000000000000000000 P f47dcf6a6c474639acbeae175cb13848 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260809 12:15:48.207258 24685 raft_consensus.cc:729] T 00000000000000000000000000000000 P f47dcf6a6c474639acbeae175cb13848 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f47dcf6a6c474639acbeae175cb13848, State: Initialized, Role: FOLLOWER
I20260809 12:15:48.207836 24685 consensus_queue.cc:260] T 00000000000000000000000000000000 P f47dcf6a6c474639acbeae175cb13848 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f47dcf6a6c474639acbeae175cb13848" member_type: VOTER last_known_addr { host: "127.0.222.62" port: 39411 } }
I20260809 12:15:48.208038 24685 raft_consensus.cc:388] T 00000000000000000000000000000000 P f47dcf6a6c474639acbeae175cb13848 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260809 12:15:48.208135 24685 raft_consensus.cc:482] T 00000000000000000000000000000000 P f47dcf6a6c474639acbeae175cb13848 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260809 12:15:48.208250 24685 raft_consensus.cc:3037] T 00000000000000000000000000000000 P f47dcf6a6c474639acbeae175cb13848 [term 0 FOLLOWER]: Advancing to term 1
I20260809 12:15:48.210421 24685 raft_consensus.cc:504] T 00000000000000000000000000000000 P f47dcf6a6c474639acbeae175cb13848 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f47dcf6a6c474639acbeae175cb13848" member_type: VOTER last_known_addr { host: "127.0.222.62" port: 39411 } }
I20260809 12:15:48.210877 24685 leader_election.cc:302] T 00000000000000000000000000000000 P f47dcf6a6c474639acbeae175cb13848 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: f47dcf6a6c474639acbeae175cb13848; no voters:
I20260809 12:15:48.211650 24685 leader_election.cc:288] T 00000000000000000000000000000000 P f47dcf6a6c474639acbeae175cb13848 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260809 12:15:48.211763 24696 raft_consensus.cc:2781] T 00000000000000000000000000000000 P f47dcf6a6c474639acbeae175cb13848 [term 1 FOLLOWER]: Leader election won for term 1
I20260809 12:15:48.212447 24696 raft_consensus.cc:686] T 00000000000000000000000000000000 P f47dcf6a6c474639acbeae175cb13848 [term 1 LEADER]: Becoming Leader. State: Replica: f47dcf6a6c474639acbeae175cb13848, State: Running, Role: LEADER
I20260809 12:15:48.213069 24696 consensus_queue.cc:237] T 00000000000000000000000000000000 P f47dcf6a6c474639acbeae175cb13848 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f47dcf6a6c474639acbeae175cb13848" member_type: VOTER last_known_addr { host: "127.0.222.62" port: 39411 } }
I20260809 12:15:48.214012 24685 sys_catalog.cc:564] T 00000000000000000000000000000000 P f47dcf6a6c474639acbeae175cb13848 [sys.catalog]: configured and running, proceeding with master startup.
I20260809 12:15:48.216012 24697 sys_catalog.cc:455] T 00000000000000000000000000000000 P f47dcf6a6c474639acbeae175cb13848 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "f47dcf6a6c474639acbeae175cb13848" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f47dcf6a6c474639acbeae175cb13848" member_type: VOTER last_known_addr { host: "127.0.222.62" port: 39411 } } }
I20260809 12:15:48.216449 24697 sys_catalog.cc:458] T 00000000000000000000000000000000 P f47dcf6a6c474639acbeae175cb13848 [sys.catalog]: This master's current role is: LEADER
I20260809 12:15:48.216478 24698 sys_catalog.cc:455] T 00000000000000000000000000000000 P f47dcf6a6c474639acbeae175cb13848 [sys.catalog]: SysCatalogTable state changed. Reason: New leader f47dcf6a6c474639acbeae175cb13848. Latest consensus state: current_term: 1 leader_uuid: "f47dcf6a6c474639acbeae175cb13848" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f47dcf6a6c474639acbeae175cb13848" member_type: VOTER last_known_addr { host: "127.0.222.62" port: 39411 } } }
I20260809 12:15:48.216864 24698 sys_catalog.cc:458] T 00000000000000000000000000000000 P f47dcf6a6c474639acbeae175cb13848 [sys.catalog]: This master's current role is: LEADER
I20260809 12:15:48.219044 24702 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260809 12:15:48.224267 24702 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260809 12:15:48.233651 24702 catalog_manager.cc:1347] Generated new cluster ID: 31db2abc6e55478e86d463dd2199b771
I20260809 12:15:48.233783 24702 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260809 12:15:48.265782 24702 catalog_manager.cc:1370] Generated new certificate authority record
I20260809 12:15:48.266304 24702 catalog_manager.cc:1504] Loading token signing keys...
I20260809 12:15:48.281095 24702 catalog_manager.cc:5868] T 00000000000000000000000000000000 P f47dcf6a6c474639acbeae175cb13848: Generated new TSK 0
I20260809 12:15:48.281930 24702 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260809 12:15:48.293319 888 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskgH5Pjm/build/asan/bin/kudu pid 24607
W20260809 12:15:48.294629 24680 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='slave'} at 127.0.0.1:60543
W20260809 12:15:48.295449 888 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskgH5Pjm/build/asan/bin/kudu pid 24607: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260809 12:15:48.295599 888 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskgH5Pjm/build/asan/bin/kudu with pid 24607
2026-08-09T12:15:48Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/14 (2092 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/15
Loading random data
Initializing database '/tmp/dist-test-taskgH5Pjm/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1786277382073938-888-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 09 12:15:48 dist-test-slave-2w3w krb5kdc[24719](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 09 12:15:48 dist-test-slave-2w3w krb5kdc[24719](info): set up 2 sockets
Aug 09 12:15:48 dist-test-slave-2w3w krb5kdc[24719](info): commencing operation
krb5kdc: starting...
W20260809 12:15:49.468140 888 mini_kdc.cc:121] Time spent starting KDC: real 1.133s user 0.004s sys 0.044s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskgH5Pjm/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1786277382073938-888-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskgH5Pjm/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1786277382073938-888-0/krb5kdc/test-user.keytab.
Aug 09 12:15:49 dist-test-slave-2w3w krb5kdc[24719](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786277749, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-09T12:15:49Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-09T12:15:49Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.222.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.222.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.222.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskgH5Pjm/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1786277382073938-888-0/krb5kdc/kudu_127.0.222.62.keytab.
Entry for principal kudu/127.0.222.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskgH5Pjm/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1786277382073938-888-0/krb5kdc/kudu_127.0.222.62.keytab.
I20260809 12:15:49.810796 888 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskgH5Pjm/build/asan/bin/kudu
/tmp/dist-test-taskgH5Pjm/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskgH5Pjm/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1786277382073938-888-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskgH5Pjm/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1786277382073938-888-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskgH5Pjm/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1786277382073938-888-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskgH5Pjm/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1786277382073938-888-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.222.62:41159
--webserver_interface=127.0.222.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskgH5Pjm/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1786277382073938-888-0/krb5kdc/kudu_127.0.222.62.keytab
--principal=kudu/127.0.222.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.222.20:40251
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.222.62:41159
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=required
--rpc_encryption=required
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taskgH5Pjm/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1786277382073938-888-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskgH5Pjm/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1786277382073938-888-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskgH5Pjm/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1786277382073938-888-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260809 12:15:50.161875 24735 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260809 12:15:50.162243 24735 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260809 12:15:50.162415 24735 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260809 12:15:50.170013 24735 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260809 12:15:50.170173 24735 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260809 12:15:50.170288 24735 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260809 12:15:50.170411 24735 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260809 12:15:50.170497 24735 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260809 12:15:50.180399 24735 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.222.20:40251
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskgH5Pjm/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1786277382073938-888-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskgH5Pjm/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1786277382073938-888-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.222.62:41159
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskgH5Pjm/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1786277382073938-888-0/krb5kdc/kudu_127.0.222.62.keytab
--principal=kudu/127.0.222.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.222.62:41159
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskgH5Pjm/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1786277382073938-888-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.222.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskgH5Pjm/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1786277382073938-888-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 09 Aug 2026 11:06:41 UTC on ccycloud.quasar-bwoogy.root.comops.site
build id 4575991
ASAN enabled
I20260809 12:15:50.181706 24735 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260809 12:15:50.183065 24735 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260809 12:15:50.195616 24741 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 12:15:50.195715 24745 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 12:15:50.195739 24742 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260809 12:15:50.196710 24735 server_base.cc:1034] running on GCE node
I20260809 12:15:50.197620 24735 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260809 12:15:50.199229 24735 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260809 12:15:50.200471 24735 hybrid_clock.cc:648] HybridClock initialized: now 1786277750200420 us; error 39 us; skew 500 ppm
I20260809 12:15:50.200848 24735 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 09 12:15:50 dist-test-slave-2w3w krb5kdc[24719](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786277750, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.222.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260809 12:15:50.204224 24735 init.cc:377] Logged in from keytab as kudu/127.0.222.62@KRBTEST.COM (short username kudu)
I20260809 12:15:50.205832 24735 webserver.cc:460] Webserver started at http://127.0.222.62:42875/ using document root <none> and password file <none>
I20260809 12:15:50.206578 24735 fs_manager.cc:359] Metadata directory not provided
I20260809 12:15:50.206768 24735 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260809 12:15:50.207165 24735 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260809 12:15:50.210204 24735 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskgH5Pjm/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1786277382073938-888-0/minicluster-data/master-0/data/instance:
uuid: "72b594fdae164a2abe5f64c4dfb1cb5c"
format_stamp: "Formatted at 2026-08-09 12:15:50 on dist-test-slave-2w3w"
server_key: "000e10fae4401b295ae7516ab2a1e13d"
server_key_iv: "52a51e0702493fb261c75980f9164650"
server_key_version: "encryptionkey@0"
I20260809 12:15:50.211090 24735 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskgH5Pjm/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1786277382073938-888-0/minicluster-data/master-0/wal/instance:
uuid: "72b594fdae164a2abe5f64c4dfb1cb5c"
format_stamp: "Formatted at 2026-08-09 12:15:50 on dist-test-slave-2w3w"
server_key: "000e10fae4401b295ae7516ab2a1e13d"
server_key_iv: "52a51e0702493fb261c75980f9164650"
server_key_version: "encryptionkey@0"
I20260809 12:15:50.216398 24735 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.004s sys 0.000s
I20260809 12:15:50.219980 24752 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260809 12:15:50.221320 24735 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.001s
I20260809 12:15:50.221529 24735 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskgH5Pjm/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1786277382073938-888-0/minicluster-data/master-0/data,/tmp/dist-test-taskgH5Pjm/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1786277382073938-888-0/minicluster-data/master-0/wal
uuid: "72b594fdae164a2abe5f64c4dfb1cb5c"
format_stamp: "Formatted at 2026-08-09 12:15:50 on dist-test-slave-2w3w"
server_key: "000e10fae4401b295ae7516ab2a1e13d"
server_key_iv: "52a51e0702493fb261c75980f9164650"
server_key_version: "encryptionkey@0"
I20260809 12:15:50.221733 24735 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskgH5Pjm/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1786277382073938-888-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskgH5Pjm/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1786277382073938-888-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskgH5Pjm/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1786277382073938-888-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260809 12:15:50.242846 24735 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260809 12:15:50.246938 24735 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260809 12:15:50.247313 24735 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260809 12:15:50.266413 24735 rpc_server.cc:307] RPC server started. Bound to: 127.0.222.62:41159
I20260809 12:15:50.266458 24808 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.222.62:41159 every 8 connection(s)
I20260809 12:15:50.267843 24735 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskgH5Pjm/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1786277382073938-888-0/minicluster-data/master-0/data/info.pb
I20260809 12:15:50.271737 24809 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260809 12:15:50.275003 888 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskgH5Pjm/build/asan/bin/kudu as pid 24735
I20260809 12:15:50.275204 888 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskgH5Pjm/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1786277382073938-888-0/minicluster-data/master-0/wal/instance
I20260809 12:15:50.275678 888 external_mini_cluster.cc:1428] Setting key 2a243ad0ce6a310370cd7b40988bcb17
I20260809 12:15:50.284243 24809 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 72b594fdae164a2abe5f64c4dfb1cb5c: Bootstrap starting.
I20260809 12:15:50.288095 24809 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 72b594fdae164a2abe5f64c4dfb1cb5c: Neither blocks nor log segments found. Creating new log.
I20260809 12:15:50.289472 24809 log.cc:824] T 00000000000000000000000000000000 P 72b594fdae164a2abe5f64c4dfb1cb5c: Log is configured to *not* fsync() on all Append() calls
Aug 09 12:15:50 dist-test-slave-2w3w krb5kdc[24719](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786277749, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.222.62@KRBTEST.COM
I20260809 12:15:50.292570 24809 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 72b594fdae164a2abe5f64c4dfb1cb5c: No bootstrap required, opened a new log
I20260809 12:15:50.299575 24812 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 12:15:50.277573 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38527 (local address 127.0.222.62:41159)
0809 12:15:50.278357 (+ 784us) server_negotiation.cc:206] Beginning negotiation
0809 12:15:50.278368 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0809 12:15:50.278407 (+ 39us) server_negotiation.cc:407] Connection header received
0809 12:15:50.280094 (+ 1687us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 12:15:50.280115 (+ 21us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 12:15:50.280505 (+ 390us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 12:15:50.280803 (+ 298us) server_negotiation.cc:226] Negotiated authn=SASL
0809 12:15:50.282629 (+ 1826us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 12:15:50.284033 (+ 1404us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 12:15:50.285927 (+ 1894us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 12:15:50.286601 (+ 674us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 12:15:50.286734 (+ 133us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 12:15:50.291987 (+ 5253us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 12:15:50.291995 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 12:15:50.292009 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0809 12:15:50.292044 (+ 35us) server_negotiation.cc:922] calling sasl_server_start()
0809 12:15:50.295318 (+ 3274us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 12:15:50.296146 (+ 828us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 12:15:50.296152 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 12:15:50.296161 (+ 9us) server_negotiation.cc:967] calling sasl_server_step()
0809 12:15:50.296343 (+ 182us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 12:15:50.296779 (+ 436us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 12:15:50.296782 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 12:15:50.296784 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0809 12:15:50.297299 (+ 515us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 12:15:50.297534 (+ 235us) server_negotiation.cc:1035] Waiting for connection context
0809 12:15:50.298018 (+ 484us) server_negotiation.cc:299] Negotiation successful
0809 12:15:50.298408 (+ 390us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":584,"thread_start_us":357,"threads_started":1}
I20260809 12:15:50.301755 24809 raft_consensus.cc:348] T 00000000000000000000000000000000 P 72b594fdae164a2abe5f64c4dfb1cb5c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "72b594fdae164a2abe5f64c4dfb1cb5c" member_type: VOTER last_known_addr { host: "127.0.222.62" port: 41159 } }
I20260809 12:15:50.302188 24809 raft_consensus.cc:374] T 00000000000000000000000000000000 P 72b594fdae164a2abe5f64c4dfb1cb5c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260809 12:15:50.302297 24809 raft_consensus.cc:729] T 00000000000000000000000000000000 P 72b594fdae164a2abe5f64c4dfb1cb5c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 72b594fdae164a2abe5f64c4dfb1cb5c, State: Initialized, Role: FOLLOWER
I20260809 12:15:50.303077 24809 consensus_queue.cc:260] T 00000000000000000000000000000000 P 72b594fdae164a2abe5f64c4dfb1cb5c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "72b594fdae164a2abe5f64c4dfb1cb5c" member_type: VOTER last_known_addr { host: "127.0.222.62" port: 41159 } }
I20260809 12:15:50.303385 24809 raft_consensus.cc:388] T 00000000000000000000000000000000 P 72b594fdae164a2abe5f64c4dfb1cb5c [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260809 12:15:50.303539 888 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260809 12:15:50.303558 24809 raft_consensus.cc:482] T 00000000000000000000000000000000 P 72b594fdae164a2abe5f64c4dfb1cb5c [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260809 12:15:50.303730 24809 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 72b594fdae164a2abe5f64c4dfb1cb5c [term 0 FOLLOWER]: Advancing to term 1
I20260809 12:15:50.307052 24809 raft_consensus.cc:504] T 00000000000000000000000000000000 P 72b594fdae164a2abe5f64c4dfb1cb5c [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "72b594fdae164a2abe5f64c4dfb1cb5c" member_type: VOTER last_known_addr { host: "127.0.222.62" port: 41159 } }
I20260809 12:15:50.307730 24809 leader_election.cc:302] T 00000000000000000000000000000000 P 72b594fdae164a2abe5f64c4dfb1cb5c [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 72b594fdae164a2abe5f64c4dfb1cb5c; no voters:
I20260809 12:15:50.308454 24809 leader_election.cc:288] T 00000000000000000000000000000000 P 72b594fdae164a2abe5f64c4dfb1cb5c [CANDIDATE]: Term 1 election: Requested vote from peers
I20260809 12:15:50.308568 24822 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 72b594fdae164a2abe5f64c4dfb1cb5c [term 1 FOLLOWER]: Leader election won for term 1
I20260809 12:15:50.310523 24809 sys_catalog.cc:564] T 00000000000000000000000000000000 P 72b594fdae164a2abe5f64c4dfb1cb5c [sys.catalog]: configured and running, proceeding with master startup.
I20260809 12:15:50.315179 24822 raft_consensus.cc:686] T 00000000000000000000000000000000 P 72b594fdae164a2abe5f64c4dfb1cb5c [term 1 LEADER]: Becoming Leader. State: Replica: 72b594fdae164a2abe5f64c4dfb1cb5c, State: Running, Role: LEADER
I20260809 12:15:50.316043 24822 consensus_queue.cc:237] T 00000000000000000000000000000000 P 72b594fdae164a2abe5f64c4dfb1cb5c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "72b594fdae164a2abe5f64c4dfb1cb5c" member_type: VOTER last_known_addr { host: "127.0.222.62" port: 41159 } }
I20260809 12:15:50.319620 24823 sys_catalog.cc:455] T 00000000000000000000000000000000 P 72b594fdae164a2abe5f64c4dfb1cb5c [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "72b594fdae164a2abe5f64c4dfb1cb5c" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "72b594fdae164a2abe5f64c4dfb1cb5c" member_type: VOTER last_known_addr { host: "127.0.222.62" port: 41159 } } }
I20260809 12:15:50.320052 24823 sys_catalog.cc:458] T 00000000000000000000000000000000 P 72b594fdae164a2abe5f64c4dfb1cb5c [sys.catalog]: This master's current role is: LEADER
I20260809 12:15:50.328585 24824 sys_catalog.cc:455] T 00000000000000000000000000000000 P 72b594fdae164a2abe5f64c4dfb1cb5c [sys.catalog]: SysCatalogTable state changed. Reason: New leader 72b594fdae164a2abe5f64c4dfb1cb5c. Latest consensus state: current_term: 1 leader_uuid: "72b594fdae164a2abe5f64c4dfb1cb5c" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "72b594fdae164a2abe5f64c4dfb1cb5c" member_type: VOTER last_known_addr { host: "127.0.222.62" port: 41159 } } }
I20260809 12:15:50.329085 24824 sys_catalog.cc:458] T 00000000000000000000000000000000 P 72b594fdae164a2abe5f64c4dfb1cb5c [sys.catalog]: This master's current role is: LEADER
I20260809 12:15:50.331259 24812 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 12:15:50.309748 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36487 (local address 127.0.222.62:41159)
0809 12:15:50.309977 (+ 229us) server_negotiation.cc:206] Beginning negotiation
0809 12:15:50.309984 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0809 12:15:50.311604 (+ 1620us) server_negotiation.cc:407] Connection header received
0809 12:15:50.311788 (+ 184us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 12:15:50.311801 (+ 13us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 12:15:50.311908 (+ 107us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 12:15:50.312041 (+ 133us) server_negotiation.cc:226] Negotiated authn=SASL
0809 12:15:50.314665 (+ 2624us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 12:15:50.315878 (+ 1213us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 12:15:50.317651 (+ 1773us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 12:15:50.318192 (+ 541us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 12:15:50.318345 (+ 153us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 12:15:50.322106 (+ 3761us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 12:15:50.322116 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 12:15:50.322125 (+ 9us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0809 12:15:50.322179 (+ 54us) server_negotiation.cc:922] calling sasl_server_start()
0809 12:15:50.327233 (+ 5054us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 12:15:50.328422 (+ 1189us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 12:15:50.328431 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 12:15:50.328437 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0809 12:15:50.328648 (+ 211us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 12:15:50.329311 (+ 663us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 12:15:50.329320 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 12:15:50.329325 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0809 12:15:50.329823 (+ 498us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 12:15:50.330084 (+ 261us) server_negotiation.cc:1035] Waiting for connection context
0809 12:15:50.330681 (+ 597us) server_negotiation.cc:299] Negotiation successful
0809 12:15:50.330853 (+ 172us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":72}
W20260809 12:15:50.335264 24837 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 72b594fdae164a2abe5f64c4dfb1cb5c: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260809 12:15:50.335412 24837 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260809 12:15:50.335563 24838 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260809 12:15:50.337435 24838 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260809 12:15:50.346398 24838 catalog_manager.cc:1347] Generated new cluster ID: b128272901a3438886b93e6ec01d3c4e
I20260809 12:15:50.346537 24838 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260809 12:15:50.359349 24838 catalog_manager.cc:1370] Generated new certificate authority record
I20260809 12:15:50.360713 24838 catalog_manager.cc:1504] Loading token signing keys...
I20260809 12:15:50.374146 24838 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 72b594fdae164a2abe5f64c4dfb1cb5c: Generated new TSK 0
I20260809 12:15:50.375103 24838 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260809 12:15:50.387727 888 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskgH5Pjm/build/asan/bin/kudu pid 24735
I20260809 12:15:50.389904 24804 generic_service.cc:196] Checking for leaks (request via RPC)
I20260809 12:15:50.486418 888 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskgH5Pjm/build/asan/bin/kudu with pid 24735
2026-08-09T12:15:50Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/15 (2192 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/16
Loading random data
Initializing database '/tmp/dist-test-taskgH5Pjm/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1786277382073938-888-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 09 12:15:50 dist-test-slave-2w3w krb5kdc[24845](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 09 12:15:50 dist-test-slave-2w3w krb5kdc[24845](info): set up 2 sockets
Aug 09 12:15:50 dist-test-slave-2w3w krb5kdc[24845](info): commencing operation
krb5kdc: starting...
W20260809 12:15:51.685185 888 mini_kdc.cc:121] Time spent starting KDC: real 1.157s user 0.000s sys 0.051s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskgH5Pjm/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1786277382073938-888-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskgH5Pjm/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1786277382073938-888-0/krb5kdc/test-user.keytab.
Aug 09 12:15:51 dist-test-slave-2w3w krb5kdc[24845](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786277751, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-09T12:15:51Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-09T12:15:51Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.222.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.222.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.222.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskgH5Pjm/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1786277382073938-888-0/krb5kdc/kudu_127.0.222.62.keytab.
Entry for principal kudu/127.0.222.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskgH5Pjm/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1786277382073938-888-0/krb5kdc/kudu_127.0.222.62.keytab.
I20260809 12:15:52.031636 888 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskgH5Pjm/build/asan/bin/kudu
/tmp/dist-test-taskgH5Pjm/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskgH5Pjm/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1786277382073938-888-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskgH5Pjm/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1786277382073938-888-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskgH5Pjm/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1786277382073938-888-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskgH5Pjm/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1786277382073938-888-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.222.62:37859
--webserver_interface=127.0.222.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskgH5Pjm/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1786277382073938-888-0/krb5kdc/kudu_127.0.222.62.keytab
--principal=kudu/127.0.222.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.222.20:43753
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.222.62:37859
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=required
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taskgH5Pjm/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1786277382073938-888-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskgH5Pjm/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1786277382073938-888-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskgH5Pjm/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1786277382073938-888-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260809 12:15:52.383636 24861 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260809 12:15:52.383924 24861 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260809 12:15:52.384089 24861 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260809 12:15:52.391350 24861 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260809 12:15:52.391459 24861 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260809 12:15:52.391606 24861 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260809 12:15:52.391714 24861 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260809 12:15:52.391784 24861 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260809 12:15:52.401631 24861 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.222.20:43753
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskgH5Pjm/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1786277382073938-888-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskgH5Pjm/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1786277382073938-888-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.222.62:37859
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskgH5Pjm/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1786277382073938-888-0/krb5kdc/kudu_127.0.222.62.keytab
--principal=kudu/127.0.222.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.222.62:37859
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskgH5Pjm/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1786277382073938-888-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.222.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskgH5Pjm/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1786277382073938-888-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 09 Aug 2026 11:06:41 UTC on ccycloud.quasar-bwoogy.root.comops.site
build id 4575991
ASAN enabled
I20260809 12:15:52.402998 24861 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260809 12:15:52.404320 24861 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260809 12:15:52.416543 24866 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 12:15:52.416738 24867 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 12:15:52.416759 24869 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260809 12:15:52.417721 24861 server_base.cc:1034] running on GCE node
I20260809 12:15:52.418570 24861 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260809 12:15:52.420092 24861 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260809 12:15:52.421306 24861 hybrid_clock.cc:648] HybridClock initialized: now 1786277752421268 us; error 30 us; skew 500 ppm
I20260809 12:15:52.421700 24861 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 09 12:15:52 dist-test-slave-2w3w krb5kdc[24845](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786277752, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.222.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260809 12:15:52.428815 24861 init.cc:377] Logged in from keytab as kudu/127.0.222.62@KRBTEST.COM (short username kudu)
I20260809 12:15:52.430553 24861 webserver.cc:460] Webserver started at http://127.0.222.62:33421/ using document root <none> and password file <none>
I20260809 12:15:52.431406 24861 fs_manager.cc:359] Metadata directory not provided
I20260809 12:15:52.431576 24861 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260809 12:15:52.431910 24861 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260809 12:15:52.435283 24861 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskgH5Pjm/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1786277382073938-888-0/minicluster-data/master-0/data/instance:
uuid: "5cfcf17dc657471094cbe1578f04cec3"
format_stamp: "Formatted at 2026-08-09 12:15:52 on dist-test-slave-2w3w"
server_key: "e10a9ea9684ae92bf47c4ef5064f9851"
server_key_iv: "91f1b77e492210ce0f2117cd764ed286"
server_key_version: "encryptionkey@0"
I20260809 12:15:52.436110 24861 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskgH5Pjm/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1786277382073938-888-0/minicluster-data/master-0/wal/instance:
uuid: "5cfcf17dc657471094cbe1578f04cec3"
format_stamp: "Formatted at 2026-08-09 12:15:52 on dist-test-slave-2w3w"
server_key: "e10a9ea9684ae92bf47c4ef5064f9851"
server_key_iv: "91f1b77e492210ce0f2117cd764ed286"
server_key_version: "encryptionkey@0"
I20260809 12:15:52.441404 24861 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.000s
I20260809 12:15:52.444947 24878 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260809 12:15:52.446221 24861 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.000s sys 0.002s
I20260809 12:15:52.446410 24861 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskgH5Pjm/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1786277382073938-888-0/minicluster-data/master-0/data,/tmp/dist-test-taskgH5Pjm/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1786277382073938-888-0/minicluster-data/master-0/wal
uuid: "5cfcf17dc657471094cbe1578f04cec3"
format_stamp: "Formatted at 2026-08-09 12:15:52 on dist-test-slave-2w3w"
server_key: "e10a9ea9684ae92bf47c4ef5064f9851"
server_key_iv: "91f1b77e492210ce0f2117cd764ed286"
server_key_version: "encryptionkey@0"
I20260809 12:15:52.446650 24861 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskgH5Pjm/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1786277382073938-888-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskgH5Pjm/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1786277382073938-888-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskgH5Pjm/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1786277382073938-888-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260809 12:15:52.467777 24861 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260809 12:15:52.472258 24861 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260809 12:15:52.472519 24861 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260809 12:15:52.492449 24861 rpc_server.cc:307] RPC server started. Bound to: 127.0.222.62:37859
I20260809 12:15:52.492488 24933 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.222.62:37859 every 8 connection(s)
I20260809 12:15:52.493875 24861 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskgH5Pjm/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1786277382073938-888-0/minicluster-data/master-0/data/info.pb
I20260809 12:15:52.495935 888 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskgH5Pjm/build/asan/bin/kudu as pid 24861
I20260809 12:15:52.496136 888 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskgH5Pjm/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1786277382073938-888-0/minicluster-data/master-0/wal/instance
I20260809 12:15:52.496556 888 external_mini_cluster.cc:1428] Setting key cb20b4834260c301de5664df2c65b27b
I20260809 12:15:52.498202 24934 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260809 12:15:52.511802 24938 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 12:15:52.498383 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45031 (local address 127.0.222.62:37859)
0809 12:15:52.499257 (+ 874us) server_negotiation.cc:206] Beginning negotiation
0809 12:15:52.499272 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0809 12:15:52.499324 (+ 52us) server_negotiation.cc:407] Connection header received
0809 12:15:52.501471 (+ 2147us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 12:15:52.501500 (+ 29us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 12:15:52.502027 (+ 527us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 12:15:52.502502 (+ 475us) server_negotiation.cc:226] Negotiated authn=SASL
0809 12:15:52.503041 (+ 539us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 12:15:52.505053 (+ 2012us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 12:15:52.506482 (+ 1429us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 12:15:52.507561 (+ 1079us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 12:15:52.507758 (+ 197us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 12:15:52.508929 (+ 1171us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 12:15:52.508939 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 12:15:52.508960 (+ 21us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0809 12:15:52.509729 (+ 769us) server_negotiation.cc:922] calling sasl_server_start()
0809 12:15:52.509836 (+ 107us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0809 12:15:52.509869 (+ 33us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 12:15:52.509997 (+ 128us) server_negotiation.cc:1035] Waiting for connection context
0809 12:15:52.510200 (+ 203us) server_negotiation.cc:299] Negotiation successful
0809 12:15:52.510489 (+ 289us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":630,"thread_start_us":330,"threads_started":1}
I20260809 12:15:52.513175 24934 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 5cfcf17dc657471094cbe1578f04cec3: Bootstrap starting.
I20260809 12:15:52.515734 888 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260809 12:15:52.516623 24934 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 5cfcf17dc657471094cbe1578f04cec3: Neither blocks nor log segments found. Creating new log.
I20260809 12:15:52.517622 24934 log.cc:824] T 00000000000000000000000000000000 P 5cfcf17dc657471094cbe1578f04cec3: Log is configured to *not* fsync() on all Append() calls
I20260809 12:15:52.521142 24934 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 5cfcf17dc657471094cbe1578f04cec3: No bootstrap required, opened a new log
I20260809 12:15:52.526631 24938 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 12:15:52.520279 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49723 (local address 127.0.222.62:37859)
0809 12:15:52.520433 (+ 154us) server_negotiation.cc:206] Beginning negotiation
0809 12:15:52.520444 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0809 12:15:52.520951 (+ 507us) server_negotiation.cc:407] Connection header received
0809 12:15:52.521182 (+ 231us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 12:15:52.521186 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 12:15:52.521291 (+ 105us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 12:15:52.521393 (+ 102us) server_negotiation.cc:226] Negotiated authn=SASL
0809 12:15:52.521901 (+ 508us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 12:15:52.522645 (+ 744us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 12:15:52.524033 (+ 1388us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 12:15:52.524524 (+ 491us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 12:15:52.524668 (+ 144us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 12:15:52.525835 (+ 1167us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 12:15:52.525840 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 12:15:52.525843 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0809 12:15:52.525860 (+ 17us) server_negotiation.cc:922] calling sasl_server_start()
0809 12:15:52.525898 (+ 38us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0809 12:15:52.525913 (+ 15us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 12:15:52.526027 (+ 114us) server_negotiation.cc:1035] Waiting for connection context
0809 12:15:52.526193 (+ 166us) server_negotiation.cc:299] Negotiation successful
0809 12:15:52.526293 (+ 100us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":52}
I20260809 12:15:52.529564 24934 raft_consensus.cc:348] T 00000000000000000000000000000000 P 5cfcf17dc657471094cbe1578f04cec3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5cfcf17dc657471094cbe1578f04cec3" member_type: VOTER last_known_addr { host: "127.0.222.62" port: 37859 } }
I20260809 12:15:52.529922 24934 raft_consensus.cc:374] T 00000000000000000000000000000000 P 5cfcf17dc657471094cbe1578f04cec3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260809 12:15:52.530002 24934 raft_consensus.cc:729] T 00000000000000000000000000000000 P 5cfcf17dc657471094cbe1578f04cec3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5cfcf17dc657471094cbe1578f04cec3, State: Initialized, Role: FOLLOWER
I20260809 12:15:52.530509 24934 consensus_queue.cc:260] T 00000000000000000000000000000000 P 5cfcf17dc657471094cbe1578f04cec3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5cfcf17dc657471094cbe1578f04cec3" member_type: VOTER last_known_addr { host: "127.0.222.62" port: 37859 } }
I20260809 12:15:52.530710 24934 raft_consensus.cc:388] T 00000000000000000000000000000000 P 5cfcf17dc657471094cbe1578f04cec3 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260809 12:15:52.530804 24934 raft_consensus.cc:482] T 00000000000000000000000000000000 P 5cfcf17dc657471094cbe1578f04cec3 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260809 12:15:52.530928 24934 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 5cfcf17dc657471094cbe1578f04cec3 [term 0 FOLLOWER]: Advancing to term 1
I20260809 12:15:52.533497 24934 raft_consensus.cc:504] T 00000000000000000000000000000000 P 5cfcf17dc657471094cbe1578f04cec3 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5cfcf17dc657471094cbe1578f04cec3" member_type: VOTER last_known_addr { host: "127.0.222.62" port: 37859 } }
I20260809 12:15:52.533950 24934 leader_election.cc:302] T 00000000000000000000000000000000 P 5cfcf17dc657471094cbe1578f04cec3 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 5cfcf17dc657471094cbe1578f04cec3; no voters:
I20260809 12:15:52.534518 24934 leader_election.cc:288] T 00000000000000000000000000000000 P 5cfcf17dc657471094cbe1578f04cec3 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260809 12:15:52.534679 24945 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 5cfcf17dc657471094cbe1578f04cec3 [term 1 FOLLOWER]: Leader election won for term 1
I20260809 12:15:52.535506 24945 raft_consensus.cc:686] T 00000000000000000000000000000000 P 5cfcf17dc657471094cbe1578f04cec3 [term 1 LEADER]: Becoming Leader. State: Replica: 5cfcf17dc657471094cbe1578f04cec3, State: Running, Role: LEADER
I20260809 12:15:52.536046 24945 consensus_queue.cc:237] T 00000000000000000000000000000000 P 5cfcf17dc657471094cbe1578f04cec3 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5cfcf17dc657471094cbe1578f04cec3" member_type: VOTER last_known_addr { host: "127.0.222.62" port: 37859 } }
I20260809 12:15:52.536919 24934 sys_catalog.cc:564] T 00000000000000000000000000000000 P 5cfcf17dc657471094cbe1578f04cec3 [sys.catalog]: configured and running, proceeding with master startup.
I20260809 12:15:52.539152 24947 sys_catalog.cc:455] T 00000000000000000000000000000000 P 5cfcf17dc657471094cbe1578f04cec3 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 5cfcf17dc657471094cbe1578f04cec3. Latest consensus state: current_term: 1 leader_uuid: "5cfcf17dc657471094cbe1578f04cec3" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5cfcf17dc657471094cbe1578f04cec3" member_type: VOTER last_known_addr { host: "127.0.222.62" port: 37859 } } }
I20260809 12:15:52.539175 24946 sys_catalog.cc:455] T 00000000000000000000000000000000 P 5cfcf17dc657471094cbe1578f04cec3 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "5cfcf17dc657471094cbe1578f04cec3" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5cfcf17dc657471094cbe1578f04cec3" member_type: VOTER last_known_addr { host: "127.0.222.62" port: 37859 } } }
I20260809 12:15:52.539716 24947 sys_catalog.cc:458] T 00000000000000000000000000000000 P 5cfcf17dc657471094cbe1578f04cec3 [sys.catalog]: This master's current role is: LEADER
I20260809 12:15:52.539736 24946 sys_catalog.cc:458] T 00000000000000000000000000000000 P 5cfcf17dc657471094cbe1578f04cec3 [sys.catalog]: This master's current role is: LEADER
I20260809 12:15:52.544035 24951 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260809 12:15:52.551175 24951 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260809 12:15:52.561043 24951 catalog_manager.cc:1347] Generated new cluster ID: 0e62a85eb6154331b990519769e4a795
I20260809 12:15:52.561154 24951 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260809 12:15:52.575903 24951 catalog_manager.cc:1370] Generated new certificate authority record
I20260809 12:15:52.577387 24951 catalog_manager.cc:1504] Loading token signing keys...
I20260809 12:15:52.585603 24951 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 5cfcf17dc657471094cbe1578f04cec3: Generated new TSK 0
I20260809 12:15:52.586541 24951 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260809 12:15:52.606556 888 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskgH5Pjm/build/asan/bin/kudu pid 24861
W20260809 12:15:52.608305 24929 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='slave'} at 127.0.0.1:45031
W20260809 12:15:52.609023 888 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskgH5Pjm/build/asan/bin/kudu pid 24861: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260809 12:15:52.609189 888 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskgH5Pjm/build/asan/bin/kudu with pid 24861
2026-08-09T12:15:52Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/16 (2123 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/17
Loading random data
Initializing database '/tmp/dist-test-taskgH5Pjm/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1786277382073938-888-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 09 12:15:52 dist-test-slave-2w3w krb5kdc[24971](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 09 12:15:52 dist-test-slave-2w3w krb5kdc[24971](info): set up 2 sockets
Aug 09 12:15:52 dist-test-slave-2w3w krb5kdc[24971](info): commencing operation
krb5kdc: starting...
W20260809 12:15:53.792685 888 mini_kdc.cc:121] Time spent starting KDC: real 1.142s user 0.007s sys 0.042s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskgH5Pjm/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1786277382073938-888-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskgH5Pjm/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1786277382073938-888-0/krb5kdc/test-user.keytab.
Aug 09 12:15:54 dist-test-slave-2w3w krb5kdc[24971](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786277754, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-09T12:15:54Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-09T12:15:54Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.222.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.222.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.222.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskgH5Pjm/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1786277382073938-888-0/krb5kdc/kudu_127.0.222.62.keytab.
Entry for principal kudu/127.0.222.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskgH5Pjm/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1786277382073938-888-0/krb5kdc/kudu_127.0.222.62.keytab.
I20260809 12:15:54.134415 888 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskgH5Pjm/build/asan/bin/kudu
/tmp/dist-test-taskgH5Pjm/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskgH5Pjm/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1786277382073938-888-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskgH5Pjm/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1786277382073938-888-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskgH5Pjm/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1786277382073938-888-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskgH5Pjm/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1786277382073938-888-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.222.62:38285
--webserver_interface=127.0.222.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskgH5Pjm/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1786277382073938-888-0/krb5kdc/kudu_127.0.222.62.keytab
--principal=kudu/127.0.222.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.222.20:37941
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.222.62:38285
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=disabled
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taskgH5Pjm/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1786277382073938-888-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskgH5Pjm/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1786277382073938-888-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskgH5Pjm/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1786277382073938-888-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260809 12:15:54.479830 24985 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260809 12:15:54.480145 24985 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260809 12:15:54.480242 24985 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260809 12:15:54.487620 24985 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260809 12:15:54.487738 24985 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260809 12:15:54.487785 24985 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260809 12:15:54.487846 24985 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260809 12:15:54.487883 24985 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260809 12:15:54.499281 24985 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.222.20:37941
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskgH5Pjm/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1786277382073938-888-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskgH5Pjm/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1786277382073938-888-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.222.62:38285
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskgH5Pjm/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1786277382073938-888-0/krb5kdc/kudu_127.0.222.62.keytab
--principal=kudu/127.0.222.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.222.62:38285
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskgH5Pjm/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1786277382073938-888-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.222.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskgH5Pjm/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1786277382073938-888-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 09 Aug 2026 11:06:41 UTC on ccycloud.quasar-bwoogy.root.comops.site
build id 4575991
ASAN enabled
I20260809 12:15:54.500595 24985 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260809 12:15:54.501879 24985 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260809 12:15:54.513818 24991 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 12:15:54.513849 24993 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 12:15:54.513837 24990 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260809 12:15:54.514951 24985 server_base.cc:1034] running on GCE node
I20260809 12:15:54.515779 24985 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260809 12:15:54.517190 24985 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260809 12:15:54.518463 24985 hybrid_clock.cc:648] HybridClock initialized: now 1786277754518421 us; error 51 us; skew 500 ppm
I20260809 12:15:54.518916 24985 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 09 12:15:54 dist-test-slave-2w3w krb5kdc[24971](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786277754, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.222.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260809 12:15:54.522126 24985 init.cc:377] Logged in from keytab as kudu/127.0.222.62@KRBTEST.COM (short username kudu)
I20260809 12:15:54.523648 24985 webserver.cc:460] Webserver started at http://127.0.222.62:40395/ using document root <none> and password file <none>
I20260809 12:15:54.524349 24985 fs_manager.cc:359] Metadata directory not provided
I20260809 12:15:54.524500 24985 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260809 12:15:54.524817 24985 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260809 12:15:54.527897 24985 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskgH5Pjm/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1786277382073938-888-0/minicluster-data/master-0/data/instance:
uuid: "250e40e25cb04271923cf36e80f8b388"
format_stamp: "Formatted at 2026-08-09 12:15:54 on dist-test-slave-2w3w"
server_key: "1e89508ddf61522b0561497ef34f568d"
server_key_iv: "9639577877736b1ef68cc9033c6b3747"
server_key_version: "encryptionkey@0"
I20260809 12:15:54.528676 24985 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskgH5Pjm/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1786277382073938-888-0/minicluster-data/master-0/wal/instance:
uuid: "250e40e25cb04271923cf36e80f8b388"
format_stamp: "Formatted at 2026-08-09 12:15:54 on dist-test-slave-2w3w"
server_key: "1e89508ddf61522b0561497ef34f568d"
server_key_iv: "9639577877736b1ef68cc9033c6b3747"
server_key_version: "encryptionkey@0"
I20260809 12:15:54.534180 24985 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.000s
I20260809 12:15:54.537732 25002 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260809 12:15:54.539018 24985 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260809 12:15:54.539280 24985 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskgH5Pjm/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1786277382073938-888-0/minicluster-data/master-0/data,/tmp/dist-test-taskgH5Pjm/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1786277382073938-888-0/minicluster-data/master-0/wal
uuid: "250e40e25cb04271923cf36e80f8b388"
format_stamp: "Formatted at 2026-08-09 12:15:54 on dist-test-slave-2w3w"
server_key: "1e89508ddf61522b0561497ef34f568d"
server_key_iv: "9639577877736b1ef68cc9033c6b3747"
server_key_version: "encryptionkey@0"
I20260809 12:15:54.539561 24985 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskgH5Pjm/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1786277382073938-888-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskgH5Pjm/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1786277382073938-888-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskgH5Pjm/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1786277382073938-888-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260809 12:15:54.545121 24985 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260809 12:15:54.549036 24985 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260809 12:15:54.549360 24985 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260809 12:15:54.568231 24985 rpc_server.cc:307] RPC server started. Bound to: 127.0.222.62:38285
I20260809 12:15:54.568280 25058 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.222.62:38285 every 8 connection(s)
I20260809 12:15:54.569651 24985 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskgH5Pjm/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1786277382073938-888-0/minicluster-data/master-0/data/info.pb
I20260809 12:15:54.573297 25059 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260809 12:15:54.576627 888 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskgH5Pjm/build/asan/bin/kudu as pid 24985
I20260809 12:15:54.576819 888 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskgH5Pjm/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1786277382073938-888-0/minicluster-data/master-0/wal/instance
I20260809 12:15:54.577297 888 external_mini_cluster.cc:1428] Setting key 34a37aa7f54b78012f4b6354d9657ca7
I20260809 12:15:54.586220 25062 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 12:15:54.579319 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60347 (local address 127.0.222.62:38285)
0809 12:15:54.580099 (+ 780us) server_negotiation.cc:206] Beginning negotiation
0809 12:15:54.580113 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0809 12:15:54.580165 (+ 52us) server_negotiation.cc:407] Connection header received
0809 12:15:54.582342 (+ 2177us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 12:15:54.582376 (+ 34us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 12:15:54.582828 (+ 452us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 12:15:54.583293 (+ 465us) server_negotiation.cc:226] Negotiated authn=SASL
0809 12:15:54.584294 (+ 1001us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 12:15:54.584302 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 12:15:54.584318 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0809 12:15:54.584346 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0809 12:15:54.584416 (+ 70us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0809 12:15:54.584441 (+ 25us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 12:15:54.584680 (+ 239us) server_negotiation.cc:1035] Waiting for connection context
0809 12:15:54.584809 (+ 129us) server_negotiation.cc:299] Negotiation successful
0809 12:15:54.585023 (+ 214us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":537,"thread_start_us":307,"threads_started":1}
I20260809 12:15:54.587217 25059 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 250e40e25cb04271923cf36e80f8b388: Bootstrap starting.
I20260809 12:15:54.590153 888 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260809 12:15:54.590754 25059 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 250e40e25cb04271923cf36e80f8b388: Neither blocks nor log segments found. Creating new log.
I20260809 12:15:54.591948 25059 log.cc:824] T 00000000000000000000000000000000 P 250e40e25cb04271923cf36e80f8b388: Log is configured to *not* fsync() on all Append() calls
I20260809 12:15:54.595203 25059 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 250e40e25cb04271923cf36e80f8b388: No bootstrap required, opened a new log
I20260809 12:15:54.597445 25062 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 12:15:54.595242 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36719 (local address 127.0.222.62:38285)
0809 12:15:54.595456 (+ 214us) server_negotiation.cc:206] Beginning negotiation
0809 12:15:54.595461 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0809 12:15:54.595484 (+ 23us) server_negotiation.cc:407] Connection header received
0809 12:15:54.595540 (+ 56us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 12:15:54.595545 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 12:15:54.595602 (+ 57us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 12:15:54.595749 (+ 147us) server_negotiation.cc:226] Negotiated authn=SASL
0809 12:15:54.596678 (+ 929us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 12:15:54.596687 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 12:15:54.596692 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0809 12:15:54.596710 (+ 18us) server_negotiation.cc:922] calling sasl_server_start()
0809 12:15:54.596777 (+ 67us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0809 12:15:54.596814 (+ 37us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 12:15:54.596954 (+ 140us) server_negotiation.cc:1035] Waiting for connection context
0809 12:15:54.597059 (+ 105us) server_negotiation.cc:299] Negotiation successful
0809 12:15:54.597124 (+ 65us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":105}
I20260809 12:15:54.603544 25059 raft_consensus.cc:348] T 00000000000000000000000000000000 P 250e40e25cb04271923cf36e80f8b388 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "250e40e25cb04271923cf36e80f8b388" member_type: VOTER last_known_addr { host: "127.0.222.62" port: 38285 } }
I20260809 12:15:54.603915 25059 raft_consensus.cc:374] T 00000000000000000000000000000000 P 250e40e25cb04271923cf36e80f8b388 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260809 12:15:54.603998 25059 raft_consensus.cc:729] T 00000000000000000000000000000000 P 250e40e25cb04271923cf36e80f8b388 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 250e40e25cb04271923cf36e80f8b388, State: Initialized, Role: FOLLOWER
I20260809 12:15:54.604498 25059 consensus_queue.cc:260] T 00000000000000000000000000000000 P 250e40e25cb04271923cf36e80f8b388 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "250e40e25cb04271923cf36e80f8b388" member_type: VOTER last_known_addr { host: "127.0.222.62" port: 38285 } }
I20260809 12:15:54.604714 25059 raft_consensus.cc:388] T 00000000000000000000000000000000 P 250e40e25cb04271923cf36e80f8b388 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260809 12:15:54.604820 25059 raft_consensus.cc:482] T 00000000000000000000000000000000 P 250e40e25cb04271923cf36e80f8b388 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260809 12:15:54.604939 25059 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 250e40e25cb04271923cf36e80f8b388 [term 0 FOLLOWER]: Advancing to term 1
I20260809 12:15:54.607403 25059 raft_consensus.cc:504] T 00000000000000000000000000000000 P 250e40e25cb04271923cf36e80f8b388 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "250e40e25cb04271923cf36e80f8b388" member_type: VOTER last_known_addr { host: "127.0.222.62" port: 38285 } }
I20260809 12:15:54.607884 25059 leader_election.cc:302] T 00000000000000000000000000000000 P 250e40e25cb04271923cf36e80f8b388 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 250e40e25cb04271923cf36e80f8b388; no voters:
I20260809 12:15:54.608449 25059 leader_election.cc:288] T 00000000000000000000000000000000 P 250e40e25cb04271923cf36e80f8b388 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260809 12:15:54.608620 25069 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 250e40e25cb04271923cf36e80f8b388 [term 1 FOLLOWER]: Leader election won for term 1
I20260809 12:15:54.609391 25069 raft_consensus.cc:686] T 00000000000000000000000000000000 P 250e40e25cb04271923cf36e80f8b388 [term 1 LEADER]: Becoming Leader. State: Replica: 250e40e25cb04271923cf36e80f8b388, State: Running, Role: LEADER
I20260809 12:15:54.609977 25069 consensus_queue.cc:237] T 00000000000000000000000000000000 P 250e40e25cb04271923cf36e80f8b388 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "250e40e25cb04271923cf36e80f8b388" member_type: VOTER last_known_addr { host: "127.0.222.62" port: 38285 } }
I20260809 12:15:54.610548 25059 sys_catalog.cc:564] T 00000000000000000000000000000000 P 250e40e25cb04271923cf36e80f8b388 [sys.catalog]: configured and running, proceeding with master startup.
I20260809 12:15:54.613235 25070 sys_catalog.cc:455] T 00000000000000000000000000000000 P 250e40e25cb04271923cf36e80f8b388 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "250e40e25cb04271923cf36e80f8b388" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "250e40e25cb04271923cf36e80f8b388" member_type: VOTER last_known_addr { host: "127.0.222.62" port: 38285 } } }
I20260809 12:15:54.613677 25070 sys_catalog.cc:458] T 00000000000000000000000000000000 P 250e40e25cb04271923cf36e80f8b388 [sys.catalog]: This master's current role is: LEADER
I20260809 12:15:54.614454 25071 sys_catalog.cc:455] T 00000000000000000000000000000000 P 250e40e25cb04271923cf36e80f8b388 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 250e40e25cb04271923cf36e80f8b388. Latest consensus state: current_term: 1 leader_uuid: "250e40e25cb04271923cf36e80f8b388" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "250e40e25cb04271923cf36e80f8b388" member_type: VOTER last_known_addr { host: "127.0.222.62" port: 38285 } } }
I20260809 12:15:54.615427 25071 sys_catalog.cc:458] T 00000000000000000000000000000000 P 250e40e25cb04271923cf36e80f8b388 [sys.catalog]: This master's current role is: LEADER
I20260809 12:15:54.617283 25076 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260809 12:15:54.623076 25076 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260809 12:15:54.641624 25076 catalog_manager.cc:1347] Generated new cluster ID: befc797018ca442cb69aa24fda9e7836
I20260809 12:15:54.641769 25076 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260809 12:15:54.663666 25076 catalog_manager.cc:1370] Generated new certificate authority record
I20260809 12:15:54.664119 25076 catalog_manager.cc:1504] Loading token signing keys...
I20260809 12:15:54.673867 25076 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 250e40e25cb04271923cf36e80f8b388: Generated new TSK 0
I20260809 12:15:54.674685 25076 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260809 12:15:54.689770 888 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskgH5Pjm/build/asan/bin/kudu pid 24985
W20260809 12:15:54.691015 25054 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='slave'} at 127.0.0.1:60347
W20260809 12:15:54.691810 888 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskgH5Pjm/build/asan/bin/kudu pid 24985: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260809 12:15:54.692004 888 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskgH5Pjm/build/asan/bin/kudu with pid 24985
2026-08-09T12:15:54Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/17 (2082 ms)
[----------] 18 tests from AuthTokenIssuingTest (38448 ms total)
[----------] 2 tests from ConnectToFollowerMasterTest
[ RUN ] ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys/0
Loading random data
Initializing database '/tmp/dist-test-taskgH5Pjm/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786277382073938-888-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 09 12:15:54 dist-test-slave-2w3w krb5kdc[25091](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 09 12:15:54 dist-test-slave-2w3w krb5kdc[25091](info): set up 2 sockets
Aug 09 12:15:54 dist-test-slave-2w3w krb5kdc[25091](info): commencing operation
krb5kdc: starting...
W20260809 12:15:55.864902 888 mini_kdc.cc:121] Time spent starting KDC: real 1.131s user 0.003s sys 0.046s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskgH5Pjm/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786277382073938-888-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskgH5Pjm/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786277382073938-888-0/krb5kdc/test-user.keytab.
Aug 09 12:15:56 dist-test-slave-2w3w krb5kdc[25091](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786277756, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-09T12:15:56Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-09T12:15:56Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.222.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.222.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.222.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskgH5Pjm/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786277382073938-888-0/krb5kdc/kudu_127.0.222.62.keytab.
Entry for principal kudu/127.0.222.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskgH5Pjm/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786277382073938-888-0/krb5kdc/kudu_127.0.222.62.keytab.
I20260809 12:15:56.210238 888 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskgH5Pjm/build/asan/bin/kudu
/tmp/dist-test-taskgH5Pjm/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskgH5Pjm/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786277382073938-888-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskgH5Pjm/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786277382073938-888-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskgH5Pjm/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786277382073938-888-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskgH5Pjm/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786277382073938-888-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.222.62:43409
--webserver_interface=127.0.222.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskgH5Pjm/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786277382073938-888-0/krb5kdc/kudu_127.0.222.62.keytab
--principal=kudu/127.0.222.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.222.20:38397
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.222.62:43409,127.0.222.61:35943,127.0.222.60:45051
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--enable_leader_failure_detection=false
--rpc_authentication=required
--rpc_encryption=optional with env {KRB5CCNAME=/tmp/dist-test-taskgH5Pjm/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786277382073938-888-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskgH5Pjm/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786277382073938-888-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskgH5Pjm/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786277382073938-888-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260809 12:15:56.560473 25106 flags.cc:425] Enabled unsafe flag: --enable_leader_failure_detection=false
W20260809 12:15:56.560882 25106 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260809 12:15:56.561036 25106 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260809 12:15:56.561187 25106 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260809 12:15:56.568409 25106 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260809 12:15:56.568548 25106 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260809 12:15:56.568656 25106 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260809 12:15:56.568758 25106 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260809 12:15:56.568858 25106 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260809 12:15:56.578613 25106 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.222.20:38397
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_leader_failure_detection=false
--fs_data_dirs=/tmp/dist-test-taskgH5Pjm/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786277382073938-888-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskgH5Pjm/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786277382073938-888-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.222.62:43409,127.0.222.61:35943,127.0.222.60:45051
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskgH5Pjm/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786277382073938-888-0/krb5kdc/kudu_127.0.222.62.keytab
--principal=kudu/127.0.222.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.222.62:43409
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskgH5Pjm/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786277382073938-888-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.222.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskgH5Pjm/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786277382073938-888-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 09 Aug 2026 11:06:41 UTC on ccycloud.quasar-bwoogy.root.comops.site
build id 4575991
ASAN enabled
I20260809 12:15:56.580227 25106 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260809 12:15:56.581715 25106 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260809 12:15:56.594161 25115 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 12:15:56.594345 25112 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 12:15:56.594185 25113 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260809 12:15:56.595570 25106 server_base.cc:1034] running on GCE node
I20260809 12:15:56.596468 25106 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260809 12:15:56.597987 25106 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260809 12:15:56.599241 25106 hybrid_clock.cc:648] HybridClock initialized: now 1786277756599171 us; error 43 us; skew 500 ppm
I20260809 12:15:56.599839 25106 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 09 12:15:56 dist-test-slave-2w3w krb5kdc[25091](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786277756, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.222.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260809 12:15:56.603374 25106 init.cc:377] Logged in from keytab as kudu/127.0.222.62@KRBTEST.COM (short username kudu)
I20260809 12:15:56.605000 25106 webserver.cc:460] Webserver started at http://127.0.222.62:36807/ using document root <none> and password file <none>
I20260809 12:15:56.605739 25106 fs_manager.cc:359] Metadata directory not provided
I20260809 12:15:56.605882 25106 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260809 12:15:56.606247 25106 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260809 12:15:56.609498 25106 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskgH5Pjm/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786277382073938-888-0/minicluster-data/master-0/data/instance:
uuid: "d1efb2909c73417fb178d06b54d1c562"
format_stamp: "Formatted at 2026-08-09 12:15:56 on dist-test-slave-2w3w"
server_key: "082678007e9805e51db78db30e57019a"
server_key_iv: "cd3dbe05feb52acf577cf1e8a2e3fb59"
server_key_version: "encryptionkey@0"
I20260809 12:15:56.610357 25106 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskgH5Pjm/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786277382073938-888-0/minicluster-data/master-0/wal/instance:
uuid: "d1efb2909c73417fb178d06b54d1c562"
format_stamp: "Formatted at 2026-08-09 12:15:56 on dist-test-slave-2w3w"
server_key: "082678007e9805e51db78db30e57019a"
server_key_iv: "cd3dbe05feb52acf577cf1e8a2e3fb59"
server_key_version: "encryptionkey@0"
I20260809 12:15:56.615752 25106 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.007s sys 0.000s
I20260809 12:15:56.619191 25124 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260809 12:15:56.620471 25106 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.001s
I20260809 12:15:56.620661 25106 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskgH5Pjm/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786277382073938-888-0/minicluster-data/master-0/data,/tmp/dist-test-taskgH5Pjm/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786277382073938-888-0/minicluster-data/master-0/wal
uuid: "d1efb2909c73417fb178d06b54d1c562"
format_stamp: "Formatted at 2026-08-09 12:15:56 on dist-test-slave-2w3w"
server_key: "082678007e9805e51db78db30e57019a"
server_key_iv: "cd3dbe05feb52acf577cf1e8a2e3fb59"
server_key_version: "encryptionkey@0"
I20260809 12:15:56.620890 25106 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskgH5Pjm/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786277382073938-888-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskgH5Pjm/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786277382073938-888-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskgH5Pjm/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786277382073938-888-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260809 12:15:56.666435 25106 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260809 12:15:56.670512 25106 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260809 12:15:56.670804 25106 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260809 12:15:56.690224 25106 rpc_server.cc:307] RPC server started. Bound to: 127.0.222.62:43409
I20260809 12:15:56.690244 25183 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.222.62:43409 every 8 connection(s)
I20260809 12:15:56.691722 25106 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskgH5Pjm/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786277382073938-888-0/minicluster-data/master-0/data/info.pb
I20260809 12:15:56.694149 888 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskgH5Pjm/build/asan/bin/kudu as pid 25106
I20260809 12:15:56.694387 888 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskgH5Pjm/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786277382073938-888-0/minicluster-data/master-0/wal/instance
I20260809 12:15:56.695148 888 external_mini_cluster.cc:1428] Setting key 220c522a54b22fcf379da799247d2bb0
I20260809 12:15:56.695714 25184 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260809 12:15:56.702111 25184 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.222.62" port: 43409 } has no permanent_uuid. Determining permanent_uuid...
Aug 09 12:15:56 dist-test-slave-2w3w krb5kdc[25091](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786277756, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.222.62@KRBTEST.COM for kudu/127.0.222.62@KRBTEST.COM
I20260809 12:15:56.725502 25187 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 12:15:56.705369 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.222.62:43409 (local address 127.0.0.1:55592)
0809 12:15:56.706694 (+ 1325us) negotiation.cc:107] Waiting for socket to connect
0809 12:15:56.706734 (+ 40us) client_negotiation.cc:174] Beginning negotiation
0809 12:15:56.708566 (+ 1832us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 12:15:56.709375 (+ 809us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 12:15:56.709381 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 12:15:56.710074 (+ 693us) client_negotiation.cc:189] Negotiated authn=SASL
0809 12:15:56.710433 (+ 359us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 12:15:56.710450 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 12:15:56.712064 (+ 1614us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 12:15:56.712070 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 12:15:56.713150 (+ 1080us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 12:15:56.713163 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 12:15:56.714104 (+ 941us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 12:15:56.714112 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 12:15:56.714296 (+ 184us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 12:15:56.715069 (+ 773us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0809 12:15:56.715091 (+ 22us) client_negotiation.cc:656] calling sasl_client_start()
0809 12:15:56.717520 (+ 2429us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0809 12:15:56.721033 (+ 3513us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0809 12:15:56.721047 (+ 14us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0809 12:15:56.721059 (+ 12us) client_negotiation.cc:760] calling sasl_client_step()
0809 12:15:56.721463 (+ 404us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0809 12:15:56.721888 (+ 425us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0809 12:15:56.721894 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0809 12:15:56.721897 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0809 12:15:56.722072 (+ 175us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0809 12:15:56.722875 (+ 803us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0809 12:15:56.722883 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0809 12:15:56.723143 (+ 260us) client_negotiation.cc:769] Sending connection context
0809 12:15:56.723393 (+ 250us) client_negotiation.cc:240] Negotiation successful
0809 12:15:56.724087 (+ 694us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1135,"mutex_wait_us":164,"thread_start_us":961,"threads_started":1}
I20260809 12:15:56.725538 25186 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 12:15:56.705368 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55592 (local address 127.0.222.62:43409)
0809 12:15:56.706265 (+ 897us) server_negotiation.cc:206] Beginning negotiation
0809 12:15:56.706275 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0809 12:15:56.706866 (+ 591us) server_negotiation.cc:407] Connection header received
0809 12:15:56.708925 (+ 2059us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 12:15:56.708931 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 12:15:56.709160 (+ 229us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 12:15:56.709372 (+ 212us) server_negotiation.cc:226] Negotiated authn=SASL
0809 12:15:56.710648 (+ 1276us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 12:15:56.711906 (+ 1258us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 12:15:56.713302 (+ 1396us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 12:15:56.713853 (+ 551us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 12:15:56.714060 (+ 207us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 12:15:56.717755 (+ 3695us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 12:15:56.717762 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 12:15:56.717774 (+ 12us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0809 12:15:56.717802 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0809 12:15:56.720889 (+ 3087us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 12:15:56.721601 (+ 712us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 12:15:56.721617 (+ 16us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 12:15:56.721619 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0809 12:15:56.721765 (+ 146us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 12:15:56.722199 (+ 434us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 12:15:56.722205 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 12:15:56.722213 (+ 8us) server_negotiation.cc:967] calling sasl_server_step()
0809 12:15:56.722731 (+ 518us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 12:15:56.722981 (+ 250us) server_negotiation.cc:1035] Waiting for connection context
0809 12:15:56.723529 (+ 548us) server_negotiation.cc:299] Negotiation successful
0809 12:15:56.723975 (+ 446us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":66,"server-negotiator.queue_time_us":705,"thread_start_us":483,"threads_started":1}
WARNING: no policy specified for kudu/127.0.222.61@KRBTEST.COM; defaulting to no policy
I20260809 12:15:56.729067 25184 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.222.61" port: 35943 } has no permanent_uuid. Determining permanent_uuid...
I20260809 12:15:56.730159 25187 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 12:15:56.729740 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.222.61:35943 (local address 127.0.0.1:40140)
0809 12:15:56.729899 (+ 159us) negotiation.cc:107] Waiting for socket to connect
0809 12:15:56.730040 (+ 141us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.222.61:35943: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":61}
W20260809 12:15:56.730472 25126 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.222.61:35943: connect: Connection refused (error 111)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.222.61@KRBTEST.COM" created.
I20260809 12:15:56.732149 25187 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 12:15:56.731695 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.222.61:35943 (local address 127.0.0.1:40152)
0809 12:15:56.731928 (+ 233us) negotiation.cc:107] Waiting for socket to connect
0809 12:15:56.731999 (+ 71us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.222.61:35943: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":51}
W20260809 12:15:56.732378 25184 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.222.61:35943: Network error: Client connection negotiation failed: client connection to 127.0.222.61:35943: connect: Connection refused (error 111)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.222.61 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskgH5Pjm/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786277382073938-888-0/krb5kdc/kudu_127.0.222.61.keytab.
Entry for principal kudu/127.0.222.61 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskgH5Pjm/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786277382073938-888-0/krb5kdc/kudu_127.0.222.61.keytab.
I20260809 12:15:56.770529 888 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskgH5Pjm/build/asan/bin/kudu
/tmp/dist-test-taskgH5Pjm/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskgH5Pjm/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786277382073938-888-0/minicluster-data/master-1/wal
--fs_data_dirs=/tmp/dist-test-taskgH5Pjm/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786277382073938-888-0/minicluster-data/master-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskgH5Pjm/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786277382073938-888-0/minicluster-data/master-1/logs
--server_dump_info_path=/tmp/dist-test-taskgH5Pjm/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786277382073938-888-0/minicluster-data/master-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.222.61:35943
--webserver_interface=127.0.222.61
--webserver_port=0
--keytab_file=/tmp/dist-test-taskgH5Pjm/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786277382073938-888-0/krb5kdc/kudu_127.0.222.61.keytab
--principal=kudu/127.0.222.61
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.222.20:38397
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.222.62:43409,127.0.222.61:35943,127.0.222.60:45051
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--enable_leader_failure_detection=false
--rpc_authentication=required
--rpc_encryption=optional with env {KRB5CCNAME=/tmp/dist-test-taskgH5Pjm/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786277382073938-888-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskgH5Pjm/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786277382073938-888-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskgH5Pjm/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786277382073938-888-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260809 12:15:56.781775 25184 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.222.61" port: 35943 } attempt: 1
I20260809 12:15:56.783133 25187 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 12:15:56.782519 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.222.61:35943 (local address 127.0.0.1:40164)
0809 12:15:56.782727 (+ 208us) negotiation.cc:107] Waiting for socket to connect
0809 12:15:56.782877 (+ 150us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.222.61:35943: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":72}
I20260809 12:15:56.784356 25187 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 12:15:56.783899 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.222.61:35943 (local address 127.0.0.1:40178)
0809 12:15:56.784086 (+ 187us) negotiation.cc:107] Waiting for socket to connect
0809 12:15:56.784187 (+ 101us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.222.61:35943: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":58}
W20260809 12:15:56.784654 25184 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.222.61:35943: Network error: Client connection negotiation failed: client connection to 127.0.222.61:35943: connect: Connection refused (error 111)
I20260809 12:15:56.852979 25184 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.222.61" port: 35943 } attempt: 2
I20260809 12:15:56.854398 25187 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 12:15:56.853884 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.222.61:35943 (local address 127.0.0.1:40192)
0809 12:15:56.854146 (+ 262us) negotiation.cc:107] Waiting for socket to connect
0809 12:15:56.854239 (+ 93us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.222.61:35943: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":88}
I20260809 12:15:56.855499 25187 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 12:15:56.855159 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.222.61:35943 (local address 127.0.0.1:40204)
0809 12:15:56.855296 (+ 137us) negotiation.cc:107] Waiting for socket to connect
0809 12:15:56.855375 (+ 79us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.222.61:35943: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":45}
W20260809 12:15:56.855715 25184 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.222.61:35943: Network error: Client connection negotiation failed: client connection to 127.0.222.61:35943: connect: Connection refused (error 111)
I20260809 12:15:56.947106 25184 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.222.61" port: 35943 } attempt: 3
I20260809 12:15:56.948341 25187 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 12:15:56.947861 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.222.61:35943 (local address 127.0.0.1:40212)
0809 12:15:56.948077 (+ 216us) negotiation.cc:107] Waiting for socket to connect
0809 12:15:56.948175 (+ 98us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.222.61:35943: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":86}
I20260809 12:15:56.949471 25187 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 12:15:56.949121 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.222.61:35943 (local address 127.0.0.1:40220)
0809 12:15:56.949262 (+ 141us) negotiation.cc:107] Waiting for socket to connect
0809 12:15:56.949335 (+ 73us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.222.61:35943: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":50}
W20260809 12:15:56.949756 25184 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.222.61:35943: Network error: Client connection negotiation failed: client connection to 127.0.222.61:35943: connect: Connection refused (error 111)
I20260809 12:15:57.093056 25184 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.222.61" port: 35943 } attempt: 4
I20260809 12:15:57.094543 25187 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 12:15:57.093857 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.222.61:35943 (local address 127.0.0.1:40234)
0809 12:15:57.094292 (+ 435us) negotiation.cc:107] Waiting for socket to connect
0809 12:15:57.094402 (+ 110us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.222.61:35943: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":212}
I20260809 12:15:57.095695 25187 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 12:15:57.095319 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.222.61:35943 (local address 127.0.0.1:40250)
0809 12:15:57.095487 (+ 168us) negotiation.cc:107] Waiting for socket to connect
0809 12:15:57.095578 (+ 91us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.222.61:35943: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":67}
W20260809 12:15:57.095885 25184 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.222.61:35943: Network error: Client connection negotiation failed: client connection to 127.0.222.61:35943: connect: Connection refused (error 111)
W20260809 12:15:57.118352 25192 flags.cc:425] Enabled unsafe flag: --enable_leader_failure_detection=false
W20260809 12:15:57.118746 25192 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260809 12:15:57.118911 25192 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260809 12:15:57.119066 25192 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260809 12:15:57.126573 25192 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260809 12:15:57.126694 25192 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260809 12:15:57.126827 25192 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260809 12:15:57.126924 25192 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260809 12:15:57.127055 25192 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260809 12:15:57.137377 25192 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.222.20:38397
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_leader_failure_detection=false
--fs_data_dirs=/tmp/dist-test-taskgH5Pjm/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786277382073938-888-0/minicluster-data/master-1/data
--fs_wal_dir=/tmp/dist-test-taskgH5Pjm/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786277382073938-888-0/minicluster-data/master-1/wal
--ipki_ca_key_size=768
--master_addresses=127.0.222.62:43409,127.0.222.61:35943,127.0.222.60:45051
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskgH5Pjm/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786277382073938-888-0/krb5kdc/kudu_127.0.222.61.keytab
--principal=kudu/127.0.222.61
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.222.61:35943
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskgH5Pjm/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786277382073938-888-0/minicluster-data/master-1/data/info.pb
--webserver_interface=127.0.222.61
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskgH5Pjm/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786277382073938-888-0/minicluster-data/master-1/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 09 Aug 2026 11:06:41 UTC on ccycloud.quasar-bwoogy.root.comops.site
build id 4575991
ASAN enabled
I20260809 12:15:57.138721 25192 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260809 12:15:57.140055 25192 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260809 12:15:57.151729 25197 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 12:15:57.151881 25198 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 12:15:57.151723 25200 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260809 12:15:57.153025 25192 server_base.cc:1034] running on GCE node
I20260809 12:15:57.153824 25192 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260809 12:15:57.155222 25192 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260809 12:15:57.156473 25192 hybrid_clock.cc:648] HybridClock initialized: now 1786277757156384 us; error 54 us; skew 500 ppm
I20260809 12:15:57.156900 25192 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 09 12:15:57 dist-test-slave-2w3w krb5kdc[25091](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786277757, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.222.61@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260809 12:15:57.160259 25192 init.cc:377] Logged in from keytab as kudu/127.0.222.61@KRBTEST.COM (short username kudu)
I20260809 12:15:57.161795 25192 webserver.cc:460] Webserver started at http://127.0.222.61:36583/ using document root <none> and password file <none>
I20260809 12:15:57.162511 25192 fs_manager.cc:359] Metadata directory not provided
I20260809 12:15:57.162652 25192 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260809 12:15:57.163046 25192 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260809 12:15:57.166142 25192 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskgH5Pjm/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786277382073938-888-0/minicluster-data/master-1/data/instance:
uuid: "c5bbf4bc9dd744db8e4f7a5b4ae2ca09"
format_stamp: "Formatted at 2026-08-09 12:15:57 on dist-test-slave-2w3w"
server_key: "b8faa43a681ddce0ac102923628c6f2e"
server_key_iv: "76e70c22d1ca6426813c44d272867c39"
server_key_version: "encryptionkey@0"
I20260809 12:15:57.167024 25192 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskgH5Pjm/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786277382073938-888-0/minicluster-data/master-1/wal/instance:
uuid: "c5bbf4bc9dd744db8e4f7a5b4ae2ca09"
format_stamp: "Formatted at 2026-08-09 12:15:57 on dist-test-slave-2w3w"
server_key: "b8faa43a681ddce0ac102923628c6f2e"
server_key_iv: "76e70c22d1ca6426813c44d272867c39"
server_key_version: "encryptionkey@0"
I20260809 12:15:57.172376 25192 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.000s sys 0.004s
I20260809 12:15:57.176005 25207 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260809 12:15:57.177358 25192 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.002s
I20260809 12:15:57.177564 25192 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskgH5Pjm/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786277382073938-888-0/minicluster-data/master-1/data,/tmp/dist-test-taskgH5Pjm/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786277382073938-888-0/minicluster-data/master-1/wal
uuid: "c5bbf4bc9dd744db8e4f7a5b4ae2ca09"
format_stamp: "Formatted at 2026-08-09 12:15:57 on dist-test-slave-2w3w"
server_key: "b8faa43a681ddce0ac102923628c6f2e"
server_key_iv: "76e70c22d1ca6426813c44d272867c39"
server_key_version: "encryptionkey@0"
I20260809 12:15:57.177839 25192 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskgH5Pjm/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786277382073938-888-0/minicluster-data/master-1/wal
metadata directory: /tmp/dist-test-taskgH5Pjm/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786277382073938-888-0/minicluster-data/master-1/wal
1 data directories: /tmp/dist-test-taskgH5Pjm/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786277382073938-888-0/minicluster-data/master-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260809 12:15:57.193558 25192 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260809 12:15:57.198176 25192 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260809 12:15:57.198462 25192 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260809 12:15:57.217326 25192 rpc_server.cc:307] RPC server started. Bound to: 127.0.222.61:35943
I20260809 12:15:57.217345 25261 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.222.61:35943 every 8 connection(s)
I20260809 12:15:57.218732 25192 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskgH5Pjm/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786277382073938-888-0/minicluster-data/master-1/data/info.pb
I20260809 12:15:57.222409 25262 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260809 12:15:57.224690 888 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskgH5Pjm/build/asan/bin/kudu as pid 25192
I20260809 12:15:57.224931 888 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskgH5Pjm/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786277382073938-888-0/minicluster-data/master-1/wal/instance
I20260809 12:15:57.225539 888 external_mini_cluster.cc:1428] Setting key 92d08e104237f6ca863a030948a64504
I20260809 12:15:57.229360 25262 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.222.62" port: 43409 } has no permanent_uuid. Determining permanent_uuid...
Aug 09 12:15:57 dist-test-slave-2w3w krb5kdc[25091](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786277757, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.222.61@KRBTEST.COM for kudu/127.0.222.62@KRBTEST.COM
I20260809 12:15:57.252055 25268 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 12:15:57.232063 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55608 (local address 127.0.222.62:43409)
0809 12:15:57.232548 (+ 485us) server_negotiation.cc:206] Beginning negotiation
0809 12:15:57.232554 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0809 12:15:57.233892 (+ 1338us) server_negotiation.cc:407] Connection header received
0809 12:15:57.236381 (+ 2489us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 12:15:57.236390 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 12:15:57.236490 (+ 100us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 12:15:57.236620 (+ 130us) server_negotiation.cc:226] Negotiated authn=SASL
0809 12:15:57.238551 (+ 1931us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 12:15:57.239752 (+ 1201us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 12:15:57.241539 (+ 1787us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 12:15:57.241964 (+ 425us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 12:15:57.242130 (+ 166us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 12:15:57.245759 (+ 3629us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 12:15:57.245764 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 12:15:57.245768 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0809 12:15:57.245815 (+ 47us) server_negotiation.cc:922] calling sasl_server_start()
0809 12:15:57.248682 (+ 2867us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 12:15:57.249528 (+ 846us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 12:15:57.249532 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 12:15:57.249535 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0809 12:15:57.249665 (+ 130us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 12:15:57.250132 (+ 467us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 12:15:57.250137 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 12:15:57.250142 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0809 12:15:57.250443 (+ 301us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 12:15:57.250637 (+ 194us) server_negotiation.cc:1035] Waiting for connection context
0809 12:15:57.251430 (+ 793us) server_negotiation.cc:299] Negotiation successful
0809 12:15:57.251675 (+ 245us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":381,"thread_start_us":272,"threads_started":1}
I20260809 12:15:57.253113 25269 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 12:15:57.232658 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.222.62:43409 (local address 127.0.0.1:55608)
0809 12:15:57.233503 (+ 845us) negotiation.cc:107] Waiting for socket to connect
0809 12:15:57.233560 (+ 57us) client_negotiation.cc:174] Beginning negotiation
0809 12:15:57.236067 (+ 2507us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 12:15:57.236858 (+ 791us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 12:15:57.236870 (+ 12us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 12:15:57.237578 (+ 708us) client_negotiation.cc:189] Negotiated authn=SASL
0809 12:15:57.238302 (+ 724us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 12:15:57.238324 (+ 22us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 12:15:57.239918 (+ 1594us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 12:15:57.239925 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 12:15:57.241363 (+ 1438us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 12:15:57.241375 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 12:15:57.242208 (+ 833us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 12:15:57.242215 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 12:15:57.242465 (+ 250us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 12:15:57.243203 (+ 738us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0809 12:15:57.243227 (+ 24us) client_negotiation.cc:656] calling sasl_client_start()
0809 12:15:57.245502 (+ 2275us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0809 12:15:57.248848 (+ 3346us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0809 12:15:57.248857 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0809 12:15:57.248875 (+ 18us) client_negotiation.cc:760] calling sasl_client_step()
0809 12:15:57.249368 (+ 493us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0809 12:15:57.249788 (+ 420us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0809 12:15:57.249793 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0809 12:15:57.249797 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0809 12:15:57.250007 (+ 210us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0809 12:15:57.250598 (+ 591us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0809 12:15:57.250608 (+ 10us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0809 12:15:57.251041 (+ 433us) client_negotiation.cc:769] Sending connection context
0809 12:15:57.251333 (+ 292us) client_negotiation.cc:240] Negotiation successful
0809 12:15:57.251870 (+ 537us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":571,"thread_start_us":368,"threads_started":1}
I20260809 12:15:57.254900 25262 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.222.61" port: 35943 } has no permanent_uuid. Determining permanent_uuid...
WARNING: no policy specified for kudu/127.0.222.60@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.222.60@KRBTEST.COM" created.
Aug 09 12:15:57 dist-test-slave-2w3w krb5kdc[25091](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786277757, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.222.61@KRBTEST.COM for kudu/127.0.222.61@KRBTEST.COM
I20260809 12:15:57.271198 25269 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 12:15:57.255680 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.222.61:35943 (local address 127.0.0.1:40262)
0809 12:15:57.255885 (+ 205us) negotiation.cc:107] Waiting for socket to connect
0809 12:15:57.255904 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0809 12:15:57.256004 (+ 100us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 12:15:57.257236 (+ 1232us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 12:15:57.257240 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 12:15:57.257612 (+ 372us) client_negotiation.cc:189] Negotiated authn=SASL
0809 12:15:57.257756 (+ 144us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 12:15:57.257764 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 12:15:57.259660 (+ 1896us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 12:15:57.259669 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 12:15:57.260570 (+ 901us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 12:15:57.260582 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 12:15:57.261715 (+ 1133us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 12:15:57.261719 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 12:15:57.261863 (+ 144us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 12:15:57.262490 (+ 627us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0809 12:15:57.262510 (+ 20us) client_negotiation.cc:656] calling sasl_client_start()
0809 12:15:57.264599 (+ 2089us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0809 12:15:57.268825 (+ 4226us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0809 12:15:57.268828 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0809 12:15:57.268831 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0809 12:15:57.269176 (+ 345us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0809 12:15:57.269582 (+ 406us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0809 12:15:57.269585 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0809 12:15:57.269587 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0809 12:15:57.269723 (+ 136us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0809 12:15:57.270448 (+ 725us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0809 12:15:57.270451 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0809 12:15:57.270567 (+ 116us) client_negotiation.cc:769] Sending connection context
0809 12:15:57.270674 (+ 107us) client_negotiation.cc:240] Negotiation successful
0809 12:15:57.270868 (+ 194us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":56}
I20260809 12:15:57.271665 25271 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 12:15:57.255959 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40262 (local address 127.0.222.61:35943)
0809 12:15:57.256659 (+ 700us) server_negotiation.cc:206] Beginning negotiation
0809 12:15:57.256670 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0809 12:15:57.256707 (+ 37us) server_negotiation.cc:407] Connection header received
0809 12:15:57.256781 (+ 74us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 12:15:57.256790 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 12:15:57.257062 (+ 272us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 12:15:57.257237 (+ 175us) server_negotiation.cc:226] Negotiated authn=SASL
0809 12:15:57.257959 (+ 722us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 12:15:57.259457 (+ 1498us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 12:15:57.260739 (+ 1282us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 12:15:57.261530 (+ 791us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 12:15:57.261685 (+ 155us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 12:15:57.264825 (+ 3140us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 12:15:57.264840 (+ 15us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 12:15:57.264862 (+ 22us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0809 12:15:57.264906 (+ 44us) server_negotiation.cc:922] calling sasl_server_start()
0809 12:15:57.268690 (+ 3784us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 12:15:57.269309 (+ 619us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 12:15:57.269320 (+ 11us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 12:15:57.269325 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0809 12:15:57.269471 (+ 146us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 12:15:57.269827 (+ 356us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 12:15:57.269835 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 12:15:57.269841 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0809 12:15:57.270314 (+ 473us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 12:15:57.270524 (+ 210us) server_negotiation.cc:1035] Waiting for connection context
0809 12:15:57.270873 (+ 349us) server_negotiation.cc:299] Negotiation successful
0809 12:15:57.271179 (+ 306us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":487,"thread_start_us":310,"threads_started":1}
I20260809 12:15:57.274132 25262 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.222.60" port: 45051 } has no permanent_uuid. Determining permanent_uuid...
I20260809 12:15:57.275287 25269 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 12:15:57.274800 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.222.60:45051 (local address 127.0.0.1:34360)
0809 12:15:57.275035 (+ 235us) negotiation.cc:107] Waiting for socket to connect
0809 12:15:57.275157 (+ 122us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.222.60:45051: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":61}
W20260809 12:15:57.275556 25209 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.222.60:45051: connect: Connection refused (error 111)
I20260809 12:15:57.277257 25269 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 12:15:57.276817 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.222.60:45051 (local address 127.0.0.1:34376)
0809 12:15:57.277001 (+ 184us) negotiation.cc:107] Waiting for socket to connect
0809 12:15:57.277110 (+ 109us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.222.60:45051: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":62}
W20260809 12:15:57.277597 25262 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.222.60:45051: Network error: Client connection negotiation failed: client connection to 127.0.222.60:45051: connect: Connection refused (error 111)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.222.60 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskgH5Pjm/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786277382073938-888-0/krb5kdc/kudu_127.0.222.60.keytab.
Entry for principal kudu/127.0.222.60 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskgH5Pjm/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786277382073938-888-0/krb5kdc/kudu_127.0.222.60.keytab.
I20260809 12:15:57.302652 888 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskgH5Pjm/build/asan/bin/kudu
/tmp/dist-test-taskgH5Pjm/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskgH5Pjm/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786277382073938-888-0/minicluster-data/master-2/wal
--fs_data_dirs=/tmp/dist-test-taskgH5Pjm/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786277382073938-888-0/minicluster-data/master-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskgH5Pjm/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786277382073938-888-0/minicluster-data/master-2/logs
--server_dump_info_path=/tmp/dist-test-taskgH5Pjm/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786277382073938-888-0/minicluster-data/master-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.222.60:45051
--webserver_interface=127.0.222.60
--webserver_port=0
--keytab_file=/tmp/dist-test-taskgH5Pjm/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786277382073938-888-0/krb5kdc/kudu_127.0.222.60.keytab
--principal=kudu/127.0.222.60
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.222.20:38397
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.222.62:43409,127.0.222.61:35943,127.0.222.60:45051
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--enable_leader_failure_detection=false
--rpc_authentication=required
--rpc_encryption=optional with env {KRB5CCNAME=/tmp/dist-test-taskgH5Pjm/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786277382073938-888-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskgH5Pjm/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786277382073938-888-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskgH5Pjm/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786277382073938-888-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260809 12:15:57.326996 25262 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.222.60" port: 45051 } attempt: 1
I20260809 12:15:57.328276 25269 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 12:15:57.327812 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.222.60:45051 (local address 127.0.0.1:34378)
0809 12:15:57.328019 (+ 207us) negotiation.cc:107] Waiting for socket to connect
0809 12:15:57.328115 (+ 96us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.222.60:45051: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":77}
I20260809 12:15:57.329501 25269 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 12:15:57.329101 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.222.60:45051 (local address 127.0.0.1:34382)
0809 12:15:57.329277 (+ 176us) negotiation.cc:107] Waiting for socket to connect
0809 12:15:57.329396 (+ 119us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.222.60:45051: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":64}
W20260809 12:15:57.329806 25262 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.222.60:45051: Network error: Client connection negotiation failed: client connection to 127.0.222.60:45051: connect: Connection refused (error 111)
I20260809 12:15:57.395087 25184 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.222.61" port: 35943 } attempt: 5
I20260809 12:15:57.400065 25262 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.222.60" port: 45051 } attempt: 2
I20260809 12:15:57.401695 25269 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 12:15:57.400983 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.222.60:45051 (local address 127.0.0.1:34398)
0809 12:15:57.401392 (+ 409us) negotiation.cc:107] Waiting for socket to connect
0809 12:15:57.401506 (+ 114us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.222.60:45051: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":293}
I20260809 12:15:57.403110 25269 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 12:15:57.402649 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.222.60:45051 (local address 127.0.0.1:34406)
0809 12:15:57.402823 (+ 174us) negotiation.cc:107] Waiting for socket to connect
0809 12:15:57.402912 (+ 89us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.222.60:45051: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":74}
W20260809 12:15:57.403366 25262 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.222.60:45051: Network error: Client connection negotiation failed: client connection to 127.0.222.60:45051: connect: Connection refused (error 111)
Aug 09 12:15:57 dist-test-slave-2w3w krb5kdc[25091](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786277756, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.222.62@KRBTEST.COM for kudu/127.0.222.61@KRBTEST.COM
I20260809 12:15:57.413453 25271 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 12:15:57.396583 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40266 (local address 127.0.222.61:35943)
0809 12:15:57.396811 (+ 228us) server_negotiation.cc:206] Beginning negotiation
0809 12:15:57.396819 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0809 12:15:57.396841 (+ 22us) server_negotiation.cc:407] Connection header received
0809 12:15:57.396913 (+ 72us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 12:15:57.396920 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 12:15:57.397035 (+ 115us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 12:15:57.397146 (+ 111us) server_negotiation.cc:226] Negotiated authn=SASL
0809 12:15:57.398568 (+ 1422us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 12:15:57.399890 (+ 1322us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 12:15:57.402137 (+ 2247us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 12:15:57.403117 (+ 980us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 12:15:57.403279 (+ 162us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 12:15:57.407414 (+ 4135us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 12:15:57.407421 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 12:15:57.407425 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0809 12:15:57.407466 (+ 41us) server_negotiation.cc:922] calling sasl_server_start()
0809 12:15:57.410746 (+ 3280us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 12:15:57.411526 (+ 780us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 12:15:57.411531 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 12:15:57.411533 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0809 12:15:57.411654 (+ 121us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 12:15:57.412044 (+ 390us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 12:15:57.412049 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 12:15:57.412054 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0809 12:15:57.412340 (+ 286us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 12:15:57.412503 (+ 163us) server_negotiation.cc:1035] Waiting for connection context
0809 12:15:57.412897 (+ 394us) server_negotiation.cc:299] Negotiation successful
0809 12:15:57.413113 (+ 216us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":55}
I20260809 12:15:57.413453 25187 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 12:15:57.395930 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.222.61:35943 (local address 127.0.0.1:40266)
0809 12:15:57.396165 (+ 235us) negotiation.cc:107] Waiting for socket to connect
0809 12:15:57.396190 (+ 25us) client_negotiation.cc:174] Beginning negotiation
0809 12:15:57.396355 (+ 165us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 12:15:57.397402 (+ 1047us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 12:15:57.397410 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 12:15:57.398062 (+ 652us) client_negotiation.cc:189] Negotiated authn=SASL
0809 12:15:57.398353 (+ 291us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 12:15:57.398371 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 12:15:57.400509 (+ 2138us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 12:15:57.400517 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 12:15:57.401940 (+ 1423us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 12:15:57.401957 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 12:15:57.403553 (+ 1596us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 12:15:57.403563 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 12:15:57.403835 (+ 272us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 12:15:57.404545 (+ 710us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0809 12:15:57.404577 (+ 32us) client_negotiation.cc:656] calling sasl_client_start()
0809 12:15:57.407205 (+ 2628us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0809 12:15:57.410885 (+ 3680us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0809 12:15:57.410889 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0809 12:15:57.410892 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0809 12:15:57.411378 (+ 486us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0809 12:15:57.411772 (+ 394us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0809 12:15:57.411777 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0809 12:15:57.411780 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0809 12:15:57.411929 (+ 149us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0809 12:15:57.412477 (+ 548us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0809 12:15:57.412481 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0809 12:15:57.412637 (+ 156us) client_negotiation.cc:769] Sending connection context
0809 12:15:57.412805 (+ 168us) client_negotiation.cc:240] Negotiation successful
0809 12:15:57.413102 (+ 297us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":63}
I20260809 12:15:57.414494 25184 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.222.60" port: 45051 } has no permanent_uuid. Determining permanent_uuid...
I20260809 12:15:57.415517 25187 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 12:15:57.415146 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.222.60:45051 (local address 127.0.0.1:34408)
0809 12:15:57.415304 (+ 158us) negotiation.cc:107] Waiting for socket to connect
0809 12:15:57.415385 (+ 81us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.222.60:45051: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":50}
I20260809 12:15:57.416348 25187 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 12:15:57.416049 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.222.60:45051 (local address 127.0.0.1:34420)
0809 12:15:57.416182 (+ 133us) negotiation.cc:107] Waiting for socket to connect
0809 12:15:57.416250 (+ 68us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.222.60:45051: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":46}
W20260809 12:15:57.416579 25184 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.222.60:45051: Network error: Client connection negotiation failed: client connection to 127.0.222.60:45051: connect: Connection refused (error 111)
I20260809 12:15:57.467828 25184 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.222.60" port: 45051 } attempt: 1
I20260809 12:15:57.469218 25187 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 12:15:57.468707 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.222.60:45051 (local address 127.0.0.1:34432)
0809 12:15:57.468947 (+ 240us) negotiation.cc:107] Waiting for socket to connect
0809 12:15:57.469044 (+ 97us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.222.60:45051: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":90}
I20260809 12:15:57.470280 25187 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 12:15:57.469942 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.222.60:45051 (local address 127.0.0.1:34434)
0809 12:15:57.470094 (+ 152us) negotiation.cc:107] Waiting for socket to connect
0809 12:15:57.470174 (+ 80us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.222.60:45051: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":50}
W20260809 12:15:57.470523 25184 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.222.60:45051: Network error: Client connection negotiation failed: client connection to 127.0.222.60:45051: connect: Connection refused (error 111)
I20260809 12:15:57.494635 25262 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.222.60" port: 45051 } attempt: 3
I20260809 12:15:57.496052 25269 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 12:15:57.495457 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.222.60:45051 (local address 127.0.0.1:34440)
0809 12:15:57.495724 (+ 267us) negotiation.cc:107] Waiting for socket to connect
0809 12:15:57.495847 (+ 123us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.222.60:45051: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":87}
I20260809 12:15:57.497246 25269 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 12:15:57.496755 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.222.60:45051 (local address 127.0.0.1:34454)
0809 12:15:57.497017 (+ 262us) negotiation.cc:107] Waiting for socket to connect
0809 12:15:57.497107 (+ 90us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.222.60:45051: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":155}
W20260809 12:15:57.497494 25262 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.222.60:45051: Network error: Client connection negotiation failed: client connection to 127.0.222.60:45051: connect: Connection refused (error 111)
I20260809 12:15:57.538733 25184 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.222.60" port: 45051 } attempt: 2
I20260809 12:15:57.539988 25187 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 12:15:57.539463 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.222.60:45051 (local address 127.0.0.1:34460)
0809 12:15:57.539702 (+ 239us) negotiation.cc:107] Waiting for socket to connect
0809 12:15:57.539817 (+ 115us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.222.60:45051: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":75}
I20260809 12:15:57.541029 25187 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 12:15:57.540686 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.222.60:45051 (local address 127.0.0.1:34468)
0809 12:15:57.540855 (+ 169us) negotiation.cc:107] Waiting for socket to connect
0809 12:15:57.540921 (+ 66us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.222.60:45051: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":51}
W20260809 12:15:57.541267 25184 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.222.60:45051: Network error: Client connection negotiation failed: client connection to 127.0.222.60:45051: connect: Connection refused (error 111)
I20260809 12:15:57.640736 25262 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.222.60" port: 45051 } attempt: 4
I20260809 12:15:57.642045 25269 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 12:15:57.641429 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.222.60:45051 (local address 127.0.0.1:34482)
0809 12:15:57.641703 (+ 274us) negotiation.cc:107] Waiting for socket to connect
0809 12:15:57.641825 (+ 122us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.222.60:45051: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":88}
I20260809 12:15:57.643188 25269 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 12:15:57.642748 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.222.60:45051 (local address 127.0.0.1:34498)
0809 12:15:57.642921 (+ 173us) negotiation.cc:107] Waiting for socket to connect
0809 12:15:57.643038 (+ 117us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.222.60:45051: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":54}
W20260809 12:15:57.643498 25262 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.222.60:45051: Network error: Client connection negotiation failed: client connection to 127.0.222.60:45051: connect: Connection refused (error 111)
I20260809 12:15:57.647490 25184 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.222.60" port: 45051 } attempt: 3
I20260809 12:15:57.648455 25187 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 12:15:57.648026 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.222.60:45051 (local address 127.0.0.1:34510)
0809 12:15:57.648211 (+ 185us) negotiation.cc:107] Waiting for socket to connect
0809 12:15:57.648298 (+ 87us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.222.60:45051: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":61}
I20260809 12:15:57.649410 25187 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 12:15:57.649060 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.222.60:45051 (local address 127.0.0.1:34522)
0809 12:15:57.649242 (+ 182us) negotiation.cc:107] Waiting for socket to connect
0809 12:15:57.649316 (+ 74us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.222.60:45051: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":52}
W20260809 12:15:57.649667 25184 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.222.60:45051: Network error: Client connection negotiation failed: client connection to 127.0.222.60:45051: connect: Connection refused (error 111)
W20260809 12:15:57.671797 25275 flags.cc:425] Enabled unsafe flag: --enable_leader_failure_detection=false
W20260809 12:15:57.672180 25275 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260809 12:15:57.672299 25275 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260809 12:15:57.672437 25275 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260809 12:15:57.680393 25275 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260809 12:15:57.680488 25275 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260809 12:15:57.680593 25275 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260809 12:15:57.680657 25275 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260809 12:15:57.680804 25275 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260809 12:15:57.691584 25275 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.222.20:38397
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_leader_failure_detection=false
--fs_data_dirs=/tmp/dist-test-taskgH5Pjm/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786277382073938-888-0/minicluster-data/master-2/data
--fs_wal_dir=/tmp/dist-test-taskgH5Pjm/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786277382073938-888-0/minicluster-data/master-2/wal
--ipki_ca_key_size=768
--master_addresses=127.0.222.62:43409,127.0.222.61:35943,127.0.222.60:45051
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskgH5Pjm/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786277382073938-888-0/krb5kdc/kudu_127.0.222.60.keytab
--principal=kudu/127.0.222.60
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.222.60:45051
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskgH5Pjm/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786277382073938-888-0/minicluster-data/master-2/data/info.pb
--webserver_interface=127.0.222.60
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskgH5Pjm/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786277382073938-888-0/minicluster-data/master-2/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 09 Aug 2026 11:06:41 UTC on ccycloud.quasar-bwoogy.root.comops.site
build id 4575991
ASAN enabled
I20260809 12:15:57.692945 25275 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260809 12:15:57.694341 25275 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260809 12:15:57.706296 25281 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 12:15:57.706259 25282 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 12:15:57.706339 25284 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260809 12:15:57.707338 25275 server_base.cc:1034] running on GCE node
I20260809 12:15:57.708158 25275 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260809 12:15:57.709537 25275 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260809 12:15:57.710870 25275 hybrid_clock.cc:648] HybridClock initialized: now 1786277757710786 us; error 56 us; skew 500 ppm
I20260809 12:15:57.711258 25275 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 09 12:15:57 dist-test-slave-2w3w krb5kdc[25091](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786277757, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.222.60@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260809 12:15:57.719578 25275 init.cc:377] Logged in from keytab as kudu/127.0.222.60@KRBTEST.COM (short username kudu)
I20260809 12:15:57.721263 25275 webserver.cc:460] Webserver started at http://127.0.222.60:40223/ using document root <none> and password file <none>
I20260809 12:15:57.721961 25275 fs_manager.cc:359] Metadata directory not provided
I20260809 12:15:57.722055 25275 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260809 12:15:57.722337 25275 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260809 12:15:57.725646 25275 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskgH5Pjm/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786277382073938-888-0/minicluster-data/master-2/data/instance:
uuid: "3a27d05012a94847b04e0675cd82d824"
format_stamp: "Formatted at 2026-08-09 12:15:57 on dist-test-slave-2w3w"
server_key: "3958c14cca82b08552f047aa2db4e5a0"
server_key_iv: "f85d58392d8cf268e8d8f5215b6b09fe"
server_key_version: "encryptionkey@0"
I20260809 12:15:57.726420 25275 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskgH5Pjm/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786277382073938-888-0/minicluster-data/master-2/wal/instance:
uuid: "3a27d05012a94847b04e0675cd82d824"
format_stamp: "Formatted at 2026-08-09 12:15:57 on dist-test-slave-2w3w"
server_key: "3958c14cca82b08552f047aa2db4e5a0"
server_key_iv: "f85d58392d8cf268e8d8f5215b6b09fe"
server_key_version: "encryptionkey@0"
I20260809 12:15:57.731688 25275 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.002s
I20260809 12:15:57.735170 25291 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260809 12:15:57.736492 25275 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.000s sys 0.004s
I20260809 12:15:57.736635 25275 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskgH5Pjm/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786277382073938-888-0/minicluster-data/master-2/data,/tmp/dist-test-taskgH5Pjm/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786277382073938-888-0/minicluster-data/master-2/wal
uuid: "3a27d05012a94847b04e0675cd82d824"
format_stamp: "Formatted at 2026-08-09 12:15:57 on dist-test-slave-2w3w"
server_key: "3958c14cca82b08552f047aa2db4e5a0"
server_key_iv: "f85d58392d8cf268e8d8f5215b6b09fe"
server_key_version: "encryptionkey@0"
I20260809 12:15:57.736809 25275 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskgH5Pjm/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786277382073938-888-0/minicluster-data/master-2/wal
metadata directory: /tmp/dist-test-taskgH5Pjm/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786277382073938-888-0/minicluster-data/master-2/wal
1 data directories: /tmp/dist-test-taskgH5Pjm/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786277382073938-888-0/minicluster-data/master-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260809 12:15:57.762360 25275 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260809 12:15:57.766724 25275 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260809 12:15:57.766950 25275 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260809 12:15:57.785775 25275 rpc_server.cc:307] RPC server started. Bound to: 127.0.222.60:45051
I20260809 12:15:57.785797 25344 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.222.60:45051 every 8 connection(s)
I20260809 12:15:57.787273 25275 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskgH5Pjm/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786277382073938-888-0/minicluster-data/master-2/data/info.pb
I20260809 12:15:57.788444 888 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskgH5Pjm/build/asan/bin/kudu as pid 25275
I20260809 12:15:57.788661 888 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskgH5Pjm/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786277382073938-888-0/minicluster-data/master-2/wal/instance
I20260809 12:15:57.789227 888 external_mini_cluster.cc:1428] Setting key 1372eb66e0a89aaf78da6d80079ecf8a
I20260809 12:15:57.791970 25345 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260809 12:15:57.800817 25345 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.222.62" port: 43409 } has no permanent_uuid. Determining permanent_uuid...
Aug 09 12:15:57 dist-test-slave-2w3w krb5kdc[25091](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786277756, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.222.60@KRBTEST.COM
Aug 09 12:15:57 dist-test-slave-2w3w krb5kdc[25091](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786277757, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.222.60@KRBTEST.COM for kudu/127.0.222.62@KRBTEST.COM
I20260809 12:15:57.813958 25348 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 12:15:57.791381 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34524 (local address 127.0.222.60:45051)
0809 12:15:57.792233 (+ 852us) server_negotiation.cc:206] Beginning negotiation
0809 12:15:57.792244 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0809 12:15:57.792288 (+ 44us) server_negotiation.cc:407] Connection header received
0809 12:15:57.793912 (+ 1624us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 12:15:57.793931 (+ 19us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 12:15:57.794294 (+ 363us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 12:15:57.794634 (+ 340us) server_negotiation.cc:226] Negotiated authn=SASL
0809 12:15:57.796570 (+ 1936us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 12:15:57.798631 (+ 2061us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 12:15:57.800046 (+ 1415us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 12:15:57.801099 (+ 1053us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 12:15:57.801295 (+ 196us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 12:15:57.806422 (+ 5127us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 12:15:57.806430 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 12:15:57.806440 (+ 10us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0809 12:15:57.806519 (+ 79us) server_negotiation.cc:922] calling sasl_server_start()
0809 12:15:57.809651 (+ 3132us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 12:15:57.810391 (+ 740us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 12:15:57.810404 (+ 13us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 12:15:57.810411 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0809 12:15:57.810651 (+ 240us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 12:15:57.811089 (+ 438us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 12:15:57.811092 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 12:15:57.811094 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0809 12:15:57.811674 (+ 580us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 12:15:57.812000 (+ 326us) server_negotiation.cc:1035] Waiting for connection context
0809 12:15:57.812399 (+ 399us) server_negotiation.cc:299] Negotiation successful
0809 12:15:57.812840 (+ 441us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":647,"spinlock_wait_cycles":13184,"thread_start_us":366,"threads_started":1}
I20260809 12:15:57.820397 25352 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 12:15:57.804075 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.222.62:43409 (local address 127.0.0.1:55616)
0809 12:15:57.805154 (+ 1079us) negotiation.cc:107] Waiting for socket to connect
0809 12:15:57.805195 (+ 41us) client_negotiation.cc:174] Beginning negotiation
0809 12:15:57.805361 (+ 166us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 12:15:57.805845 (+ 484us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 12:15:57.805850 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 12:15:57.806479 (+ 629us) client_negotiation.cc:189] Negotiated authn=SASL
0809 12:15:57.806769 (+ 290us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 12:15:57.806777 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 12:15:57.807877 (+ 1100us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 12:15:57.807881 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 12:15:57.809011 (+ 1130us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 12:15:57.809023 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 12:15:57.809799 (+ 776us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 12:15:57.809806 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 12:15:57.809979 (+ 173us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 12:15:57.810564 (+ 585us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0809 12:15:57.810582 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0809 12:15:57.813027 (+ 2445us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0809 12:15:57.817709 (+ 4682us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0809 12:15:57.817722 (+ 13us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0809 12:15:57.817729 (+ 7us) client_negotiation.cc:760] calling sasl_client_step()
0809 12:15:57.818120 (+ 391us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0809 12:15:57.818542 (+ 422us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0809 12:15:57.818547 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0809 12:15:57.818556 (+ 9us) client_negotiation.cc:760] calling sasl_client_step()
0809 12:15:57.818682 (+ 126us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0809 12:15:57.819258 (+ 576us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0809 12:15:57.819265 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0809 12:15:57.819447 (+ 182us) client_negotiation.cc:769] Sending connection context
0809 12:15:57.819638 (+ 191us) client_negotiation.cc:240] Negotiation successful
0809 12:15:57.819934 (+ 296us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":825,"thread_start_us":653,"threads_started":1}
I20260809 12:15:57.820393 25349 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 12:15:57.804103 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55616 (local address 127.0.222.62:43409)
0809 12:15:57.804800 (+ 697us) server_negotiation.cc:206] Beginning negotiation
0809 12:15:57.804807 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0809 12:15:57.805314 (+ 507us) server_negotiation.cc:407] Connection header received
0809 12:15:57.805511 (+ 197us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 12:15:57.805518 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 12:15:57.805632 (+ 114us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 12:15:57.805767 (+ 135us) server_negotiation.cc:226] Negotiated authn=SASL
0809 12:15:57.806936 (+ 1169us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 12:15:57.807692 (+ 756us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 12:15:57.809170 (+ 1478us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 12:15:57.809600 (+ 430us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 12:15:57.809747 (+ 147us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 12:15:57.813378 (+ 3631us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 12:15:57.813383 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 12:15:57.813386 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0809 12:15:57.813420 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0809 12:15:57.817575 (+ 4155us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 12:15:57.818245 (+ 670us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 12:15:57.818252 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 12:15:57.818257 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0809 12:15:57.818406 (+ 149us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 12:15:57.818793 (+ 387us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 12:15:57.818798 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 12:15:57.818802 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0809 12:15:57.819129 (+ 327us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 12:15:57.819314 (+ 185us) server_negotiation.cc:1035] Waiting for connection context
0809 12:15:57.819747 (+ 433us) server_negotiation.cc:299] Negotiation successful
0809 12:15:57.819955 (+ 208us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":551,"thread_start_us":316,"threads_started":1}
I20260809 12:15:57.821704 25345 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.222.61" port: 35943 } has no permanent_uuid. Determining permanent_uuid...
Aug 09 12:15:57 dist-test-slave-2w3w krb5kdc[25091](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786277756, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.222.62@KRBTEST.COM
I20260809 12:15:57.826874 25184 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.222.60" port: 45051 } attempt: 4
Aug 09 12:15:57 dist-test-slave-2w3w krb5kdc[25091](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786277757, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.222.60@KRBTEST.COM for kudu/127.0.222.61@KRBTEST.COM
I20260809 12:15:57.834841 25353 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 12:15:57.817105 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55618 (local address 127.0.222.62:43409)
0809 12:15:57.817784 (+ 679us) server_negotiation.cc:206] Beginning negotiation
0809 12:15:57.817788 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0809 12:15:57.817807 (+ 19us) server_negotiation.cc:407] Connection header received
0809 12:15:57.817846 (+ 39us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 12:15:57.817849 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 12:15:57.817906 (+ 57us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 12:15:57.818022 (+ 116us) server_negotiation.cc:226] Negotiated authn=SASL
0809 12:15:57.819288 (+ 1266us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 12:15:57.820084 (+ 796us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 12:15:57.821671 (+ 1587us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 12:15:57.822165 (+ 494us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 12:15:57.822299 (+ 134us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 12:15:57.827282 (+ 4983us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 12:15:57.827289 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 12:15:57.827296 (+ 7us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0809 12:15:57.827336 (+ 40us) server_negotiation.cc:922] calling sasl_server_start()
0809 12:15:57.831030 (+ 3694us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 12:15:57.831760 (+ 730us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 12:15:57.831766 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 12:15:57.831769 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0809 12:15:57.831998 (+ 229us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 12:15:57.832487 (+ 489us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 12:15:57.832492 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 12:15:57.832495 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0809 12:15:57.832756 (+ 261us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 12:15:57.833110 (+ 354us) server_negotiation.cc:1035] Waiting for connection context
0809 12:15:57.834076 (+ 966us) server_negotiation.cc:299] Negotiation successful
0809 12:15:57.834351 (+ 275us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":574,"thread_start_us":393,"threads_started":1}
Aug 09 12:15:57 dist-test-slave-2w3w krb5kdc[25091](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786277756, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.222.62@KRBTEST.COM for kudu/127.0.222.60@KRBTEST.COM
I20260809 12:15:57.839054 25352 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 12:15:57.822357 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.222.61:35943 (local address 127.0.0.1:40270)
0809 12:15:57.822561 (+ 204us) negotiation.cc:107] Waiting for socket to connect
0809 12:15:57.822582 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0809 12:15:57.822679 (+ 97us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 12:15:57.823133 (+ 454us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 12:15:57.823137 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 12:15:57.823603 (+ 466us) client_negotiation.cc:189] Negotiated authn=SASL
0809 12:15:57.823791 (+ 188us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 12:15:57.823804 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 12:15:57.824815 (+ 1011us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 12:15:57.824823 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 12:15:57.826124 (+ 1301us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 12:15:57.826137 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 12:15:57.827374 (+ 1237us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 12:15:57.827381 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 12:15:57.827603 (+ 222us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 12:15:57.828431 (+ 828us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0809 12:15:57.828455 (+ 24us) client_negotiation.cc:656] calling sasl_client_start()
0809 12:15:57.831155 (+ 2700us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0809 12:15:57.835542 (+ 4387us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0809 12:15:57.835545 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0809 12:15:57.835548 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0809 12:15:57.836057 (+ 509us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0809 12:15:57.836771 (+ 714us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0809 12:15:57.836776 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0809 12:15:57.836779 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0809 12:15:57.837047 (+ 268us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0809 12:15:57.837871 (+ 824us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0809 12:15:57.837874 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0809 12:15:57.838065 (+ 191us) client_negotiation.cc:769] Sending connection context
0809 12:15:57.838257 (+ 192us) client_negotiation.cc:240] Negotiation successful
0809 12:15:57.838601 (+ 344us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":94}
I20260809 12:15:57.839107 25271 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 12:15:57.822597 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40270 (local address 127.0.222.61:35943)
0809 12:15:57.822785 (+ 188us) server_negotiation.cc:206] Beginning negotiation
0809 12:15:57.822792 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0809 12:15:57.822812 (+ 20us) server_negotiation.cc:407] Connection header received
0809 12:15:57.822876 (+ 64us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 12:15:57.822883 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 12:15:57.822992 (+ 109us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 12:15:57.823083 (+ 91us) server_negotiation.cc:226] Negotiated authn=SASL
0809 12:15:57.823947 (+ 864us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 12:15:57.824689 (+ 742us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 12:15:57.826274 (+ 1585us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 12:15:57.826978 (+ 704us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 12:15:57.827161 (+ 183us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 12:15:57.831353 (+ 4192us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 12:15:57.831360 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 12:15:57.831365 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0809 12:15:57.831408 (+ 43us) server_negotiation.cc:922] calling sasl_server_start()
0809 12:15:57.835379 (+ 3971us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 12:15:57.836456 (+ 1077us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 12:15:57.836460 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 12:15:57.836462 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0809 12:15:57.836631 (+ 169us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 12:15:57.837236 (+ 605us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 12:15:57.837245 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 12:15:57.837250 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0809 12:15:57.837635 (+ 385us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 12:15:57.837781 (+ 146us) server_negotiation.cc:1035] Waiting for connection context
0809 12:15:57.838366 (+ 585us) server_negotiation.cc:299] Negotiation successful
0809 12:15:57.838637 (+ 271us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":57}
I20260809 12:15:57.840658 25345 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.222.60" port: 45051 } has no permanent_uuid. Determining permanent_uuid...
I20260809 12:15:57.843665 25187 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 12:15:57.827535 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.222.60:45051 (local address 127.0.0.1:34528)
0809 12:15:57.827714 (+ 179us) negotiation.cc:107] Waiting for socket to connect
0809 12:15:57.827733 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0809 12:15:57.827831 (+ 98us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 12:15:57.828338 (+ 507us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 12:15:57.828345 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 12:15:57.828886 (+ 541us) client_negotiation.cc:189] Negotiated authn=SASL
0809 12:15:57.829114 (+ 228us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 12:15:57.829129 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 12:15:57.830340 (+ 1211us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 12:15:57.830345 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 12:15:57.831966 (+ 1621us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 12:15:57.831995 (+ 29us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 12:15:57.833017 (+ 1022us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 12:15:57.833024 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 12:15:57.833260 (+ 236us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 12:15:57.833773 (+ 513us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0809 12:15:57.833789 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0809 12:15:57.836371 (+ 2582us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0809 12:15:57.840531 (+ 4160us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0809 12:15:57.840536 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0809 12:15:57.840540 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0809 12:15:57.841202 (+ 662us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0809 12:15:57.841721 (+ 519us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0809 12:15:57.841726 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0809 12:15:57.841730 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0809 12:15:57.841908 (+ 178us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0809 12:15:57.842616 (+ 708us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0809 12:15:57.842621 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0809 12:15:57.842794 (+ 173us) client_negotiation.cc:769] Sending connection context
0809 12:15:57.842996 (+ 202us) client_negotiation.cc:240] Negotiation successful
0809 12:15:57.843289 (+ 293us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":53}
I20260809 12:15:57.843781 25348 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 12:15:57.827837 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34528 (local address 127.0.222.60:45051)
0809 12:15:57.828022 (+ 185us) server_negotiation.cc:206] Beginning negotiation
0809 12:15:57.828045 (+ 23us) server_negotiation.cc:399] Waiting for connection header
0809 12:15:57.828068 (+ 23us) server_negotiation.cc:407] Connection header received
0809 12:15:57.828114 (+ 46us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 12:15:57.828121 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 12:15:57.828192 (+ 71us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 12:15:57.828290 (+ 98us) server_negotiation.cc:226] Negotiated authn=SASL
0809 12:15:57.829255 (+ 965us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 12:15:57.830184 (+ 929us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 12:15:57.832109 (+ 1925us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 12:15:57.832807 (+ 698us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 12:15:57.833546 (+ 739us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 12:15:57.836601 (+ 3055us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 12:15:57.836607 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 12:15:57.836611 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0809 12:15:57.836647 (+ 36us) server_negotiation.cc:922] calling sasl_server_start()
0809 12:15:57.840395 (+ 3748us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 12:15:57.841352 (+ 957us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 12:15:57.841358 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 12:15:57.841361 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0809 12:15:57.841582 (+ 221us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 12:15:57.842040 (+ 458us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 12:15:57.842045 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 12:15:57.842049 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0809 12:15:57.842407 (+ 358us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 12:15:57.842551 (+ 144us) server_negotiation.cc:1035] Waiting for connection context
0809 12:15:57.843105 (+ 554us) server_negotiation.cc:299] Negotiation successful
0809 12:15:57.843377 (+ 272us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":63}
Aug 09 12:15:57 dist-test-slave-2w3w krb5kdc[25091](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786277756, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.222.61@KRBTEST.COM
Aug 09 12:15:57 dist-test-slave-2w3w krb5kdc[25091](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786277757, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.222.60@KRBTEST.COM for kudu/127.0.222.60@KRBTEST.COM
I20260809 12:15:57.850061 25184 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P d1efb2909c73417fb178d06b54d1c562: Bootstrap starting.
I20260809 12:15:57.853608 25184 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P d1efb2909c73417fb178d06b54d1c562: Neither blocks nor log segments found. Creating new log.
I20260809 12:15:57.855046 25184 log.cc:824] T 00000000000000000000000000000000 P d1efb2909c73417fb178d06b54d1c562: Log is configured to *not* fsync() on all Append() calls
I20260809 12:15:57.855562 25354 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 12:15:57.836778 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40278 (local address 127.0.222.61:35943)
0809 12:15:57.837642 (+ 864us) server_negotiation.cc:206] Beginning negotiation
0809 12:15:57.837650 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0809 12:15:57.837681 (+ 31us) server_negotiation.cc:407] Connection header received
0809 12:15:57.837731 (+ 50us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 12:15:57.837754 (+ 23us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 12:15:57.837829 (+ 75us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 12:15:57.837948 (+ 119us) server_negotiation.cc:226] Negotiated authn=SASL
0809 12:15:57.839730 (+ 1782us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 12:15:57.840808 (+ 1078us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 12:15:57.842582 (+ 1774us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 12:15:57.843237 (+ 655us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 12:15:57.846587 (+ 3350us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 12:15:57.848512 (+ 1925us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 12:15:57.848522 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 12:15:57.848528 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0809 12:15:57.848572 (+ 44us) server_negotiation.cc:922] calling sasl_server_start()
0809 12:15:57.852317 (+ 3745us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 12:15:57.852998 (+ 681us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 12:15:57.853006 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 12:15:57.853010 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0809 12:15:57.853174 (+ 164us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 12:15:57.853693 (+ 519us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 12:15:57.853699 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 12:15:57.853703 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0809 12:15:57.854069 (+ 366us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 12:15:57.854353 (+ 284us) server_negotiation.cc:1035] Waiting for connection context
0809 12:15:57.854771 (+ 418us) server_negotiation.cc:299] Negotiation successful
0809 12:15:57.855090 (+ 319us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":720,"thread_start_us":511,"threads_started":1}
I20260809 12:15:57.857072 25352 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 12:15:57.841351 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.222.60:45051 (local address 127.0.0.1:34534)
0809 12:15:57.841649 (+ 298us) negotiation.cc:107] Waiting for socket to connect
0809 12:15:57.841673 (+ 24us) client_negotiation.cc:174] Beginning negotiation
0809 12:15:57.841792 (+ 119us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 12:15:57.842566 (+ 774us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 12:15:57.842570 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 12:15:57.843054 (+ 484us) client_negotiation.cc:189] Negotiated authn=SASL
0809 12:15:57.843228 (+ 174us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 12:15:57.843244 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 12:15:57.844673 (+ 1429us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 12:15:57.844680 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 12:15:57.845935 (+ 1255us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 12:15:57.845949 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 12:15:57.846920 (+ 971us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 12:15:57.846928 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 12:15:57.847164 (+ 236us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 12:15:57.847746 (+ 582us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0809 12:15:57.847779 (+ 33us) client_negotiation.cc:656] calling sasl_client_start()
0809 12:15:57.850091 (+ 2312us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0809 12:15:57.854517 (+ 4426us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0809 12:15:57.854523 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0809 12:15:57.854527 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0809 12:15:57.855079 (+ 552us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0809 12:15:57.855466 (+ 387us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0809 12:15:57.855470 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0809 12:15:57.855474 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0809 12:15:57.855634 (+ 160us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0809 12:15:57.856229 (+ 595us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0809 12:15:57.856233 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0809 12:15:57.856364 (+ 131us) client_negotiation.cc:769] Sending connection context
0809 12:15:57.856453 (+ 89us) client_negotiation.cc:240] Negotiation successful
0809 12:15:57.856735 (+ 282us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":165}
I20260809 12:15:57.858083 25355 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 12:15:57.841397 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34534 (local address 127.0.222.60:45051)
0809 12:15:57.842330 (+ 933us) server_negotiation.cc:206] Beginning negotiation
0809 12:15:57.842334 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0809 12:15:57.842351 (+ 17us) server_negotiation.cc:407] Connection header received
0809 12:15:57.842384 (+ 33us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 12:15:57.842387 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 12:15:57.842432 (+ 45us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 12:15:57.842547 (+ 115us) server_negotiation.cc:226] Negotiated authn=SASL
0809 12:15:57.843412 (+ 865us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 12:15:57.844547 (+ 1135us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 12:15:57.846098 (+ 1551us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 12:15:57.846732 (+ 634us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 12:15:57.846857 (+ 125us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 12:15:57.850331 (+ 3474us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 12:15:57.850341 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 12:15:57.850348 (+ 7us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0809 12:15:57.850395 (+ 47us) server_negotiation.cc:922] calling sasl_server_start()
0809 12:15:57.854356 (+ 3961us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 12:15:57.855206 (+ 850us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 12:15:57.855211 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 12:15:57.855214 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0809 12:15:57.855357 (+ 143us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 12:15:57.855774 (+ 417us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 12:15:57.855779 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 12:15:57.855781 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0809 12:15:57.856103 (+ 322us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 12:15:57.857530 (+ 1427us) server_negotiation.cc:1035] Waiting for connection context
0809 12:15:57.857659 (+ 129us) server_negotiation.cc:299] Negotiation successful
0809 12:15:57.857805 (+ 146us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":843,"thread_start_us":641,"threads_started":1}
I20260809 12:15:57.858899 25184 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P d1efb2909c73417fb178d06b54d1c562: No bootstrap required, opened a new log
I20260809 12:15:57.861018 888 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260809 12:15:57.863853 25345 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 3a27d05012a94847b04e0675cd82d824: Bootstrap starting.
I20260809 12:15:57.864753 25154 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "00000000000000000000000000000000"
dest_uuid: "d1efb2909c73417fb178d06b54d1c562"
from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:55618
I20260809 12:15:57.867182 25345 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 3a27d05012a94847b04e0675cd82d824: Neither blocks nor log segments found. Creating new log.
I20260809 12:15:57.868270 25345 log.cc:824] T 00000000000000000000000000000000 P 3a27d05012a94847b04e0675cd82d824: Log is configured to *not* fsync() on all Append() calls
I20260809 12:15:57.871647 25345 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 3a27d05012a94847b04e0675cd82d824: No bootstrap required, opened a new log
I20260809 12:15:57.871948 25184 raft_consensus.cc:348] T 00000000000000000000000000000000 P d1efb2909c73417fb178d06b54d1c562 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d1efb2909c73417fb178d06b54d1c562" member_type: VOTER last_known_addr { host: "127.0.222.62" port: 43409 } } peers { permanent_uuid: "c5bbf4bc9dd744db8e4f7a5b4ae2ca09" member_type: VOTER last_known_addr { host: "127.0.222.61" port: 35943 } } peers { permanent_uuid: "3a27d05012a94847b04e0675cd82d824" member_type: VOTER last_known_addr { host: "127.0.222.60" port: 45051 } }
I20260809 12:15:57.872290 25184 raft_consensus.cc:729] T 00000000000000000000000000000000 P d1efb2909c73417fb178d06b54d1c562 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d1efb2909c73417fb178d06b54d1c562, State: Initialized, Role: FOLLOWER
I20260809 12:15:57.872730 25184 consensus_queue.cc:260] T 00000000000000000000000000000000 P d1efb2909c73417fb178d06b54d1c562 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d1efb2909c73417fb178d06b54d1c562" member_type: VOTER last_known_addr { host: "127.0.222.62" port: 43409 } } peers { permanent_uuid: "c5bbf4bc9dd744db8e4f7a5b4ae2ca09" member_type: VOTER last_known_addr { host: "127.0.222.61" port: 35943 } } peers { permanent_uuid: "3a27d05012a94847b04e0675cd82d824" member_type: VOTER last_known_addr { host: "127.0.222.60" port: 45051 } }
I20260809 12:15:57.873723 25358 sys_catalog.cc:455] T 00000000000000000000000000000000 P d1efb2909c73417fb178d06b54d1c562 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d1efb2909c73417fb178d06b54d1c562" member_type: VOTER last_known_addr { host: "127.0.222.62" port: 43409 } } peers { permanent_uuid: "c5bbf4bc9dd744db8e4f7a5b4ae2ca09" member_type: VOTER last_known_addr { host: "127.0.222.61" port: 35943 } } peers { permanent_uuid: "3a27d05012a94847b04e0675cd82d824" member_type: VOTER last_known_addr { host: "127.0.222.60" port: 45051 } } }
I20260809 12:15:57.874167 25358 sys_catalog.cc:458] T 00000000000000000000000000000000 P d1efb2909c73417fb178d06b54d1c562 [sys.catalog]: This master's current role is: FOLLOWER
I20260809 12:15:57.875056 25184 sys_catalog.cc:564] T 00000000000000000000000000000000 P d1efb2909c73417fb178d06b54d1c562 [sys.catalog]: configured and running, proceeding with master startup.
I20260809 12:15:57.875144 25154 raft_consensus.cc:482] T 00000000000000000000000000000000 P d1efb2909c73417fb178d06b54d1c562 [term 0 FOLLOWER]: Starting forced leader election (received explicit request)
I20260809 12:15:57.875337 25154 raft_consensus.cc:3037] T 00000000000000000000000000000000 P d1efb2909c73417fb178d06b54d1c562 [term 0 FOLLOWER]: Advancing to term 1
I20260809 12:15:57.879771 25154 raft_consensus.cc:504] T 00000000000000000000000000000000 P d1efb2909c73417fb178d06b54d1c562 [term 1 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d1efb2909c73417fb178d06b54d1c562" member_type: VOTER last_known_addr { host: "127.0.222.62" port: 43409 } } peers { permanent_uuid: "c5bbf4bc9dd744db8e4f7a5b4ae2ca09" member_type: VOTER last_known_addr { host: "127.0.222.61" port: 35943 } } peers { permanent_uuid: "3a27d05012a94847b04e0675cd82d824" member_type: VOTER last_known_addr { host: "127.0.222.60" port: 45051 } }
I20260809 12:15:57.880173 25345 raft_consensus.cc:348] T 00000000000000000000000000000000 P 3a27d05012a94847b04e0675cd82d824 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d1efb2909c73417fb178d06b54d1c562" member_type: VOTER last_known_addr { host: "127.0.222.62" port: 43409 } } peers { permanent_uuid: "c5bbf4bc9dd744db8e4f7a5b4ae2ca09" member_type: VOTER last_known_addr { host: "127.0.222.61" port: 35943 } } peers { permanent_uuid: "3a27d05012a94847b04e0675cd82d824" member_type: VOTER last_known_addr { host: "127.0.222.60" port: 45051 } }
I20260809 12:15:57.880539 25345 raft_consensus.cc:729] T 00000000000000000000000000000000 P 3a27d05012a94847b04e0675cd82d824 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3a27d05012a94847b04e0675cd82d824, State: Initialized, Role: FOLLOWER
I20260809 12:15:57.881044 25345 consensus_queue.cc:260] T 00000000000000000000000000000000 P 3a27d05012a94847b04e0675cd82d824 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d1efb2909c73417fb178d06b54d1c562" member_type: VOTER last_known_addr { host: "127.0.222.62" port: 43409 } } peers { permanent_uuid: "c5bbf4bc9dd744db8e4f7a5b4ae2ca09" member_type: VOTER last_known_addr { host: "127.0.222.61" port: 35943 } } peers { permanent_uuid: "3a27d05012a94847b04e0675cd82d824" member_type: VOTER last_known_addr { host: "127.0.222.60" port: 45051 } }
I20260809 12:15:57.881702 25154 leader_election.cc:288] T 00000000000000000000000000000000 P d1efb2909c73417fb178d06b54d1c562 [CANDIDATE]: Term 1 election: Requested vote from peers c5bbf4bc9dd744db8e4f7a5b4ae2ca09 (127.0.222.61:35943), 3a27d05012a94847b04e0675cd82d824 (127.0.222.60:45051)
I20260809 12:15:57.882948 25361 sys_catalog.cc:455] T 00000000000000000000000000000000 P 3a27d05012a94847b04e0675cd82d824 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d1efb2909c73417fb178d06b54d1c562" member_type: VOTER last_known_addr { host: "127.0.222.62" port: 43409 } } peers { permanent_uuid: "c5bbf4bc9dd744db8e4f7a5b4ae2ca09" member_type: VOTER last_known_addr { host: "127.0.222.61" port: 35943 } } peers { permanent_uuid: "3a27d05012a94847b04e0675cd82d824" member_type: VOTER last_known_addr { host: "127.0.222.60" port: 45051 } } }
I20260809 12:15:57.883574 25361 sys_catalog.cc:458] T 00000000000000000000000000000000 P 3a27d05012a94847b04e0675cd82d824 [sys.catalog]: This master's current role is: FOLLOWER
I20260809 12:15:57.884142 25345 sys_catalog.cc:564] T 00000000000000000000000000000000 P 3a27d05012a94847b04e0675cd82d824 [sys.catalog]: configured and running, proceeding with master startup.
I20260809 12:15:57.885661 25320 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "d1efb2909c73417fb178d06b54d1c562" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: true dest_uuid: "3a27d05012a94847b04e0675cd82d824"
I20260809 12:15:57.886211 25320 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 3a27d05012a94847b04e0675cd82d824 [term 0 FOLLOWER]: Advancing to term 1
I20260809 12:15:57.891041 25320 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 3a27d05012a94847b04e0675cd82d824 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d1efb2909c73417fb178d06b54d1c562 in term 1.
I20260809 12:15:57.894483 25126 leader_election.cc:302] T 00000000000000000000000000000000 P d1efb2909c73417fb178d06b54d1c562 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3a27d05012a94847b04e0675cd82d824, d1efb2909c73417fb178d06b54d1c562; no voters:
I20260809 12:15:57.895080 25358 raft_consensus.cc:2781] T 00000000000000000000000000000000 P d1efb2909c73417fb178d06b54d1c562 [term 1 FOLLOWER]: Leader election won for term 1
W20260809 12:15:57.905707 25375 catalog_manager.cc:1558] T 00000000000000000000000000000000 P d1efb2909c73417fb178d06b54d1c562: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260809 12:15:57.905875 25375 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260809 12:15:57.906236 25358 raft_consensus.cc:686] T 00000000000000000000000000000000 P d1efb2909c73417fb178d06b54d1c562 [term 1 LEADER]: Becoming Leader. State: Replica: d1efb2909c73417fb178d06b54d1c562, State: Running, Role: LEADER
I20260809 12:15:57.906816 25358 consensus_queue.cc:237] T 00000000000000000000000000000000 P d1efb2909c73417fb178d06b54d1c562 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d1efb2909c73417fb178d06b54d1c562" member_type: VOTER last_known_addr { host: "127.0.222.62" port: 43409 } } peers { permanent_uuid: "c5bbf4bc9dd744db8e4f7a5b4ae2ca09" member_type: VOTER last_known_addr { host: "127.0.222.61" port: 35943 } } peers { permanent_uuid: "3a27d05012a94847b04e0675cd82d824" member_type: VOTER last_known_addr { host: "127.0.222.60" port: 45051 } }
I20260809 12:15:57.916676 25380 sys_catalog.cc:455] T 00000000000000000000000000000000 P d1efb2909c73417fb178d06b54d1c562 [sys.catalog]: SysCatalogTable state changed. Reason: New leader d1efb2909c73417fb178d06b54d1c562. Latest consensus state: current_term: 1 leader_uuid: "d1efb2909c73417fb178d06b54d1c562" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d1efb2909c73417fb178d06b54d1c562" member_type: VOTER last_known_addr { host: "127.0.222.62" port: 43409 } } peers { permanent_uuid: "c5bbf4bc9dd744db8e4f7a5b4ae2ca09" member_type: VOTER last_known_addr { host: "127.0.222.61" port: 35943 } } peers { permanent_uuid: "3a27d05012a94847b04e0675cd82d824" member_type: VOTER last_known_addr { host: "127.0.222.60" port: 45051 } } }
I20260809 12:15:57.917183 25380 sys_catalog.cc:458] T 00000000000000000000000000000000 P d1efb2909c73417fb178d06b54d1c562 [sys.catalog]: This master's current role is: LEADER
I20260809 12:15:57.918700 25387 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260809 12:15:57.895051 25106 master_runner.cc:186] Error getting master registration for 127.0.222.61:35943: OK, instance_id { permanent_uuid: "c5bbf4bc9dd744db8e4f7a5b4ae2ca09" instance_seqno: 1786277757212031 } error { code: CATALOG_MANAGER_NOT_INITIALIZED status { code: SERVICE_UNAVAILABLE message: "Catalog manager is not initialized. State: Starting" } }
I20260809 12:15:57.919240 25106 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
W20260809 12:15:57.923768 25131 tablet.cc:2369] T 00000000000000000000000000000000 P d1efb2909c73417fb178d06b54d1c562: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260809 12:15:57.924685 25387 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260809 12:15:57.942721 25262 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.222.60" port: 45051 } attempt: 5
I20260809 12:15:57.953841 25320 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 3a27d05012a94847b04e0675cd82d824 [term 1 FOLLOWER]: Refusing update from remote peer d1efb2909c73417fb178d06b54d1c562: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
Aug 09 12:15:57 dist-test-slave-2w3w krb5kdc[25091](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786277757, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.222.61@KRBTEST.COM for kudu/127.0.222.60@KRBTEST.COM
I20260809 12:15:57.954995 25358 consensus_queue.cc:1035] T 00000000000000000000000000000000 P d1efb2909c73417fb178d06b54d1c562 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3a27d05012a94847b04e0675cd82d824" member_type: VOTER last_known_addr { host: "127.0.222.60" port: 45051 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260809 12:15:57.962554 25269 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 12:15:57.943524 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.222.60:45051 (local address 127.0.0.1:34550)
0809 12:15:57.944064 (+ 540us) negotiation.cc:107] Waiting for socket to connect
0809 12:15:57.944089 (+ 25us) client_negotiation.cc:174] Beginning negotiation
0809 12:15:57.944201 (+ 112us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 12:15:57.945057 (+ 856us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 12:15:57.945062 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 12:15:57.945578 (+ 516us) client_negotiation.cc:189] Negotiated authn=SASL
0809 12:15:57.945775 (+ 197us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 12:15:57.945787 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 12:15:57.947258 (+ 1471us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 12:15:57.947267 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 12:15:57.949197 (+ 1930us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 12:15:57.949215 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 12:15:57.950830 (+ 1615us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 12:15:57.950838 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 12:15:57.951206 (+ 368us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 12:15:57.952030 (+ 824us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0809 12:15:57.952056 (+ 26us) client_negotiation.cc:656] calling sasl_client_start()
0809 12:15:57.954866 (+ 2810us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0809 12:15:57.959151 (+ 4285us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0809 12:15:57.959156 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0809 12:15:57.959161 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0809 12:15:57.959728 (+ 567us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0809 12:15:57.960167 (+ 439us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0809 12:15:57.960172 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0809 12:15:57.960176 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0809 12:15:57.960366 (+ 190us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0809 12:15:57.961234 (+ 868us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0809 12:15:57.961242 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0809 12:15:57.961463 (+ 221us) client_negotiation.cc:769] Sending connection context
0809 12:15:57.961610 (+ 147us) client_negotiation.cc:240] Negotiation successful
0809 12:15:57.961987 (+ 377us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":385}
I20260809 12:15:57.963845 25348 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 12:15:57.944422 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34550 (local address 127.0.222.60:45051)
0809 12:15:57.944618 (+ 196us) server_negotiation.cc:206] Beginning negotiation
0809 12:15:57.944624 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0809 12:15:57.944643 (+ 19us) server_negotiation.cc:407] Connection header received
0809 12:15:57.944700 (+ 57us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 12:15:57.944704 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 12:15:57.944777 (+ 73us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 12:15:57.944873 (+ 96us) server_negotiation.cc:226] Negotiated authn=SASL
0809 12:15:57.945985 (+ 1112us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 12:15:57.947022 (+ 1037us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 12:15:57.949434 (+ 2412us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 12:15:57.950380 (+ 946us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 12:15:57.950520 (+ 140us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 12:15:57.955133 (+ 4613us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 12:15:57.955139 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 12:15:57.955144 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0809 12:15:57.955187 (+ 43us) server_negotiation.cc:922] calling sasl_server_start()
0809 12:15:57.958912 (+ 3725us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 12:15:57.959874 (+ 962us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 12:15:57.959879 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 12:15:57.959882 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0809 12:15:57.960023 (+ 141us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 12:15:57.960505 (+ 482us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 12:15:57.960510 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 12:15:57.960512 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0809 12:15:57.960966 (+ 454us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 12:15:57.961159 (+ 193us) server_negotiation.cc:1035] Waiting for connection context
0809 12:15:57.963044 (+ 1885us) server_negotiation.cc:299] Negotiation successful
0809 12:15:57.963357 (+ 313us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":80}
I20260809 12:15:57.970270 25353 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 12:15:57.918428 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55626 (local address 127.0.222.62:43409)
0809 12:15:57.918675 (+ 247us) server_negotiation.cc:206] Beginning negotiation
0809 12:15:57.918682 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0809 12:15:57.928189 (+ 9507us) server_negotiation.cc:407] Connection header received
0809 12:15:57.928735 (+ 546us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 12:15:57.928744 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 12:15:57.928852 (+ 108us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 12:15:57.928954 (+ 102us) server_negotiation.cc:226] Negotiated authn=SASL
0809 12:15:57.932085 (+ 3131us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 12:15:57.933640 (+ 1555us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 12:15:57.935921 (+ 2281us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 12:15:57.936869 (+ 948us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 12:15:57.937035 (+ 166us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 12:15:57.944924 (+ 7889us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 12:15:57.944934 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 12:15:57.944942 (+ 8us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0809 12:15:57.945001 (+ 59us) server_negotiation.cc:922] calling sasl_server_start()
0809 12:15:57.949772 (+ 4771us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 12:15:57.951141 (+ 1369us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 12:15:57.951151 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 12:15:57.951157 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0809 12:15:57.951362 (+ 205us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 12:15:57.968189 (+ 16827us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 12:15:57.968198 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 12:15:57.968204 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0809 12:15:57.968706 (+ 502us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 12:15:57.969018 (+ 312us) server_negotiation.cc:1035] Waiting for connection context
0809 12:15:57.969706 (+ 688us) server_negotiation.cc:299] Negotiation successful
0809 12:15:57.969958 (+ 252us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":66}
I20260809 12:15:57.974555 25262 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P c5bbf4bc9dd744db8e4f7a5b4ae2ca09: Bootstrap starting.
I20260809 12:15:57.993147 25361 sys_catalog.cc:455] T 00000000000000000000000000000000 P 3a27d05012a94847b04e0675cd82d824 [sys.catalog]: SysCatalogTable state changed. Reason: New leader d1efb2909c73417fb178d06b54d1c562. Latest consensus state: current_term: 1 leader_uuid: "d1efb2909c73417fb178d06b54d1c562" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d1efb2909c73417fb178d06b54d1c562" member_type: VOTER last_known_addr { host: "127.0.222.62" port: 43409 } } peers { permanent_uuid: "c5bbf4bc9dd744db8e4f7a5b4ae2ca09" member_type: VOTER last_known_addr { host: "127.0.222.61" port: 35943 } } peers { permanent_uuid: "3a27d05012a94847b04e0675cd82d824" member_type: VOTER last_known_addr { host: "127.0.222.60" port: 45051 } } }
I20260809 12:15:57.993520 25361 sys_catalog.cc:458] T 00000000000000000000000000000000 P 3a27d05012a94847b04e0675cd82d824 [sys.catalog]: This master's current role is: FOLLOWER
I20260809 12:15:57.997056 25354 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 12:15:57.928386 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40286 (local address 127.0.222.61:35943)
0809 12:15:57.928605 (+ 219us) server_negotiation.cc:206] Beginning negotiation
0809 12:15:57.928611 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0809 12:15:57.935313 (+ 6702us) server_negotiation.cc:407] Connection header received
0809 12:15:57.942081 (+ 6768us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 12:15:57.942091 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 12:15:57.942209 (+ 118us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 12:15:57.942300 (+ 91us) server_negotiation.cc:226] Negotiated authn=SASL
0809 12:15:57.964606 (+ 22306us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 12:15:57.965869 (+ 1263us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 12:15:57.973852 (+ 7983us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 12:15:57.974785 (+ 933us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 12:15:57.974928 (+ 143us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 12:15:57.985544 (+ 10616us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 12:15:57.985555 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 12:15:57.985563 (+ 8us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0809 12:15:57.985630 (+ 67us) server_negotiation.cc:922] calling sasl_server_start()
0809 12:15:57.990425 (+ 4795us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 12:15:57.993752 (+ 3327us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 12:15:57.993761 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 12:15:57.993767 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0809 12:15:57.993954 (+ 187us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 12:15:57.995044 (+ 1090us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 12:15:57.995052 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 12:15:57.995058 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0809 12:15:57.995465 (+ 407us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 12:15:57.995721 (+ 256us) server_negotiation.cc:1035] Waiting for connection context
0809 12:15:57.996298 (+ 577us) server_negotiation.cc:299] Negotiation successful
0809 12:15:57.996543 (+ 245us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":62}
I20260809 12:15:58.001181 25262 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P c5bbf4bc9dd744db8e4f7a5b4ae2ca09: Neither blocks nor log segments found. Creating new log.
I20260809 12:15:58.002633 25262 log.cc:824] T 00000000000000000000000000000000 P c5bbf4bc9dd744db8e4f7a5b4ae2ca09: Log is configured to *not* fsync() on all Append() calls
I20260809 12:15:58.005462 25395 mvcc.cc:205] Tried to move back new op lower bound from 7316593696549244928 to 7316593696392335360. Current Snapshot: MvccSnapshot[applied={T|T < 7316593696549244928}]
I20260809 12:15:58.013094 25355 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 12:15:57.935962 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34548 (local address 127.0.222.60:45051)
0809 12:15:57.936144 (+ 182us) server_negotiation.cc:206] Beginning negotiation
0809 12:15:57.936149 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0809 12:15:57.971292 (+ 35143us) server_negotiation.cc:407] Connection header received
0809 12:15:57.971539 (+ 247us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 12:15:57.971544 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 12:15:57.971652 (+ 108us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 12:15:57.971723 (+ 71us) server_negotiation.cc:226] Negotiated authn=SASL
0809 12:15:57.979122 (+ 7399us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 12:15:57.980463 (+ 1341us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 12:15:57.983099 (+ 2636us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 12:15:57.984032 (+ 933us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 12:15:57.984174 (+ 142us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 12:15:57.996728 (+ 12554us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 12:15:57.996735 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 12:15:57.996740 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0809 12:15:57.996785 (+ 45us) server_negotiation.cc:922] calling sasl_server_start()
0809 12:15:58.002614 (+ 5829us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 12:15:58.005191 (+ 2577us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 12:15:58.005230 (+ 39us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 12:15:58.005234 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0809 12:15:58.005773 (+ 539us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 12:15:58.007114 (+ 1341us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 12:15:58.007167 (+ 53us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 12:15:58.007173 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0809 12:15:58.008386 (+ 1213us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 12:15:58.009140 (+ 754us) server_negotiation.cc:1035] Waiting for connection context
0809 12:15:58.011026 (+ 1886us) server_negotiation.cc:299] Negotiation successful
0809 12:15:58.011869 (+ 843us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":73}
I20260809 12:15:58.017284 25380 sys_catalog.cc:455] T 00000000000000000000000000000000 P d1efb2909c73417fb178d06b54d1c562 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "d1efb2909c73417fb178d06b54d1c562" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d1efb2909c73417fb178d06b54d1c562" member_type: VOTER last_known_addr { host: "127.0.222.62" port: 43409 } } peers { permanent_uuid: "c5bbf4bc9dd744db8e4f7a5b4ae2ca09" member_type: VOTER last_known_addr { host: "127.0.222.61" port: 35943 } } peers { permanent_uuid: "3a27d05012a94847b04e0675cd82d824" member_type: VOTER last_known_addr { host: "127.0.222.60" port: 45051 } } }
I20260809 12:15:58.017603 25380 sys_catalog.cc:458] T 00000000000000000000000000000000 P d1efb2909c73417fb178d06b54d1c562 [sys.catalog]: This master's current role is: LEADER
I20260809 12:15:58.023016 25361 sys_catalog.cc:455] T 00000000000000000000000000000000 P 3a27d05012a94847b04e0675cd82d824 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "d1efb2909c73417fb178d06b54d1c562" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d1efb2909c73417fb178d06b54d1c562" member_type: VOTER last_known_addr { host: "127.0.222.62" port: 43409 } } peers { permanent_uuid: "c5bbf4bc9dd744db8e4f7a5b4ae2ca09" member_type: VOTER last_known_addr { host: "127.0.222.61" port: 35943 } } peers { permanent_uuid: "3a27d05012a94847b04e0675cd82d824" member_type: VOTER last_known_addr { host: "127.0.222.60" port: 45051 } } }
I20260809 12:15:58.023418 25361 sys_catalog.cc:458] T 00000000000000000000000000000000 P 3a27d05012a94847b04e0675cd82d824 [sys.catalog]: This master's current role is: FOLLOWER
I20260809 12:15:58.034916 25262 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P c5bbf4bc9dd744db8e4f7a5b4ae2ca09: No bootstrap required, opened a new log
I20260809 12:15:58.041180 25387 catalog_manager.cc:1347] Generated new cluster ID: f94c80285bd84a2cbaf674d108ed0956
I20260809 12:15:58.041301 25387 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260809 12:15:58.046852 25262 raft_consensus.cc:348] T 00000000000000000000000000000000 P c5bbf4bc9dd744db8e4f7a5b4ae2ca09 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d1efb2909c73417fb178d06b54d1c562" member_type: VOTER last_known_addr { host: "127.0.222.62" port: 43409 } } peers { permanent_uuid: "c5bbf4bc9dd744db8e4f7a5b4ae2ca09" member_type: VOTER last_known_addr { host: "127.0.222.61" port: 35943 } } peers { permanent_uuid: "3a27d05012a94847b04e0675cd82d824" member_type: VOTER last_known_addr { host: "127.0.222.60" port: 45051 } }
I20260809 12:15:58.047274 25262 raft_consensus.cc:729] T 00000000000000000000000000000000 P c5bbf4bc9dd744db8e4f7a5b4ae2ca09 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c5bbf4bc9dd744db8e4f7a5b4ae2ca09, State: Initialized, Role: FOLLOWER
I20260809 12:15:58.047811 25262 consensus_queue.cc:260] T 00000000000000000000000000000000 P c5bbf4bc9dd744db8e4f7a5b4ae2ca09 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d1efb2909c73417fb178d06b54d1c562" member_type: VOTER last_known_addr { host: "127.0.222.62" port: 43409 } } peers { permanent_uuid: "c5bbf4bc9dd744db8e4f7a5b4ae2ca09" member_type: VOTER last_known_addr { host: "127.0.222.61" port: 35943 } } peers { permanent_uuid: "3a27d05012a94847b04e0675cd82d824" member_type: VOTER last_known_addr { host: "127.0.222.60" port: 45051 } }
W20260809 12:15:58.051184 25400 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 3a27d05012a94847b04e0675cd82d824: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260809 12:15:58.051386 25400 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260809 12:15:58.030851 25275 master_runner.cc:186] Error getting master registration for 127.0.222.61:35943: OK, instance_id { permanent_uuid: "c5bbf4bc9dd744db8e4f7a5b4ae2ca09" instance_seqno: 1786277757212031 } error { code: CATALOG_MANAGER_NOT_INITIALIZED status { code: SERVICE_UNAVAILABLE message: "Catalog manager is not initialized. State: Starting" } }
I20260809 12:15:58.053094 25275 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20260809 12:15:58.054044 25406 sys_catalog.cc:455] T 00000000000000000000000000000000 P c5bbf4bc9dd744db8e4f7a5b4ae2ca09 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d1efb2909c73417fb178d06b54d1c562" member_type: VOTER last_known_addr { host: "127.0.222.62" port: 43409 } } peers { permanent_uuid: "c5bbf4bc9dd744db8e4f7a5b4ae2ca09" member_type: VOTER last_known_addr { host: "127.0.222.61" port: 35943 } } peers { permanent_uuid: "3a27d05012a94847b04e0675cd82d824" member_type: VOTER last_known_addr { host: "127.0.222.60" port: 45051 } } }
I20260809 12:15:58.054693 25406 sys_catalog.cc:458] T 00000000000000000000000000000000 P c5bbf4bc9dd744db8e4f7a5b4ae2ca09 [sys.catalog]: This master's current role is: FOLLOWER
I20260809 12:15:58.056373 25262 sys_catalog.cc:564] T 00000000000000000000000000000000 P c5bbf4bc9dd744db8e4f7a5b4ae2ca09 [sys.catalog]: configured and running, proceeding with master startup.
I20260809 12:15:58.056767 25233 raft_consensus.cc:3037] T 00000000000000000000000000000000 P c5bbf4bc9dd744db8e4f7a5b4ae2ca09 [term 0 FOLLOWER]: Advancing to term 1
I20260809 12:15:58.056705 25234 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "d1efb2909c73417fb178d06b54d1c562" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: true dest_uuid: "c5bbf4bc9dd744db8e4f7a5b4ae2ca09"
I20260809 12:15:58.062610 25233 raft_consensus.cc:1261] T 00000000000000000000000000000000 P c5bbf4bc9dd744db8e4f7a5b4ae2ca09 [term 1 FOLLOWER]: Refusing update from remote peer d1efb2909c73417fb178d06b54d1c562: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260809 12:15:58.066323 25234 raft_consensus.cc:2454] T 00000000000000000000000000000000 P c5bbf4bc9dd744db8e4f7a5b4ae2ca09 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d1efb2909c73417fb178d06b54d1c562 in term 1.
I20260809 12:15:58.071272 25358 consensus_queue.cc:1035] T 00000000000000000000000000000000 P d1efb2909c73417fb178d06b54d1c562 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c5bbf4bc9dd744db8e4f7a5b4ae2ca09" member_type: VOTER last_known_addr { host: "127.0.222.61" port: 35943 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260809 12:15:58.103441 25387 catalog_manager.cc:1370] Generated new certificate authority record
I20260809 12:15:58.105880 25387 catalog_manager.cc:1504] Loading token signing keys...
I20260809 12:15:58.122606 25406 sys_catalog.cc:455] T 00000000000000000000000000000000 P c5bbf4bc9dd744db8e4f7a5b4ae2ca09 [sys.catalog]: SysCatalogTable state changed. Reason: New leader d1efb2909c73417fb178d06b54d1c562. Latest consensus state: current_term: 1 leader_uuid: "d1efb2909c73417fb178d06b54d1c562" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d1efb2909c73417fb178d06b54d1c562" member_type: VOTER last_known_addr { host: "127.0.222.62" port: 43409 } } peers { permanent_uuid: "c5bbf4bc9dd744db8e4f7a5b4ae2ca09" member_type: VOTER last_known_addr { host: "127.0.222.61" port: 35943 } } peers { permanent_uuid: "3a27d05012a94847b04e0675cd82d824" member_type: VOTER last_known_addr { host: "127.0.222.60" port: 45051 } } }
I20260809 12:15:58.123009 25406 sys_catalog.cc:458] T 00000000000000000000000000000000 P c5bbf4bc9dd744db8e4f7a5b4ae2ca09 [sys.catalog]: This master's current role is: FOLLOWER
I20260809 12:15:58.123594 25416 sys_catalog.cc:455] T 00000000000000000000000000000000 P c5bbf4bc9dd744db8e4f7a5b4ae2ca09 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "d1efb2909c73417fb178d06b54d1c562" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d1efb2909c73417fb178d06b54d1c562" member_type: VOTER last_known_addr { host: "127.0.222.62" port: 43409 } } peers { permanent_uuid: "c5bbf4bc9dd744db8e4f7a5b4ae2ca09" member_type: VOTER last_known_addr { host: "127.0.222.61" port: 35943 } } peers { permanent_uuid: "3a27d05012a94847b04e0675cd82d824" member_type: VOTER last_known_addr { host: "127.0.222.60" port: 45051 } } }
I20260809 12:15:58.124074 25416 sys_catalog.cc:458] T 00000000000000000000000000000000 P c5bbf4bc9dd744db8e4f7a5b4ae2ca09 [sys.catalog]: This master's current role is: FOLLOWER
I20260809 12:15:58.125430 25380 sys_catalog.cc:455] T 00000000000000000000000000000000 P d1efb2909c73417fb178d06b54d1c562 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "d1efb2909c73417fb178d06b54d1c562" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d1efb2909c73417fb178d06b54d1c562" member_type: VOTER last_known_addr { host: "127.0.222.62" port: 43409 } } peers { permanent_uuid: "c5bbf4bc9dd744db8e4f7a5b4ae2ca09" member_type: VOTER last_known_addr { host: "127.0.222.61" port: 35943 } } peers { permanent_uuid: "3a27d05012a94847b04e0675cd82d824" member_type: VOTER last_known_addr { host: "127.0.222.60" port: 45051 } } }
I20260809 12:15:58.125701 25380 sys_catalog.cc:458] T 00000000000000000000000000000000 P d1efb2909c73417fb178d06b54d1c562 [sys.catalog]: This master's current role is: LEADER
I20260809 12:15:58.127122 25387 catalog_manager.cc:5868] T 00000000000000000000000000000000 P d1efb2909c73417fb178d06b54d1c562: Generated new TSK 0
I20260809 12:15:58.128481 25387 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260809 12:15:58.178741 25422 catalog_manager.cc:1259] Loaded cluster ID: f94c80285bd84a2cbaf674d108ed0956
I20260809 12:15:58.178876 25422 catalog_manager.cc:1552] T 00000000000000000000000000000000 P c5bbf4bc9dd744db8e4f7a5b4ae2ca09: loading cluster ID for follower catalog manager: success
I20260809 12:15:58.182703 25422 catalog_manager.cc:1574] T 00000000000000000000000000000000 P c5bbf4bc9dd744db8e4f7a5b4ae2ca09: acquiring CA information for follower catalog manager: success
I20260809 12:15:58.185551 25422 catalog_manager.cc:1602] T 00000000000000000000000000000000 P c5bbf4bc9dd744db8e4f7a5b4ae2ca09: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
W20260809 12:15:58.252868 888 mini_kdc.cc:348] Time spent kdestroy: real 0.113s user 0.000s sys 0.031s
W20260809 12:15:58.304380 25353 negotiation.cc:337] Failed RPC negotiation. Trace:
0809 12:15:58.287842 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55640 (local address 127.0.222.62:43409)
0809 12:15:58.288045 (+ 203us) server_negotiation.cc:206] Beginning negotiation
0809 12:15:58.288053 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0809 12:15:58.300816 (+ 12763us) server_negotiation.cc:407] Connection header received
0809 12:15:58.302522 (+ 1706us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 12:15:58.302530 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 12:15:58.302634 (+ 104us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 12:15:58.302712 (+ 78us) server_negotiation.cc:226] Negotiated authn=SASL
0809 12:15:58.304071 (+ 1359us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:55640: BlockingRecv error: recv got EOF from 127.0.0.1:55640 (error 108)
Metrics: {"server-negotiator.queue_time_us":73}
W20260809 12:15:58.308454 25354 negotiation.cc:337] Failed RPC negotiation. Trace:
0809 12:15:58.301561 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40288 (local address 127.0.222.61:35943)
0809 12:15:58.301760 (+ 199us) server_negotiation.cc:206] Beginning negotiation
0809 12:15:58.301766 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0809 12:15:58.306154 (+ 4388us) server_negotiation.cc:407] Connection header received
0809 12:15:58.306894 (+ 740us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 12:15:58.306900 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 12:15:58.307045 (+ 145us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 12:15:58.307133 (+ 88us) server_negotiation.cc:226] Negotiated authn=SASL
0809 12:15:58.308184 (+ 1051us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:40288: BlockingRecv error: recv got EOF from 127.0.0.1:40288 (error 108)
Metrics: {"server-negotiator.queue_time_us":72}
W20260809 12:15:58.314409 25355 negotiation.cc:337] Failed RPC negotiation. Trace:
0809 12:15:58.311528 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34554 (local address 127.0.222.60:45051)
0809 12:15:58.311715 (+ 187us) server_negotiation.cc:206] Beginning negotiation
0809 12:15:58.311721 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0809 12:15:58.312460 (+ 739us) server_negotiation.cc:407] Connection header received
0809 12:15:58.312726 (+ 266us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 12:15:58.312733 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 12:15:58.312813 (+ 80us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 12:15:58.312884 (+ 71us) server_negotiation.cc:226] Negotiated authn=SASL
0809 12:15:58.314119 (+ 1235us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:34554: BlockingRecv error: recv got EOF from 127.0.0.1:34554 (error 108)
Metrics: {"server-negotiator.queue_time_us":72}
I20260809 12:15:58.369527 25354 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 12:15:58.349692 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40300 (local address 127.0.222.61:35943)
0809 12:15:58.349881 (+ 189us) server_negotiation.cc:206] Beginning negotiation
0809 12:15:58.349887 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0809 12:15:58.361052 (+ 11165us) server_negotiation.cc:407] Connection header received
0809 12:15:58.361632 (+ 580us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 12:15:58.361639 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 12:15:58.361895 (+ 256us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 12:15:58.362055 (+ 160us) server_negotiation.cc:226] Negotiated authn=TOKEN
0809 12:15:58.362855 (+ 800us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 12:15:58.364082 (+ 1227us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 12:15:58.366177 (+ 2095us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 12:15:58.366911 (+ 734us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 12:15:58.367059 (+ 148us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 12:15:58.367863 (+ 804us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0809 12:15:58.368656 (+ 793us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0809 12:15:58.368787 (+ 131us) server_negotiation.cc:1035] Waiting for connection context
0809 12:15:58.369046 (+ 259us) server_negotiation.cc:299] Negotiation successful
0809 12:15:58.369140 (+ 94us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":65}
W20260809 12:15:58.426415 25355 negotiation.cc:337] Failed RPC negotiation. Trace:
0809 12:15:58.413070 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35974 (local address 127.0.222.60:45051)
0809 12:15:58.413274 (+ 204us) server_negotiation.cc:206] Beginning negotiation
0809 12:15:58.413281 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0809 12:15:58.423897 (+ 10616us) server_negotiation.cc:407] Connection header received
0809 12:15:58.424569 (+ 672us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 12:15:58.424576 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 12:15:58.424717 (+ 141us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 12:15:58.424803 (+ 86us) server_negotiation.cc:226] Negotiated authn=SASL
0809 12:15:58.426144 (+ 1341us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:35974: BlockingRecv error: recv got EOF from 127.0.0.1:35974 (error 108)
Metrics: {"server-negotiator.queue_time_us":69}
I20260809 12:15:58.481128 25354 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 12:15:58.462408 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56446 (local address 127.0.222.61:35943)
0809 12:15:58.462619 (+ 211us) server_negotiation.cc:206] Beginning negotiation
0809 12:15:58.462625 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0809 12:15:58.473284 (+ 10659us) server_negotiation.cc:407] Connection header received
0809 12:15:58.473857 (+ 573us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 12:15:58.473864 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 12:15:58.473967 (+ 103us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 12:15:58.474038 (+ 71us) server_negotiation.cc:226] Negotiated authn=TOKEN
0809 12:15:58.474732 (+ 694us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 12:15:58.475979 (+ 1247us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 12:15:58.478140 (+ 2161us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 12:15:58.478897 (+ 757us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 12:15:58.479034 (+ 137us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 12:15:58.479736 (+ 702us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0809 12:15:58.479898 (+ 162us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0809 12:15:58.479967 (+ 69us) server_negotiation.cc:1035] Waiting for connection context
0809 12:15:58.480703 (+ 736us) server_negotiation.cc:299] Negotiation successful
0809 12:15:58.480795 (+ 92us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":78}
W20260809 12:15:58.531949 25355 negotiation.cc:337] Failed RPC negotiation. Trace:
0809 12:15:58.518224 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35986 (local address 127.0.222.60:45051)
0809 12:15:58.518432 (+ 208us) server_negotiation.cc:206] Beginning negotiation
0809 12:15:58.518439 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0809 12:15:58.529696 (+ 11257us) server_negotiation.cc:407] Connection header received
0809 12:15:58.530291 (+ 595us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 12:15:58.530299 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 12:15:58.530434 (+ 135us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 12:15:58.530530 (+ 96us) server_negotiation.cc:226] Negotiated authn=SASL
0809 12:15:58.531701 (+ 1171us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:35986: BlockingRecv error: recv got EOF from 127.0.0.1:35986 (error 108)
Metrics: {"server-negotiator.queue_time_us":83}
I20260809 12:15:58.584306 25354 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 12:15:58.566687 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56460 (local address 127.0.222.61:35943)
0809 12:15:58.566952 (+ 265us) server_negotiation.cc:206] Beginning negotiation
0809 12:15:58.566988 (+ 36us) server_negotiation.cc:399] Waiting for connection header
0809 12:15:58.575853 (+ 8865us) server_negotiation.cc:407] Connection header received
0809 12:15:58.576502 (+ 649us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 12:15:58.576510 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 12:15:58.576619 (+ 109us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 12:15:58.576699 (+ 80us) server_negotiation.cc:226] Negotiated authn=TOKEN
0809 12:15:58.577499 (+ 800us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 12:15:58.578813 (+ 1314us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 12:15:58.581318 (+ 2505us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 12:15:58.582171 (+ 853us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 12:15:58.582311 (+ 140us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 12:15:58.583224 (+ 913us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0809 12:15:58.583426 (+ 202us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0809 12:15:58.583513 (+ 87us) server_negotiation.cc:1035] Waiting for connection context
0809 12:15:58.583822 (+ 309us) server_negotiation.cc:299] Negotiation successful
0809 12:15:58.583942 (+ 120us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":92}
W20260809 12:15:58.629160 25355 negotiation.cc:337] Failed RPC negotiation. Trace:
0809 12:15:58.620319 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35988 (local address 127.0.222.60:45051)
0809 12:15:58.620531 (+ 212us) server_negotiation.cc:206] Beginning negotiation
0809 12:15:58.620539 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0809 12:15:58.627166 (+ 6627us) server_negotiation.cc:407] Connection header received
0809 12:15:58.627568 (+ 402us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 12:15:58.627574 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 12:15:58.627685 (+ 111us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 12:15:58.627770 (+ 85us) server_negotiation.cc:226] Negotiated authn=SASL
0809 12:15:58.628893 (+ 1123us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:35988: BlockingRecv error: recv got EOF from 127.0.0.1:35988 (error 108)
Metrics: {"server-negotiator.queue_time_us":81}
I20260809 12:15:58.648983 25354 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 12:15:58.641361 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56470 (local address 127.0.222.61:35943)
0809 12:15:58.641602 (+ 241us) server_negotiation.cc:206] Beginning negotiation
0809 12:15:58.641612 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0809 12:15:58.642164 (+ 552us) server_negotiation.cc:407] Connection header received
0809 12:15:58.642413 (+ 249us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 12:15:58.642420 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 12:15:58.642529 (+ 109us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 12:15:58.642652 (+ 123us) server_negotiation.cc:226] Negotiated authn=TOKEN
0809 12:15:58.643197 (+ 545us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 12:15:58.644543 (+ 1346us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 12:15:58.646549 (+ 2006us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 12:15:58.647397 (+ 848us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 12:15:58.647565 (+ 168us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 12:15:58.648079 (+ 514us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0809 12:15:58.648226 (+ 147us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0809 12:15:58.648331 (+ 105us) server_negotiation.cc:1035] Waiting for connection context
0809 12:15:58.648535 (+ 204us) server_negotiation.cc:299] Negotiation successful
0809 12:15:58.648628 (+ 93us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":98}
W20260809 12:15:58.659528 25355 negotiation.cc:337] Failed RPC negotiation. Trace:
0809 12:15:58.657321 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36004 (local address 127.0.222.60:45051)
0809 12:15:58.657581 (+ 260us) server_negotiation.cc:206] Beginning negotiation
0809 12:15:58.657592 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0809 12:15:58.657761 (+ 169us) server_negotiation.cc:407] Connection header received
0809 12:15:58.657967 (+ 206us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 12:15:58.657974 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 12:15:58.658080 (+ 106us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 12:15:58.658224 (+ 144us) server_negotiation.cc:226] Negotiated authn=SASL
0809 12:15:58.659209 (+ 985us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:36004: BlockingRecv error: recv got EOF from 127.0.0.1:36004 (error 108)
Metrics: {"server-negotiator.queue_time_us":113}
I20260809 12:15:58.680176 25354 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 12:15:58.671897 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56478 (local address 127.0.222.61:35943)
0809 12:15:58.672110 (+ 213us) server_negotiation.cc:206] Beginning negotiation
0809 12:15:58.672118 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0809 12:15:58.672733 (+ 615us) server_negotiation.cc:407] Connection header received
0809 12:15:58.672986 (+ 253us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 12:15:58.672994 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 12:15:58.673132 (+ 138us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 12:15:58.673247 (+ 115us) server_negotiation.cc:226] Negotiated authn=TOKEN
0809 12:15:58.673836 (+ 589us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 12:15:58.675182 (+ 1346us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 12:15:58.677571 (+ 2389us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 12:15:58.678352 (+ 781us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 12:15:58.678508 (+ 156us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 12:15:58.679227 (+ 719us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0809 12:15:58.679395 (+ 168us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0809 12:15:58.679507 (+ 112us) server_negotiation.cc:1035] Waiting for connection context
0809 12:15:58.679713 (+ 206us) server_negotiation.cc:299] Negotiation successful
0809 12:15:58.679815 (+ 102us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":86}
W20260809 12:15:58.705683 25355 negotiation.cc:337] Failed RPC negotiation. Trace:
0809 12:15:58.688305 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36006 (local address 127.0.222.60:45051)
0809 12:15:58.688536 (+ 231us) server_negotiation.cc:206] Beginning negotiation
0809 12:15:58.688545 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0809 12:15:58.703742 (+ 15197us) server_negotiation.cc:407] Connection header received
0809 12:15:58.704042 (+ 300us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 12:15:58.704051 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 12:15:58.704158 (+ 107us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 12:15:58.704270 (+ 112us) server_negotiation.cc:226] Negotiated authn=SASL
0809 12:15:58.705396 (+ 1126us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:36006: BlockingRecv error: recv got EOF from 127.0.0.1:36006 (error 108)
Metrics: {"server-negotiator.queue_time_us":91}
I20260809 12:15:58.738983 25354 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 12:15:58.726565 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56488 (local address 127.0.222.61:35943)
0809 12:15:58.726768 (+ 203us) server_negotiation.cc:206] Beginning negotiation
0809 12:15:58.726775 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0809 12:15:58.731995 (+ 5220us) server_negotiation.cc:407] Connection header received
0809 12:15:58.732230 (+ 235us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 12:15:58.732236 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 12:15:58.732350 (+ 114us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 12:15:58.732460 (+ 110us) server_negotiation.cc:226] Negotiated authn=TOKEN
0809 12:15:58.733156 (+ 696us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 12:15:58.734323 (+ 1167us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 12:15:58.736636 (+ 2313us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 12:15:58.737344 (+ 708us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 12:15:58.737478 (+ 134us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 12:15:58.738049 (+ 571us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0809 12:15:58.738207 (+ 158us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0809 12:15:58.738315 (+ 108us) server_negotiation.cc:1035] Waiting for connection context
0809 12:15:58.738526 (+ 211us) server_negotiation.cc:299] Negotiation successful
0809 12:15:58.738622 (+ 96us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":65}
W20260809 12:15:58.748749 25355 negotiation.cc:337] Failed RPC negotiation. Trace:
0809 12:15:58.746093 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36022 (local address 127.0.222.60:45051)
0809 12:15:58.746312 (+ 219us) server_negotiation.cc:206] Beginning negotiation
0809 12:15:58.746318 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0809 12:15:58.747001 (+ 683us) server_negotiation.cc:407] Connection header received
0809 12:15:58.747258 (+ 257us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 12:15:58.747269 (+ 11us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 12:15:58.747404 (+ 135us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 12:15:58.747534 (+ 130us) server_negotiation.cc:226] Negotiated authn=SASL
0809 12:15:58.748447 (+ 913us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:36022: BlockingRecv error: recv got EOF from 127.0.0.1:36022 (error 108)
Metrics: {"server-negotiator.queue_time_us":71}
I20260809 12:15:58.791280 25354 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 12:15:58.784971 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56502 (local address 127.0.222.61:35943)
0809 12:15:58.785205 (+ 234us) server_negotiation.cc:206] Beginning negotiation
0809 12:15:58.785213 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0809 12:15:58.785655 (+ 442us) server_negotiation.cc:407] Connection header received
0809 12:15:58.785925 (+ 270us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 12:15:58.785930 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 12:15:58.786026 (+ 96us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 12:15:58.786131 (+ 105us) server_negotiation.cc:226] Negotiated authn=TOKEN
0809 12:15:58.786591 (+ 460us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 12:15:58.787790 (+ 1199us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 12:15:58.789295 (+ 1505us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 12:15:58.789934 (+ 639us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 12:15:58.790073 (+ 139us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 12:15:58.790461 (+ 388us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0809 12:15:58.790612 (+ 151us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0809 12:15:58.790706 (+ 94us) server_negotiation.cc:1035] Waiting for connection context
0809 12:15:58.790853 (+ 147us) server_negotiation.cc:299] Negotiation successful
0809 12:15:58.790935 (+ 82us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":62}
W20260809 12:15:58.799820 25355 negotiation.cc:337] Failed RPC negotiation. Trace:
0809 12:15:58.797616 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36024 (local address 127.0.222.60:45051)
0809 12:15:58.797791 (+ 175us) server_negotiation.cc:206] Beginning negotiation
0809 12:15:58.797795 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0809 12:15:58.798284 (+ 489us) server_negotiation.cc:407] Connection header received
0809 12:15:58.798494 (+ 210us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 12:15:58.798498 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 12:15:58.798570 (+ 72us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 12:15:58.798674 (+ 104us) server_negotiation.cc:226] Negotiated authn=SASL
0809 12:15:58.799584 (+ 910us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:36024: BlockingRecv error: recv got EOF from 127.0.0.1:36024 (error 108)
Metrics: {"server-negotiator.queue_time_us":55}
I20260809 12:15:58.873008 25354 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 12:15:58.867808 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56510 (local address 127.0.222.61:35943)
0809 12:15:58.868095 (+ 287us) server_negotiation.cc:206] Beginning negotiation
0809 12:15:58.868104 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0809 12:15:58.868377 (+ 273us) server_negotiation.cc:407] Connection header received
0809 12:15:58.868621 (+ 244us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 12:15:58.868626 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 12:15:58.868695 (+ 69us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 12:15:58.868799 (+ 104us) server_negotiation.cc:226] Negotiated authn=TOKEN
0809 12:15:58.869207 (+ 408us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 12:15:58.869951 (+ 744us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 12:15:58.871312 (+ 1361us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 12:15:58.871744 (+ 432us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 12:15:58.871884 (+ 140us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 12:15:58.872279 (+ 395us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0809 12:15:58.872392 (+ 113us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0809 12:15:58.872491 (+ 99us) server_negotiation.cc:1035] Waiting for connection context
0809 12:15:58.872638 (+ 147us) server_negotiation.cc:299] Negotiation successful
0809 12:15:58.872719 (+ 81us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":85}
W20260809 12:15:58.880509 25355 negotiation.cc:337] Failed RPC negotiation. Trace:
0809 12:15:58.878687 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36036 (local address 127.0.222.60:45051)
0809 12:15:58.878916 (+ 229us) server_negotiation.cc:206] Beginning negotiation
0809 12:15:58.878923 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0809 12:15:58.879240 (+ 317us) server_negotiation.cc:407] Connection header received
0809 12:15:58.879361 (+ 121us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 12:15:58.879364 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 12:15:58.879447 (+ 83us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 12:15:58.879533 (+ 86us) server_negotiation.cc:226] Negotiated authn=SASL
0809 12:15:58.880281 (+ 748us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:36036: BlockingRecv error: recv got EOF from 127.0.0.1:36036 (error 108)
Metrics: {"server-negotiator.queue_time_us":68}
I20260809 12:15:59.017755 25354 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 12:15:59.012401 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56524 (local address 127.0.222.61:35943)
0809 12:15:59.012662 (+ 261us) server_negotiation.cc:206] Beginning negotiation
0809 12:15:59.012669 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0809 12:15:59.012893 (+ 224us) server_negotiation.cc:407] Connection header received
0809 12:15:59.013162 (+ 269us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 12:15:59.013166 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 12:15:59.013238 (+ 72us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 12:15:59.013345 (+ 107us) server_negotiation.cc:226] Negotiated authn=TOKEN
0809 12:15:59.013810 (+ 465us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 12:15:59.014540 (+ 730us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 12:15:59.015999 (+ 1459us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 12:15:59.016439 (+ 440us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 12:15:59.016611 (+ 172us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 12:15:59.017015 (+ 404us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0809 12:15:59.017123 (+ 108us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0809 12:15:59.017228 (+ 105us) server_negotiation.cc:1035] Waiting for connection context
0809 12:15:59.017385 (+ 157us) server_negotiation.cc:299] Negotiation successful
0809 12:15:59.017469 (+ 84us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":98}
W20260809 12:15:59.026398 25355 negotiation.cc:337] Failed RPC negotiation. Trace:
0809 12:15:59.024253 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36038 (local address 127.0.222.60:45051)
0809 12:15:59.024514 (+ 261us) server_negotiation.cc:206] Beginning negotiation
0809 12:15:59.024521 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0809 12:15:59.024890 (+ 369us) server_negotiation.cc:407] Connection header received
0809 12:15:59.025106 (+ 216us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 12:15:59.025111 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 12:15:59.025187 (+ 76us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 12:15:59.025303 (+ 116us) server_negotiation.cc:226] Negotiated authn=SASL
0809 12:15:59.026117 (+ 814us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:36038: BlockingRecv error: recv got EOF from 127.0.0.1:36038 (error 108)
Metrics: {"server-negotiator.queue_time_us":75}
I20260809 12:15:59.053269 25400 catalog_manager.cc:1259] Loaded cluster ID: f94c80285bd84a2cbaf674d108ed0956
I20260809 12:15:59.053419 25400 catalog_manager.cc:1552] T 00000000000000000000000000000000 P 3a27d05012a94847b04e0675cd82d824: loading cluster ID for follower catalog manager: success
I20260809 12:15:59.056835 25400 catalog_manager.cc:1574] T 00000000000000000000000000000000 P 3a27d05012a94847b04e0675cd82d824: acquiring CA information for follower catalog manager: success
I20260809 12:15:59.058871 25400 catalog_manager.cc:1602] T 00000000000000000000000000000000 P 3a27d05012a94847b04e0675cd82d824: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260809 12:15:59.292475 25354 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 12:15:59.287449 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56538 (local address 127.0.222.61:35943)
0809 12:15:59.287644 (+ 195us) server_negotiation.cc:206] Beginning negotiation
0809 12:15:59.287648 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0809 12:15:59.287972 (+ 324us) server_negotiation.cc:407] Connection header received
0809 12:15:59.288144 (+ 172us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 12:15:59.288149 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 12:15:59.288241 (+ 92us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 12:15:59.288337 (+ 96us) server_negotiation.cc:226] Negotiated authn=TOKEN
0809 12:15:59.288732 (+ 395us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 12:15:59.289529 (+ 797us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 12:15:59.290851 (+ 1322us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 12:15:59.291323 (+ 472us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 12:15:59.291415 (+ 92us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 12:15:59.291789 (+ 374us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0809 12:15:59.291891 (+ 102us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0809 12:15:59.291958 (+ 67us) server_negotiation.cc:1035] Waiting for connection context
0809 12:15:59.292097 (+ 139us) server_negotiation.cc:299] Negotiation successful
0809 12:15:59.292161 (+ 64us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":62}
I20260809 12:15:59.303891 25355 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 12:15:59.298477 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36046 (local address 127.0.222.60:45051)
0809 12:15:59.298645 (+ 168us) server_negotiation.cc:206] Beginning negotiation
0809 12:15:59.298649 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0809 12:15:59.298920 (+ 271us) server_negotiation.cc:407] Connection header received
0809 12:15:59.299116 (+ 196us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 12:15:59.299120 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 12:15:59.299273 (+ 153us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 12:15:59.299378 (+ 105us) server_negotiation.cc:226] Negotiated authn=TOKEN
0809 12:15:59.299748 (+ 370us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 12:15:59.300582 (+ 834us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 12:15:59.301790 (+ 1208us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 12:15:59.302206 (+ 416us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 12:15:59.302304 (+ 98us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 12:15:59.302754 (+ 450us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0809 12:15:59.303342 (+ 588us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0809 12:15:59.303453 (+ 111us) server_negotiation.cc:1035] Waiting for connection context
0809 12:15:59.303566 (+ 113us) server_negotiation.cc:299] Negotiation successful
0809 12:15:59.303639 (+ 73us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":56}
I20260809 12:15:59.317760 25554 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 12:15:59.310360 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36726 (local address 127.0.222.62:43409)
0809 12:15:59.311339 (+ 979us) server_negotiation.cc:206] Beginning negotiation
0809 12:15:59.311352 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0809 12:15:59.311368 (+ 16us) server_negotiation.cc:407] Connection header received
0809 12:15:59.311470 (+ 102us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 12:15:59.311474 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 12:15:59.311652 (+ 178us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 12:15:59.312151 (+ 499us) server_negotiation.cc:226] Negotiated authn=TOKEN
0809 12:15:59.312558 (+ 407us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 12:15:59.313701 (+ 1143us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 12:15:59.315475 (+ 1774us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 12:15:59.315913 (+ 438us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 12:15:59.316058 (+ 145us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 12:15:59.316602 (+ 544us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0809 12:15:59.317069 (+ 467us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0809 12:15:59.317185 (+ 116us) server_negotiation.cc:1035] Waiting for connection context
0809 12:15:59.317342 (+ 157us) server_negotiation.cc:299] Negotiation successful
0809 12:15:59.317435 (+ 93us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":856,"thread_start_us":752,"threads_started":1}
I20260809 12:15:59.318883 25354 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 12:15:59.311112 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56552 (local address 127.0.222.61:35943)
0809 12:15:59.311292 (+ 180us) server_negotiation.cc:206] Beginning negotiation
0809 12:15:59.311298 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0809 12:15:59.311779 (+ 481us) server_negotiation.cc:407] Connection header received
0809 12:15:59.312745 (+ 966us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 12:15:59.312751 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 12:15:59.312851 (+ 100us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 12:15:59.312918 (+ 67us) server_negotiation.cc:226] Negotiated authn=TOKEN
0809 12:15:59.313780 (+ 862us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 12:15:59.314634 (+ 854us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 12:15:59.316830 (+ 2196us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 12:15:59.317383 (+ 553us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 12:15:59.317498 (+ 115us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 12:15:59.318148 (+ 650us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0809 12:15:59.318247 (+ 99us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0809 12:15:59.318355 (+ 108us) server_negotiation.cc:1035] Waiting for connection context
0809 12:15:59.318526 (+ 171us) server_negotiation.cc:299] Negotiation successful
0809 12:15:59.318608 (+ 82us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":72}
I20260809 12:15:59.319216 25355 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 12:15:59.311843 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36060 (local address 127.0.222.60:45051)
0809 12:15:59.312058 (+ 215us) server_negotiation.cc:206] Beginning negotiation
0809 12:15:59.312065 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0809 12:15:59.312475 (+ 410us) server_negotiation.cc:407] Connection header received
0809 12:15:59.312674 (+ 199us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 12:15:59.312679 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 12:15:59.312739 (+ 60us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 12:15:59.312836 (+ 97us) server_negotiation.cc:226] Negotiated authn=TOKEN
0809 12:15:59.313369 (+ 533us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 12:15:59.314308 (+ 939us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 12:15:59.316637 (+ 2329us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 12:15:59.317252 (+ 615us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 12:15:59.317387 (+ 135us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 12:15:59.318408 (+ 1021us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0809 12:15:59.318538 (+ 130us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0809 12:15:59.318619 (+ 81us) server_negotiation.cc:1035] Waiting for connection context
0809 12:15:59.318847 (+ 228us) server_negotiation.cc:299] Negotiation successful
0809 12:15:59.318909 (+ 62us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":51}
I20260809 12:15:59.324026 888 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskgH5Pjm/build/asan/bin/kudu pid 25106
I20260809 12:15:59.325093 25178 generic_service.cc:196] Checking for leaks (request via RPC)
I20260809 12:15:59.437819 888 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskgH5Pjm/build/asan/bin/kudu with pid 25106
I20260809 12:15:59.464597 888 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskgH5Pjm/build/asan/bin/kudu pid 25192
I20260809 12:15:59.465837 25257 generic_service.cc:196] Checking for leaks (request via RPC)
I20260809 12:15:59.577175 888 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskgH5Pjm/build/asan/bin/kudu with pid 25192
I20260809 12:15:59.601509 888 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskgH5Pjm/build/asan/bin/kudu pid 25275
I20260809 12:15:59.602674 25340 generic_service.cc:196] Checking for leaks (request via RPC)
I20260809 12:15:59.711658 888 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskgH5Pjm/build/asan/bin/kudu with pid 25275
2026-08-09T12:15:59Z chronyd exiting
[ OK ] ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys/0 (5025 ms)
[ RUN ] ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys/1
Loading random data
Initializing database '/tmp/dist-test-taskgH5Pjm/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786277382073938-888-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 09 12:15:59 dist-test-slave-2w3w krb5kdc[25563](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 09 12:15:59 dist-test-slave-2w3w krb5kdc[25563](info): set up 2 sockets
Aug 09 12:15:59 dist-test-slave-2w3w krb5kdc[25563](info): commencing operation
krb5kdc: starting...
W20260809 12:16:00.892571 888 mini_kdc.cc:121] Time spent starting KDC: real 1.134s user 0.001s sys 0.049s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskgH5Pjm/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786277382073938-888-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskgH5Pjm/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786277382073938-888-0/krb5kdc/test-user.keytab.
Aug 09 12:16:01 dist-test-slave-2w3w krb5kdc[25563](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786277761, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-09T12:16:01Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-09T12:16:01Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.222.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.222.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.222.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskgH5Pjm/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786277382073938-888-0/krb5kdc/kudu_127.0.222.62.keytab.
Entry for principal kudu/127.0.222.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskgH5Pjm/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786277382073938-888-0/krb5kdc/kudu_127.0.222.62.keytab.
I20260809 12:16:01.248383 888 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskgH5Pjm/build/asan/bin/kudu
/tmp/dist-test-taskgH5Pjm/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskgH5Pjm/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786277382073938-888-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskgH5Pjm/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786277382073938-888-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskgH5Pjm/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786277382073938-888-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskgH5Pjm/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786277382073938-888-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.222.62:39441
--webserver_interface=127.0.222.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskgH5Pjm/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786277382073938-888-0/krb5kdc/kudu_127.0.222.62.keytab
--principal=kudu/127.0.222.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.222.20:32943
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.222.62:39441,127.0.222.61:42263,127.0.222.60:37589
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--enable_leader_failure_detection=false
--rpc_authentication=required
--rpc_encryption=required with env {KRB5CCNAME=/tmp/dist-test-taskgH5Pjm/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786277382073938-888-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskgH5Pjm/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786277382073938-888-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskgH5Pjm/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786277382073938-888-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260809 12:16:01.598503 25577 flags.cc:425] Enabled unsafe flag: --enable_leader_failure_detection=false
W20260809 12:16:01.598836 25577 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260809 12:16:01.598903 25577 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260809 12:16:01.599061 25577 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260809 12:16:01.606434 25577 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260809 12:16:01.606524 25577 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260809 12:16:01.606582 25577 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260809 12:16:01.606634 25577 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260809 12:16:01.606689 25577 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260809 12:16:01.616699 25577 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.222.20:32943
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_leader_failure_detection=false
--fs_data_dirs=/tmp/dist-test-taskgH5Pjm/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786277382073938-888-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskgH5Pjm/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786277382073938-888-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.222.62:39441,127.0.222.61:42263,127.0.222.60:37589
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskgH5Pjm/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786277382073938-888-0/krb5kdc/kudu_127.0.222.62.keytab
--principal=kudu/127.0.222.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.222.62:39441
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskgH5Pjm/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786277382073938-888-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.222.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskgH5Pjm/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786277382073938-888-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 09 Aug 2026 11:06:41 UTC on ccycloud.quasar-bwoogy.root.comops.site
build id 4575991
ASAN enabled
I20260809 12:16:01.617967 25577 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260809 12:16:01.619264 25577 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260809 12:16:01.631064 25584 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 12:16:01.631038 25587 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 12:16:01.631178 25585 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260809 12:16:01.632742 25577 server_base.cc:1034] running on GCE node
I20260809 12:16:01.633533 25577 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260809 12:16:01.634944 25577 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260809 12:16:01.636231 25577 hybrid_clock.cc:648] HybridClock initialized: now 1786277761636151 us; error 48 us; skew 500 ppm
I20260809 12:16:01.636587 25577 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 09 12:16:01 dist-test-slave-2w3w krb5kdc[25563](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786277761, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.222.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260809 12:16:01.639824 25577 init.cc:377] Logged in from keytab as kudu/127.0.222.62@KRBTEST.COM (short username kudu)
I20260809 12:16:01.641234 25577 webserver.cc:460] Webserver started at http://127.0.222.62:34893/ using document root <none> and password file <none>
I20260809 12:16:01.641872 25577 fs_manager.cc:359] Metadata directory not provided
I20260809 12:16:01.641958 25577 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260809 12:16:01.642225 25577 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260809 12:16:01.645146 25577 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskgH5Pjm/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786277382073938-888-0/minicluster-data/master-0/data/instance:
uuid: "d343a8cfeb614de282f065a5966ca6cd"
format_stamp: "Formatted at 2026-08-09 12:16:01 on dist-test-slave-2w3w"
server_key: "2126134e8b367ec225682320dceaa6b9"
server_key_iv: "a68621d35517bb9d53b0ec66153a9840"
server_key_version: "encryptionkey@0"
I20260809 12:16:01.645886 25577 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskgH5Pjm/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786277382073938-888-0/minicluster-data/master-0/wal/instance:
uuid: "d343a8cfeb614de282f065a5966ca6cd"
format_stamp: "Formatted at 2026-08-09 12:16:01 on dist-test-slave-2w3w"
server_key: "2126134e8b367ec225682320dceaa6b9"
server_key_iv: "a68621d35517bb9d53b0ec66153a9840"
server_key_version: "encryptionkey@0"
I20260809 12:16:01.651095 25577 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.003s sys 0.005s
I20260809 12:16:01.654426 25594 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260809 12:16:01.655776 25577 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260809 12:16:01.655954 25577 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskgH5Pjm/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786277382073938-888-0/minicluster-data/master-0/data,/tmp/dist-test-taskgH5Pjm/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786277382073938-888-0/minicluster-data/master-0/wal
uuid: "d343a8cfeb614de282f065a5966ca6cd"
format_stamp: "Formatted at 2026-08-09 12:16:01 on dist-test-slave-2w3w"
server_key: "2126134e8b367ec225682320dceaa6b9"
server_key_iv: "a68621d35517bb9d53b0ec66153a9840"
server_key_version: "encryptionkey@0"
I20260809 12:16:01.656178 25577 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskgH5Pjm/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786277382073938-888-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskgH5Pjm/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786277382073938-888-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskgH5Pjm/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786277382073938-888-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260809 12:16:01.701563 25577 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260809 12:16:01.706350 25577 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260809 12:16:01.706689 25577 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260809 12:16:01.725996 25577 rpc_server.cc:307] RPC server started. Bound to: 127.0.222.62:39441
I20260809 12:16:01.726019 25650 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.222.62:39441 every 8 connection(s)
I20260809 12:16:01.727418 25577 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskgH5Pjm/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786277382073938-888-0/minicluster-data/master-0/data/info.pb
I20260809 12:16:01.731235 25652 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260809 12:16:01.732955 888 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskgH5Pjm/build/asan/bin/kudu as pid 25577
I20260809 12:16:01.733161 888 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskgH5Pjm/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786277382073938-888-0/minicluster-data/master-0/wal/instance
I20260809 12:16:01.733639 888 external_mini_cluster.cc:1428] Setting key 0b0c3964a11c54e80f42090af6c08c93
I20260809 12:16:01.737907 25652 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.222.62" port: 39441 } has no permanent_uuid. Determining permanent_uuid...
Aug 09 12:16:01 dist-test-slave-2w3w krb5kdc[25563](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786277761, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.222.62@KRBTEST.COM for kudu/127.0.222.62@KRBTEST.COM
I20260809 12:16:01.764427 25655 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 12:16:01.741038 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.222.62:39441 (local address 127.0.0.1:59310)
0809 12:16:01.742279 (+ 1241us) negotiation.cc:107] Waiting for socket to connect
0809 12:16:01.742326 (+ 47us) client_negotiation.cc:174] Beginning negotiation
0809 12:16:01.745072 (+ 2746us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 12:16:01.746110 (+ 1038us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 12:16:01.746115 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 12:16:01.746708 (+ 593us) client_negotiation.cc:189] Negotiated authn=SASL
0809 12:16:01.747103 (+ 395us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 12:16:01.747121 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 12:16:01.749382 (+ 2261us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 12:16:01.749389 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 12:16:01.750903 (+ 1514us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 12:16:01.750916 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 12:16:01.752360 (+ 1444us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 12:16:01.752366 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 12:16:01.752493 (+ 127us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 12:16:01.753216 (+ 723us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0809 12:16:01.753238 (+ 22us) client_negotiation.cc:656] calling sasl_client_start()
0809 12:16:01.755729 (+ 2491us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0809 12:16:01.760351 (+ 4622us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0809 12:16:01.760360 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0809 12:16:01.760369 (+ 9us) client_negotiation.cc:760] calling sasl_client_step()
0809 12:16:01.760844 (+ 475us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0809 12:16:01.761317 (+ 473us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0809 12:16:01.761320 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0809 12:16:01.761322 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0809 12:16:01.761485 (+ 163us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0809 12:16:01.762534 (+ 1049us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0809 12:16:01.762539 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0809 12:16:01.762723 (+ 184us) client_negotiation.cc:769] Sending connection context
0809 12:16:01.762920 (+ 197us) client_negotiation.cc:240] Negotiation successful
0809 12:16:01.763373 (+ 453us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":960,"spinlock_wait_cycles":56064,"thread_start_us":725,"threads_started":1}
I20260809 12:16:01.764467 25654 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 12:16:01.741039 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59310 (local address 127.0.222.62:39441)
0809 12:16:01.741948 (+ 909us) server_negotiation.cc:206] Beginning negotiation
0809 12:16:01.741975 (+ 27us) server_negotiation.cc:399] Waiting for connection header
0809 12:16:01.742461 (+ 486us) server_negotiation.cc:407] Connection header received
0809 12:16:01.745540 (+ 3079us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 12:16:01.745550 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 12:16:01.745866 (+ 316us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 12:16:01.746064 (+ 198us) server_negotiation.cc:226] Negotiated authn=SASL
0809 12:16:01.747340 (+ 1276us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 12:16:01.749226 (+ 1886us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 12:16:01.751123 (+ 1897us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 12:16:01.752126 (+ 1003us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 12:16:01.752360 (+ 234us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 12:16:01.756001 (+ 3641us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 12:16:01.756013 (+ 12us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 12:16:01.756030 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0809 12:16:01.756071 (+ 41us) server_negotiation.cc:922] calling sasl_server_start()
0809 12:16:01.760161 (+ 4090us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 12:16:01.760992 (+ 831us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 12:16:01.761003 (+ 11us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 12:16:01.761009 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0809 12:16:01.761201 (+ 192us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 12:16:01.761606 (+ 405us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 12:16:01.761614 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 12:16:01.761619 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0809 12:16:01.762397 (+ 778us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 12:16:01.762612 (+ 215us) server_negotiation.cc:1035] Waiting for connection context
0809 12:16:01.763145 (+ 533us) server_negotiation.cc:299] Negotiation successful
0809 12:16:01.763584 (+ 439us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":106,"server-negotiator.queue_time_us":624,"thread_start_us":374,"threads_started":1}
I20260809 12:16:01.767679 25652 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.222.61" port: 42263 } has no permanent_uuid. Determining permanent_uuid...
I20260809 12:16:01.768834 25655 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 12:16:01.768453 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.222.61:42263 (local address 127.0.0.1:47408)
0809 12:16:01.768626 (+ 173us) negotiation.cc:107] Waiting for socket to connect
0809 12:16:01.768731 (+ 105us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.222.61:42263: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":71}
W20260809 12:16:01.769084 25596 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.222.61:42263: connect: Connection refused (error 111)
WARNING: no policy specified for kudu/127.0.222.61@KRBTEST.COM; defaulting to no policy
I20260809 12:16:01.770733 25655 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 12:16:01.770332 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.222.61:42263 (local address 127.0.0.1:47422)
0809 12:16:01.770524 (+ 192us) negotiation.cc:107] Waiting for socket to connect
0809 12:16:01.770590 (+ 66us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.222.61:42263: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":50}
W20260809 12:16:01.771008 25652 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.222.61:42263: Network error: Client connection negotiation failed: client connection to 127.0.222.61:42263: connect: Connection refused (error 111)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.222.61@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.222.61 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskgH5Pjm/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786277382073938-888-0/krb5kdc/kudu_127.0.222.61.keytab.
Entry for principal kudu/127.0.222.61 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskgH5Pjm/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786277382073938-888-0/krb5kdc/kudu_127.0.222.61.keytab.
I20260809 12:16:01.814845 888 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskgH5Pjm/build/asan/bin/kudu
/tmp/dist-test-taskgH5Pjm/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskgH5Pjm/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786277382073938-888-0/minicluster-data/master-1/wal
--fs_data_dirs=/tmp/dist-test-taskgH5Pjm/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786277382073938-888-0/minicluster-data/master-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskgH5Pjm/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786277382073938-888-0/minicluster-data/master-1/logs
--server_dump_info_path=/tmp/dist-test-taskgH5Pjm/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786277382073938-888-0/minicluster-data/master-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.222.61:42263
--webserver_interface=127.0.222.61
--webserver_port=0
--keytab_file=/tmp/dist-test-taskgH5Pjm/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786277382073938-888-0/krb5kdc/kudu_127.0.222.61.keytab
--principal=kudu/127.0.222.61
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.222.20:32943
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.222.62:39441,127.0.222.61:42263,127.0.222.60:37589
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--enable_leader_failure_detection=false
--rpc_authentication=required
--rpc_encryption=required with env {KRB5CCNAME=/tmp/dist-test-taskgH5Pjm/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786277382073938-888-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskgH5Pjm/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786277382073938-888-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskgH5Pjm/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786277382073938-888-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260809 12:16:01.820379 25652 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.222.61" port: 42263 } attempt: 1
I20260809 12:16:01.821682 25655 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 12:16:01.821183 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.222.61:42263 (local address 127.0.0.1:47432)
0809 12:16:01.821409 (+ 226us) negotiation.cc:107] Waiting for socket to connect
0809 12:16:01.821517 (+ 108us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.222.61:42263: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":93}
I20260809 12:16:01.823081 25655 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 12:16:01.822568 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.222.61:42263 (local address 127.0.0.1:47444)
0809 12:16:01.822736 (+ 168us) negotiation.cc:107] Waiting for socket to connect
0809 12:16:01.822828 (+ 92us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.222.61:42263: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":57}
W20260809 12:16:01.823347 25652 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.222.61:42263: Network error: Client connection negotiation failed: client connection to 127.0.222.61:42263: connect: Connection refused (error 111)
I20260809 12:16:01.891626 25652 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.222.61" port: 42263 } attempt: 2
I20260809 12:16:01.892903 25655 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 12:16:01.892450 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.222.61:42263 (local address 127.0.0.1:47454)
0809 12:16:01.892656 (+ 206us) negotiation.cc:107] Waiting for socket to connect
0809 12:16:01.892750 (+ 94us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.222.61:42263: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":76}
I20260809 12:16:01.894018 25655 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 12:16:01.893637 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.222.61:42263 (local address 127.0.0.1:47458)
0809 12:16:01.893803 (+ 166us) negotiation.cc:107] Waiting for socket to connect
0809 12:16:01.893890 (+ 87us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.222.61:42263: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":59}
W20260809 12:16:01.894245 25652 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.222.61:42263: Network error: Client connection negotiation failed: client connection to 127.0.222.61:42263: connect: Connection refused (error 111)
I20260809 12:16:01.985570 25652 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.222.61" port: 42263 } attempt: 3
I20260809 12:16:01.986882 25655 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 12:16:01.986395 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.222.61:42263 (local address 127.0.0.1:47460)
0809 12:16:01.986625 (+ 230us) negotiation.cc:107] Waiting for socket to connect
0809 12:16:01.986737 (+ 112us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.222.61:42263: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":68}
I20260809 12:16:01.988052 25655 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 12:16:01.987685 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.222.61:42263 (local address 127.0.0.1:47468)
0809 12:16:01.987830 (+ 145us) negotiation.cc:107] Waiting for socket to connect
0809 12:16:01.987909 (+ 79us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.222.61:42263: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":56}
W20260809 12:16:01.988353 25652 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.222.61:42263: Network error: Client connection negotiation failed: client connection to 127.0.222.61:42263: connect: Connection refused (error 111)
I20260809 12:16:02.131613 25652 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.222.61" port: 42263 } attempt: 4
I20260809 12:16:02.132921 25655 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 12:16:02.132391 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.222.61:42263 (local address 127.0.0.1:47482)
0809 12:16:02.132701 (+ 310us) negotiation.cc:107] Waiting for socket to connect
0809 12:16:02.132791 (+ 90us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.222.61:42263: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":177}
I20260809 12:16:02.133953 25655 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 12:16:02.133639 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.222.61:42263 (local address 127.0.0.1:47494)
0809 12:16:02.133764 (+ 125us) negotiation.cc:107] Waiting for socket to connect
0809 12:16:02.133842 (+ 78us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.222.61:42263: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":47}
W20260809 12:16:02.134174 25652 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.222.61:42263: Network error: Client connection negotiation failed: client connection to 127.0.222.61:42263: connect: Connection refused (error 111)
W20260809 12:16:02.166592 25660 flags.cc:425] Enabled unsafe flag: --enable_leader_failure_detection=false
W20260809 12:16:02.166925 25660 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260809 12:16:02.167057 25660 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260809 12:16:02.167192 25660 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260809 12:16:02.174396 25660 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260809 12:16:02.174490 25660 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260809 12:16:02.174549 25660 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260809 12:16:02.174615 25660 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260809 12:16:02.174705 25660 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260809 12:16:02.184645 25660 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.222.20:32943
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_leader_failure_detection=false
--fs_data_dirs=/tmp/dist-test-taskgH5Pjm/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786277382073938-888-0/minicluster-data/master-1/data
--fs_wal_dir=/tmp/dist-test-taskgH5Pjm/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786277382073938-888-0/minicluster-data/master-1/wal
--ipki_ca_key_size=768
--master_addresses=127.0.222.62:39441,127.0.222.61:42263,127.0.222.60:37589
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskgH5Pjm/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786277382073938-888-0/krb5kdc/kudu_127.0.222.61.keytab
--principal=kudu/127.0.222.61
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.222.61:42263
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskgH5Pjm/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786277382073938-888-0/minicluster-data/master-1/data/info.pb
--webserver_interface=127.0.222.61
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskgH5Pjm/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786277382073938-888-0/minicluster-data/master-1/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 09 Aug 2026 11:06:41 UTC on ccycloud.quasar-bwoogy.root.comops.site
build id 4575991
ASAN enabled
I20260809 12:16:02.186048 25660 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260809 12:16:02.187441 25660 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260809 12:16:02.199899 25668 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 12:16:02.200011 25665 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 12:16:02.199981 25666 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260809 12:16:02.200743 25660 server_base.cc:1034] running on GCE node
I20260809 12:16:02.201516 25660 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260809 12:16:02.202936 25660 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260809 12:16:02.204236 25660 hybrid_clock.cc:648] HybridClock initialized: now 1786277762204173 us; error 36 us; skew 500 ppm
I20260809 12:16:02.204569 25660 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 09 12:16:02 dist-test-slave-2w3w krb5kdc[25563](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786277762, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.222.61@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260809 12:16:02.207895 25660 init.cc:377] Logged in from keytab as kudu/127.0.222.61@KRBTEST.COM (short username kudu)
I20260809 12:16:02.209455 25660 webserver.cc:460] Webserver started at http://127.0.222.61:34377/ using document root <none> and password file <none>
I20260809 12:16:02.210127 25660 fs_manager.cc:359] Metadata directory not provided
I20260809 12:16:02.210218 25660 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260809 12:16:02.210497 25660 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260809 12:16:02.213711 25660 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskgH5Pjm/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786277382073938-888-0/minicluster-data/master-1/data/instance:
uuid: "39e4d93ef8de492bae8bf9b7677b0128"
format_stamp: "Formatted at 2026-08-09 12:16:02 on dist-test-slave-2w3w"
server_key: "80e0d0769f901e21834fae9c420f670b"
server_key_iv: "1f330fcf0ff321bd7bb8435d328ae4d2"
server_key_version: "encryptionkey@0"
I20260809 12:16:02.214483 25660 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskgH5Pjm/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786277382073938-888-0/minicluster-data/master-1/wal/instance:
uuid: "39e4d93ef8de492bae8bf9b7677b0128"
format_stamp: "Formatted at 2026-08-09 12:16:02 on dist-test-slave-2w3w"
server_key: "80e0d0769f901e21834fae9c420f670b"
server_key_iv: "1f330fcf0ff321bd7bb8435d328ae4d2"
server_key_version: "encryptionkey@0"
I20260809 12:16:02.219925 25660 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.007s sys 0.001s
I20260809 12:16:02.223479 25675 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260809 12:16:02.224814 25660 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.002s
I20260809 12:16:02.225018 25660 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskgH5Pjm/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786277382073938-888-0/minicluster-data/master-1/data,/tmp/dist-test-taskgH5Pjm/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786277382073938-888-0/minicluster-data/master-1/wal
uuid: "39e4d93ef8de492bae8bf9b7677b0128"
format_stamp: "Formatted at 2026-08-09 12:16:02 on dist-test-slave-2w3w"
server_key: "80e0d0769f901e21834fae9c420f670b"
server_key_iv: "1f330fcf0ff321bd7bb8435d328ae4d2"
server_key_version: "encryptionkey@0"
I20260809 12:16:02.225241 25660 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskgH5Pjm/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786277382073938-888-0/minicluster-data/master-1/wal
metadata directory: /tmp/dist-test-taskgH5Pjm/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786277382073938-888-0/minicluster-data/master-1/wal
1 data directories: /tmp/dist-test-taskgH5Pjm/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786277382073938-888-0/minicluster-data/master-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260809 12:16:02.266295 25660 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260809 12:16:02.270243 25660 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260809 12:16:02.270519 25660 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260809 12:16:02.288906 25660 rpc_server.cc:307] RPC server started. Bound to: 127.0.222.61:42263
I20260809 12:16:02.288914 25731 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.222.61:42263 every 8 connection(s)
I20260809 12:16:02.290277 25660 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskgH5Pjm/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786277382073938-888-0/minicluster-data/master-1/data/info.pb
I20260809 12:16:02.292169 888 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskgH5Pjm/build/asan/bin/kudu as pid 25660
I20260809 12:16:02.292375 888 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskgH5Pjm/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786277382073938-888-0/minicluster-data/master-1/wal/instance
I20260809 12:16:02.292889 888 external_mini_cluster.cc:1428] Setting key aacafa5cb5ba340ba96584b668254d21
I20260809 12:16:02.294122 25732 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260809 12:16:02.300760 25732 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.222.62" port: 39441 } has no permanent_uuid. Determining permanent_uuid...
Aug 09 12:16:02 dist-test-slave-2w3w krb5kdc[25563](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786277762, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.222.61@KRBTEST.COM for kudu/127.0.222.62@KRBTEST.COM
I20260809 12:16:02.323052 25734 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 12:16:02.303377 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59312 (local address 127.0.222.62:39441)
0809 12:16:02.303977 (+ 600us) server_negotiation.cc:206] Beginning negotiation
0809 12:16:02.303986 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0809 12:16:02.304973 (+ 987us) server_negotiation.cc:407] Connection header received
0809 12:16:02.306939 (+ 1966us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 12:16:02.306948 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 12:16:02.307067 (+ 119us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 12:16:02.307203 (+ 136us) server_negotiation.cc:226] Negotiated authn=SASL
0809 12:16:02.308828 (+ 1625us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 12:16:02.309726 (+ 898us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 12:16:02.312227 (+ 2501us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 12:16:02.312642 (+ 415us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 12:16:02.312740 (+ 98us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 12:16:02.317104 (+ 4364us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 12:16:02.317109 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 12:16:02.317113 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0809 12:16:02.317149 (+ 36us) server_negotiation.cc:922] calling sasl_server_start()
0809 12:16:02.319935 (+ 2786us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 12:16:02.320674 (+ 739us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 12:16:02.320688 (+ 14us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 12:16:02.320692 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0809 12:16:02.320813 (+ 121us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 12:16:02.321243 (+ 430us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 12:16:02.321249 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 12:16:02.321252 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0809 12:16:02.321537 (+ 285us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 12:16:02.321710 (+ 173us) server_negotiation.cc:1035] Waiting for connection context
0809 12:16:02.322400 (+ 690us) server_negotiation.cc:299] Negotiation successful
0809 12:16:02.322626 (+ 226us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":432,"thread_start_us":280,"threads_started":1}
I20260809 12:16:02.324138 25735 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 12:16:02.303962 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.222.62:39441 (local address 127.0.0.1:59312)
0809 12:16:02.304776 (+ 814us) negotiation.cc:107] Waiting for socket to connect
0809 12:16:02.304840 (+ 64us) client_negotiation.cc:174] Beginning negotiation
0809 12:16:02.306659 (+ 1819us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 12:16:02.307394 (+ 735us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 12:16:02.307401 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 12:16:02.308042 (+ 641us) client_negotiation.cc:189] Negotiated authn=SASL
0809 12:16:02.308586 (+ 544us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 12:16:02.308603 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 12:16:02.309909 (+ 1306us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 12:16:02.309917 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 12:16:02.312085 (+ 2168us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 12:16:02.312103 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 12:16:02.312875 (+ 772us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 12:16:02.312885 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 12:16:02.313260 (+ 375us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 12:16:02.314041 (+ 781us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0809 12:16:02.314075 (+ 34us) client_negotiation.cc:656] calling sasl_client_start()
0809 12:16:02.316867 (+ 2792us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0809 12:16:02.320083 (+ 3216us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0809 12:16:02.320092 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0809 12:16:02.320110 (+ 18us) client_negotiation.cc:760] calling sasl_client_step()
0809 12:16:02.320548 (+ 438us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0809 12:16:02.320931 (+ 383us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0809 12:16:02.320936 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0809 12:16:02.320940 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0809 12:16:02.321130 (+ 190us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0809 12:16:02.321668 (+ 538us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0809 12:16:02.321677 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0809 12:16:02.322050 (+ 373us) client_negotiation.cc:769] Sending connection context
0809 12:16:02.322333 (+ 283us) client_negotiation.cc:240] Negotiation successful
0809 12:16:02.322854 (+ 521us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":551,"thread_start_us":356,"threads_started":1}
I20260809 12:16:02.326146 25732 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.222.61" port: 42263 } has no permanent_uuid. Determining permanent_uuid...
WARNING: no policy specified for kudu/127.0.222.60@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.222.60@KRBTEST.COM" created.
Aug 09 12:16:02 dist-test-slave-2w3w krb5kdc[25563](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786277762, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.222.61@KRBTEST.COM for kudu/127.0.222.61@KRBTEST.COM
I20260809 12:16:02.342233 25737 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 12:16:02.327047 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47502 (local address 127.0.222.61:42263)
0809 12:16:02.327699 (+ 652us) server_negotiation.cc:206] Beginning negotiation
0809 12:16:02.327704 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0809 12:16:02.327729 (+ 25us) server_negotiation.cc:407] Connection header received
0809 12:16:02.327772 (+ 43us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 12:16:02.327777 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 12:16:02.327950 (+ 173us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 12:16:02.328067 (+ 117us) server_negotiation.cc:226] Negotiated authn=SASL
0809 12:16:02.328980 (+ 913us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 12:16:02.330309 (+ 1329us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 12:16:02.331774 (+ 1465us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 12:16:02.332746 (+ 972us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 12:16:02.332888 (+ 142us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 12:16:02.336399 (+ 3511us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 12:16:02.336411 (+ 12us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 12:16:02.336425 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0809 12:16:02.336453 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0809 12:16:02.339281 (+ 2828us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 12:16:02.340072 (+ 791us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 12:16:02.340079 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 12:16:02.340081 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0809 12:16:02.340201 (+ 120us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 12:16:02.340620 (+ 419us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 12:16:02.340624 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 12:16:02.340625 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0809 12:16:02.340968 (+ 343us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 12:16:02.341138 (+ 170us) server_negotiation.cc:1035] Waiting for connection context
0809 12:16:02.341613 (+ 475us) server_negotiation.cc:299] Negotiation successful
0809 12:16:02.341907 (+ 294us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":93,"server-negotiator.queue_time_us":499,"thread_start_us":353,"threads_started":1}
I20260809 12:16:02.342332 25735 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 12:16:02.326891 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.222.61:42263 (local address 127.0.0.1:47502)
0809 12:16:02.327148 (+ 257us) negotiation.cc:107] Waiting for socket to connect
0809 12:16:02.327170 (+ 22us) client_negotiation.cc:174] Beginning negotiation
0809 12:16:02.327285 (+ 115us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 12:16:02.328123 (+ 838us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 12:16:02.328129 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 12:16:02.328664 (+ 535us) client_negotiation.cc:189] Negotiated authn=SASL
0809 12:16:02.328812 (+ 148us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 12:16:02.328820 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 12:16:02.330456 (+ 1636us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 12:16:02.330460 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 12:16:02.331630 (+ 1170us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 12:16:02.331643 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 12:16:02.332902 (+ 1259us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 12:16:02.332906 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 12:16:02.333041 (+ 135us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 12:16:02.333669 (+ 628us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0809 12:16:02.333694 (+ 25us) client_negotiation.cc:656] calling sasl_client_start()
0809 12:16:02.336190 (+ 2496us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0809 12:16:02.339412 (+ 3222us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0809 12:16:02.339419 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0809 12:16:02.339425 (+ 6us) client_negotiation.cc:760] calling sasl_client_step()
0809 12:16:02.339946 (+ 521us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0809 12:16:02.340306 (+ 360us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0809 12:16:02.340314 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0809 12:16:02.340320 (+ 6us) client_negotiation.cc:760] calling sasl_client_step()
0809 12:16:02.340511 (+ 191us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0809 12:16:02.341120 (+ 609us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0809 12:16:02.341127 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0809 12:16:02.341298 (+ 171us) client_negotiation.cc:769] Sending connection context
0809 12:16:02.341462 (+ 164us) client_negotiation.cc:240] Negotiation successful
0809 12:16:02.341797 (+ 335us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":102}
I20260809 12:16:02.344959 25732 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.222.60" port: 37589 } has no permanent_uuid. Determining permanent_uuid...
I20260809 12:16:02.346184 25735 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 12:16:02.345735 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.222.60:37589 (local address 127.0.0.1:40288)
0809 12:16:02.345904 (+ 169us) negotiation.cc:107] Waiting for socket to connect
0809 12:16:02.346051 (+ 147us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.222.60:37589: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":56}
W20260809 12:16:02.346570 25677 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.222.60:37589: connect: Connection refused (error 111)
I20260809 12:16:02.348320 25735 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 12:16:02.347884 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.222.60:37589 (local address 127.0.0.1:40294)
0809 12:16:02.348075 (+ 191us) negotiation.cc:107] Waiting for socket to connect
0809 12:16:02.348178 (+ 103us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.222.60:37589: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":56}
W20260809 12:16:02.348553 25732 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.222.60:37589: Network error: Client connection negotiation failed: client connection to 127.0.222.60:37589: connect: Connection refused (error 111)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.222.60 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskgH5Pjm/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786277382073938-888-0/krb5kdc/kudu_127.0.222.60.keytab.
Entry for principal kudu/127.0.222.60 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskgH5Pjm/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786277382073938-888-0/krb5kdc/kudu_127.0.222.60.keytab.
I20260809 12:16:02.372296 888 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskgH5Pjm/build/asan/bin/kudu
/tmp/dist-test-taskgH5Pjm/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskgH5Pjm/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786277382073938-888-0/minicluster-data/master-2/wal
--fs_data_dirs=/tmp/dist-test-taskgH5Pjm/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786277382073938-888-0/minicluster-data/master-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskgH5Pjm/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786277382073938-888-0/minicluster-data/master-2/logs
--server_dump_info_path=/tmp/dist-test-taskgH5Pjm/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786277382073938-888-0/minicluster-data/master-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.222.60:37589
--webserver_interface=127.0.222.60
--webserver_port=0
--keytab_file=/tmp/dist-test-taskgH5Pjm/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786277382073938-888-0/krb5kdc/kudu_127.0.222.60.keytab
--principal=kudu/127.0.222.60
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.222.20:32943
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.222.62:39441,127.0.222.61:42263,127.0.222.60:37589
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--enable_leader_failure_detection=false
--rpc_authentication=required
--rpc_encryption=required with env {KRB5CCNAME=/tmp/dist-test-taskgH5Pjm/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786277382073938-888-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskgH5Pjm/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786277382073938-888-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskgH5Pjm/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786277382073938-888-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260809 12:16:02.397913 25732 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.222.60" port: 37589 } attempt: 1
I20260809 12:16:02.399362 25735 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 12:16:02.398709 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.222.60:37589 (local address 127.0.0.1:40310)
0809 12:16:02.399002 (+ 293us) negotiation.cc:107] Waiting for socket to connect
0809 12:16:02.399147 (+ 145us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.222.60:37589: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":87}
I20260809 12:16:02.400553 25735 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 12:16:02.400182 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.222.60:37589 (local address 127.0.0.1:40324)
0809 12:16:02.400333 (+ 151us) negotiation.cc:107] Waiting for socket to connect
0809 12:16:02.400431 (+ 98us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.222.60:37589: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":51}
W20260809 12:16:02.400821 25732 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.222.60:37589: Network error: Client connection negotiation failed: client connection to 127.0.222.60:37589: connect: Connection refused (error 111)
I20260809 12:16:02.433466 25652 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.222.61" port: 42263 } attempt: 5
Aug 09 12:16:02 dist-test-slave-2w3w krb5kdc[25563](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786277761, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.222.62@KRBTEST.COM for kudu/127.0.222.61@KRBTEST.COM
I20260809 12:16:02.448596 25655 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 12:16:02.434319 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.222.61:42263 (local address 127.0.0.1:47516)
0809 12:16:02.434581 (+ 262us) negotiation.cc:107] Waiting for socket to connect
0809 12:16:02.434608 (+ 27us) client_negotiation.cc:174] Beginning negotiation
0809 12:16:02.434781 (+ 173us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 12:16:02.435227 (+ 446us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 12:16:02.435234 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 12:16:02.435861 (+ 627us) client_negotiation.cc:189] Negotiated authn=SASL
0809 12:16:02.436181 (+ 320us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 12:16:02.436199 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 12:16:02.437251 (+ 1052us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 12:16:02.437259 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 12:16:02.438672 (+ 1413us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 12:16:02.438689 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 12:16:02.439503 (+ 814us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 12:16:02.439509 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 12:16:02.439756 (+ 247us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 12:16:02.440448 (+ 692us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0809 12:16:02.440471 (+ 23us) client_negotiation.cc:656] calling sasl_client_start()
0809 12:16:02.443036 (+ 2565us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0809 12:16:02.446085 (+ 3049us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0809 12:16:02.446090 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0809 12:16:02.446095 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0809 12:16:02.446474 (+ 379us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0809 12:16:02.446840 (+ 366us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0809 12:16:02.446845 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0809 12:16:02.446849 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0809 12:16:02.447046 (+ 197us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0809 12:16:02.447582 (+ 536us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0809 12:16:02.447587 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0809 12:16:02.447745 (+ 158us) client_negotiation.cc:769] Sending connection context
0809 12:16:02.447935 (+ 190us) client_negotiation.cc:240] Negotiation successful
0809 12:16:02.448231 (+ 296us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":80}
I20260809 12:16:02.448623 25737 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 12:16:02.434446 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47516 (local address 127.0.222.61:42263)
0809 12:16:02.434640 (+ 194us) server_negotiation.cc:206] Beginning negotiation
0809 12:16:02.434646 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0809 12:16:02.434671 (+ 25us) server_negotiation.cc:407] Connection header received
0809 12:16:02.434950 (+ 279us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 12:16:02.434955 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 12:16:02.435047 (+ 92us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 12:16:02.435135 (+ 88us) server_negotiation.cc:226] Negotiated authn=SASL
0809 12:16:02.436368 (+ 1233us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 12:16:02.437093 (+ 725us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 12:16:02.438849 (+ 1756us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 12:16:02.439347 (+ 498us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 12:16:02.439448 (+ 101us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 12:16:02.443238 (+ 3790us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 12:16:02.443243 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 12:16:02.443246 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0809 12:16:02.443286 (+ 40us) server_negotiation.cc:922] calling sasl_server_start()
0809 12:16:02.445946 (+ 2660us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 12:16:02.446591 (+ 645us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 12:16:02.446597 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 12:16:02.446600 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0809 12:16:02.446727 (+ 127us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 12:16:02.447163 (+ 436us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 12:16:02.447168 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 12:16:02.447176 (+ 8us) server_negotiation.cc:967] calling sasl_server_step()
0809 12:16:02.447458 (+ 282us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 12:16:02.447616 (+ 158us) server_negotiation.cc:1035] Waiting for connection context
0809 12:16:02.448049 (+ 433us) server_negotiation.cc:299] Negotiation successful
0809 12:16:02.448327 (+ 278us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":81}
I20260809 12:16:02.449693 25652 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.222.60" port: 37589 } has no permanent_uuid. Determining permanent_uuid...
I20260809 12:16:02.450866 25655 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 12:16:02.450401 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.222.60:37589 (local address 127.0.0.1:40328)
0809 12:16:02.450575 (+ 174us) negotiation.cc:107] Waiting for socket to connect
0809 12:16:02.450661 (+ 86us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.222.60:37589: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":59}
I20260809 12:16:02.451902 25655 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 12:16:02.451527 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.222.60:37589 (local address 127.0.0.1:40330)
0809 12:16:02.451679 (+ 152us) negotiation.cc:107] Waiting for socket to connect
0809 12:16:02.451785 (+ 106us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.222.60:37589: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":59}
W20260809 12:16:02.452106 25652 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.222.60:37589: Network error: Client connection negotiation failed: client connection to 127.0.222.60:37589: connect: Connection refused (error 111)
I20260809 12:16:02.469043 25732 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.222.60" port: 37589 } attempt: 2
I20260809 12:16:02.470175 25735 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 12:16:02.469738 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.222.60:37589 (local address 127.0.0.1:40340)
0809 12:16:02.469926 (+ 188us) negotiation.cc:107] Waiting for socket to connect
0809 12:16:02.470017 (+ 91us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.222.60:37589: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":65}
I20260809 12:16:02.471282 25735 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 12:16:02.470858 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.222.60:37589 (local address 127.0.0.1:40344)
0809 12:16:02.471046 (+ 188us) negotiation.cc:107] Waiting for socket to connect
0809 12:16:02.471113 (+ 67us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.222.60:37589: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":58}
W20260809 12:16:02.471517 25732 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.222.60:37589: Network error: Client connection negotiation failed: client connection to 127.0.222.60:37589: connect: Connection refused (error 111)
I20260809 12:16:02.503326 25652 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.222.60" port: 37589 } attempt: 1
I20260809 12:16:02.504504 25655 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 12:16:02.504026 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.222.60:37589 (local address 127.0.0.1:40346)
0809 12:16:02.504227 (+ 201us) negotiation.cc:107] Waiting for socket to connect
0809 12:16:02.504329 (+ 102us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.222.60:37589: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":67}
I20260809 12:16:02.505589 25655 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 12:16:02.505256 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.222.60:37589 (local address 127.0.0.1:40352)
0809 12:16:02.505420 (+ 164us) negotiation.cc:107] Waiting for socket to connect
0809 12:16:02.505484 (+ 64us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.222.60:37589: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":52}
W20260809 12:16:02.505803 25652 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.222.60:37589: Network error: Client connection negotiation failed: client connection to 127.0.222.60:37589: connect: Connection refused (error 111)
I20260809 12:16:02.562772 25732 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.222.60" port: 37589 } attempt: 3
I20260809 12:16:02.564065 25735 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 12:16:02.563543 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.222.60:37589 (local address 127.0.0.1:40354)
0809 12:16:02.563793 (+ 250us) negotiation.cc:107] Waiting for socket to connect
0809 12:16:02.563891 (+ 98us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.222.60:37589: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":78}
I20260809 12:16:02.565441 25735 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 12:16:02.564954 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.222.60:37589 (local address 127.0.0.1:40362)
0809 12:16:02.565215 (+ 261us) negotiation.cc:107] Waiting for socket to connect
0809 12:16:02.565318 (+ 103us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.222.60:37589: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":149}
W20260809 12:16:02.565672 25732 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.222.60:37589: Network error: Client connection negotiation failed: client connection to 127.0.222.60:37589: connect: Connection refused (error 111)
I20260809 12:16:02.574028 25652 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.222.60" port: 37589 } attempt: 2
I20260809 12:16:02.575237 25655 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 12:16:02.574699 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.222.60:37589 (local address 127.0.0.1:40370)
0809 12:16:02.574916 (+ 217us) negotiation.cc:107] Waiting for socket to connect
0809 12:16:02.575068 (+ 152us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.222.60:37589: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":67}
I20260809 12:16:02.576259 25655 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 12:16:02.575952 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.222.60:37589 (local address 127.0.0.1:40374)
0809 12:16:02.576099 (+ 147us) negotiation.cc:107] Waiting for socket to connect
0809 12:16:02.576161 (+ 62us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.222.60:37589: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":47}
W20260809 12:16:02.576469 25652 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.222.60:37589: Network error: Client connection negotiation failed: client connection to 127.0.222.60:37589: connect: Connection refused (error 111)
I20260809 12:16:02.682698 25652 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.222.60" port: 37589 } attempt: 3
I20260809 12:16:02.683888 25655 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 12:16:02.683400 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.222.60:37589 (local address 127.0.0.1:40388)
0809 12:16:02.683611 (+ 211us) negotiation.cc:107] Waiting for socket to connect
0809 12:16:02.683708 (+ 97us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.222.60:37589: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":74}
I20260809 12:16:02.685115 25655 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 12:16:02.684645 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.222.60:37589 (local address 127.0.0.1:40404)
0809 12:16:02.684872 (+ 227us) negotiation.cc:107] Waiting for socket to connect
0809 12:16:02.684987 (+ 115us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.222.60:37589: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":90}
W20260809 12:16:02.685313 25652 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.222.60:37589: Network error: Client connection negotiation failed: client connection to 127.0.222.60:37589: connect: Connection refused (error 111)
I20260809 12:16:02.708916 25732 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.222.60" port: 37589 } attempt: 4
I20260809 12:16:02.710060 25735 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 12:16:02.709569 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.222.60:37589 (local address 127.0.0.1:40406)
0809 12:16:02.709813 (+ 244us) negotiation.cc:107] Waiting for socket to connect
0809 12:16:02.709908 (+ 95us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.222.60:37589: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":72}
I20260809 12:16:02.711126 25735 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 12:16:02.710750 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.222.60:37589 (local address 127.0.0.1:40408)
0809 12:16:02.710905 (+ 155us) negotiation.cc:107] Waiting for socket to connect
0809 12:16:02.711018 (+ 113us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.222.60:37589: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":48}
W20260809 12:16:02.711320 25732 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.222.60:37589: Network error: Client connection negotiation failed: client connection to 127.0.222.60:37589: connect: Connection refused (error 111)
W20260809 12:16:02.730017 25740 flags.cc:425] Enabled unsafe flag: --enable_leader_failure_detection=false
W20260809 12:16:02.730405 25740 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260809 12:16:02.730528 25740 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260809 12:16:02.730664 25740 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260809 12:16:02.738265 25740 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260809 12:16:02.738410 25740 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260809 12:16:02.738536 25740 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260809 12:16:02.738633 25740 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260809 12:16:02.738745 25740 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260809 12:16:02.749037 25740 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.222.20:32943
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_leader_failure_detection=false
--fs_data_dirs=/tmp/dist-test-taskgH5Pjm/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786277382073938-888-0/minicluster-data/master-2/data
--fs_wal_dir=/tmp/dist-test-taskgH5Pjm/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786277382073938-888-0/minicluster-data/master-2/wal
--ipki_ca_key_size=768
--master_addresses=127.0.222.62:39441,127.0.222.61:42263,127.0.222.60:37589
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskgH5Pjm/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786277382073938-888-0/krb5kdc/kudu_127.0.222.60.keytab
--principal=kudu/127.0.222.60
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.222.60:37589
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskgH5Pjm/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786277382073938-888-0/minicluster-data/master-2/data/info.pb
--webserver_interface=127.0.222.60
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskgH5Pjm/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786277382073938-888-0/minicluster-data/master-2/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 09 Aug 2026 11:06:41 UTC on ccycloud.quasar-bwoogy.root.comops.site
build id 4575991
ASAN enabled
I20260809 12:16:02.750401 25740 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260809 12:16:02.751818 25740 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260809 12:16:02.764072 25745 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 12:16:02.764117 25746 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 12:16:02.764030 25748 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260809 12:16:02.764958 25740 server_base.cc:1034] running on GCE node
I20260809 12:16:02.765751 25740 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260809 12:16:02.767202 25740 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260809 12:16:02.768424 25740 hybrid_clock.cc:648] HybridClock initialized: now 1786277762768359 us; error 37 us; skew 500 ppm
I20260809 12:16:02.768764 25740 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 09 12:16:02 dist-test-slave-2w3w krb5kdc[25563](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786277762, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.222.60@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260809 12:16:02.775285 25740 init.cc:377] Logged in from keytab as kudu/127.0.222.60@KRBTEST.COM (short username kudu)
I20260809 12:16:02.776904 25740 webserver.cc:460] Webserver started at http://127.0.222.60:36793/ using document root <none> and password file <none>
I20260809 12:16:02.777616 25740 fs_manager.cc:359] Metadata directory not provided
I20260809 12:16:02.777732 25740 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260809 12:16:02.778079 25740 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260809 12:16:02.781140 25740 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskgH5Pjm/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786277382073938-888-0/minicluster-data/master-2/data/instance:
uuid: "865df34449d0455398e47baaff333baf"
format_stamp: "Formatted at 2026-08-09 12:16:02 on dist-test-slave-2w3w"
server_key: "cd354decb5b1bb094d8edf09468008c0"
server_key_iv: "ed860decd93be644e63c55350570b35b"
server_key_version: "encryptionkey@0"
I20260809 12:16:02.781945 25740 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskgH5Pjm/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786277382073938-888-0/minicluster-data/master-2/wal/instance:
uuid: "865df34449d0455398e47baaff333baf"
format_stamp: "Formatted at 2026-08-09 12:16:02 on dist-test-slave-2w3w"
server_key: "cd354decb5b1bb094d8edf09468008c0"
server_key_iv: "ed860decd93be644e63c55350570b35b"
server_key_version: "encryptionkey@0"
I20260809 12:16:02.787372 25740 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.007s sys 0.000s
I20260809 12:16:02.791031 25757 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260809 12:16:02.792403 25740 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260809 12:16:02.792590 25740 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskgH5Pjm/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786277382073938-888-0/minicluster-data/master-2/data,/tmp/dist-test-taskgH5Pjm/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786277382073938-888-0/minicluster-data/master-2/wal
uuid: "865df34449d0455398e47baaff333baf"
format_stamp: "Formatted at 2026-08-09 12:16:02 on dist-test-slave-2w3w"
server_key: "cd354decb5b1bb094d8edf09468008c0"
server_key_iv: "ed860decd93be644e63c55350570b35b"
server_key_version: "encryptionkey@0"
I20260809 12:16:02.792819 25740 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskgH5Pjm/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786277382073938-888-0/minicluster-data/master-2/wal
metadata directory: /tmp/dist-test-taskgH5Pjm/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786277382073938-888-0/minicluster-data/master-2/wal
1 data directories: /tmp/dist-test-taskgH5Pjm/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786277382073938-888-0/minicluster-data/master-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260809 12:16:02.815968 25740 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260809 12:16:02.820397 25740 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260809 12:16:02.820712 25740 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260809 12:16:02.841142 25740 rpc_server.cc:307] RPC server started. Bound to: 127.0.222.60:37589
I20260809 12:16:02.841162 25817 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.222.60:37589 every 8 connection(s)
I20260809 12:16:02.842615 25740 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskgH5Pjm/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786277382073938-888-0/minicluster-data/master-2/data/info.pb
I20260809 12:16:02.846354 25818 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260809 12:16:02.848373 888 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskgH5Pjm/build/asan/bin/kudu as pid 25740
I20260809 12:16:02.848596 888 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskgH5Pjm/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786277382073938-888-0/minicluster-data/master-2/wal/instance
I20260809 12:16:02.849206 888 external_mini_cluster.cc:1428] Setting key e71f67c69f9b912367a4f5236caa22ea
I20260809 12:16:02.858121 25818 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.222.62" port: 39441 } has no permanent_uuid. Determining permanent_uuid...
I20260809 12:16:02.862838 25652 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.222.60" port: 37589 } attempt: 4
Aug 09 12:16:02 dist-test-slave-2w3w krb5kdc[25563](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786277761, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.222.60@KRBTEST.COM
Aug 09 12:16:02 dist-test-slave-2w3w krb5kdc[25563](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786277761, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.222.62@KRBTEST.COM for kudu/127.0.222.60@KRBTEST.COM
Aug 09 12:16:02 dist-test-slave-2w3w krb5kdc[25563](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786277762, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.222.60@KRBTEST.COM for kudu/127.0.222.62@KRBTEST.COM
I20260809 12:16:02.878218 25821 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 12:16:02.852095 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40420 (local address 127.0.222.60:37589)
0809 12:16:02.852990 (+ 895us) server_negotiation.cc:206] Beginning negotiation
0809 12:16:02.853007 (+ 17us) server_negotiation.cc:399] Waiting for connection header
0809 12:16:02.853069 (+ 62us) server_negotiation.cc:407] Connection header received
0809 12:16:02.854930 (+ 1861us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 12:16:02.855005 (+ 75us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 12:16:02.855579 (+ 574us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 12:16:02.856059 (+ 480us) server_negotiation.cc:226] Negotiated authn=SASL
0809 12:16:02.857431 (+ 1372us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 12:16:02.859727 (+ 2296us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 12:16:02.861217 (+ 1490us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 12:16:02.862124 (+ 907us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 12:16:02.862436 (+ 312us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 12:16:02.867451 (+ 5015us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 12:16:02.867466 (+ 15us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 12:16:02.867481 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0809 12:16:02.867550 (+ 69us) server_negotiation.cc:922] calling sasl_server_start()
0809 12:16:02.871889 (+ 4339us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 12:16:02.873930 (+ 2041us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 12:16:02.873942 (+ 12us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 12:16:02.873953 (+ 11us) server_negotiation.cc:967] calling sasl_server_step()
0809 12:16:02.874143 (+ 190us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 12:16:02.874557 (+ 414us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 12:16:02.874563 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 12:16:02.874568 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0809 12:16:02.875361 (+ 793us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 12:16:02.875719 (+ 358us) server_negotiation.cc:1035] Waiting for connection context
0809 12:16:02.876166 (+ 447us) server_negotiation.cc:299] Negotiation successful
0809 12:16:02.876636 (+ 470us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":631,"thread_start_us":388,"threads_started":1}
I20260809 12:16:02.880803 25824 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 12:16:02.861334 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.222.62:39441 (local address 127.0.0.1:59328)
0809 12:16:02.862432 (+ 1098us) negotiation.cc:107] Waiting for socket to connect
0809 12:16:02.862481 (+ 49us) client_negotiation.cc:174] Beginning negotiation
0809 12:16:02.862648 (+ 167us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 12:16:02.864713 (+ 2065us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 12:16:02.864718 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 12:16:02.865364 (+ 646us) client_negotiation.cc:189] Negotiated authn=SASL
0809 12:16:02.865851 (+ 487us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 12:16:02.865873 (+ 22us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 12:16:02.867275 (+ 1402us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 12:16:02.867280 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 12:16:02.868504 (+ 1224us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 12:16:02.868521 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 12:16:02.869780 (+ 1259us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 12:16:02.869792 (+ 12us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 12:16:02.869956 (+ 164us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 12:16:02.870704 (+ 748us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0809 12:16:02.870734 (+ 30us) client_negotiation.cc:656] calling sasl_client_start()
0809 12:16:02.874205 (+ 3471us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0809 12:16:02.877690 (+ 3485us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0809 12:16:02.877703 (+ 13us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0809 12:16:02.877710 (+ 7us) client_negotiation.cc:760] calling sasl_client_step()
0809 12:16:02.878255 (+ 545us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0809 12:16:02.878667 (+ 412us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0809 12:16:02.878673 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0809 12:16:02.878676 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0809 12:16:02.878853 (+ 177us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0809 12:16:02.879517 (+ 664us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0809 12:16:02.879525 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0809 12:16:02.879718 (+ 193us) client_negotiation.cc:769] Sending connection context
0809 12:16:02.879969 (+ 251us) client_negotiation.cc:240] Negotiation successful
0809 12:16:02.880329 (+ 360us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":744,"thread_start_us":565,"threads_started":1}
I20260809 12:16:02.881561 25655 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 12:16:02.863439 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.222.60:37589 (local address 127.0.0.1:40422)
0809 12:16:02.863633 (+ 194us) negotiation.cc:107] Waiting for socket to connect
0809 12:16:02.863652 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0809 12:16:02.863755 (+ 103us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 12:16:02.864574 (+ 819us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 12:16:02.864581 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 12:16:02.865139 (+ 558us) client_negotiation.cc:189] Negotiated authn=SASL
0809 12:16:02.865356 (+ 217us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 12:16:02.865372 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 12:16:02.867042 (+ 1670us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 12:16:02.867048 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 12:16:02.868646 (+ 1598us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 12:16:02.868664 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 12:16:02.869732 (+ 1068us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 12:16:02.869738 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 12:16:02.869876 (+ 138us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 12:16:02.870438 (+ 562us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0809 12:16:02.870488 (+ 50us) client_negotiation.cc:656] calling sasl_client_start()
0809 12:16:02.873149 (+ 2661us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0809 12:16:02.878283 (+ 5134us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0809 12:16:02.878541 (+ 258us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0809 12:16:02.878551 (+ 10us) client_negotiation.cc:760] calling sasl_client_step()
0809 12:16:02.879136 (+ 585us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0809 12:16:02.879578 (+ 442us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0809 12:16:02.879585 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0809 12:16:02.879589 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0809 12:16:02.879743 (+ 154us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0809 12:16:02.880506 (+ 763us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0809 12:16:02.880511 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0809 12:16:02.880676 (+ 165us) client_negotiation.cc:769] Sending connection context
0809 12:16:02.880833 (+ 157us) client_negotiation.cc:240] Negotiation successful
0809 12:16:02.881146 (+ 313us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":66}
I20260809 12:16:02.881271 25822 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 12:16:02.861305 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59328 (local address 127.0.222.62:39441)
0809 12:16:02.861822 (+ 517us) server_negotiation.cc:206] Beginning negotiation
0809 12:16:02.861829 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0809 12:16:02.862671 (+ 842us) server_negotiation.cc:407] Connection header received
0809 12:16:02.862755 (+ 84us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 12:16:02.862763 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 12:16:02.862892 (+ 129us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 12:16:02.863054 (+ 162us) server_negotiation.cc:226] Negotiated authn=SASL
0809 12:16:02.866048 (+ 2994us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 12:16:02.867110 (+ 1062us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 12:16:02.868717 (+ 1607us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 12:16:02.869390 (+ 673us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 12:16:02.869523 (+ 133us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 12:16:02.874425 (+ 4902us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 12:16:02.874431 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 12:16:02.874435 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0809 12:16:02.874475 (+ 40us) server_negotiation.cc:922] calling sasl_server_start()
0809 12:16:02.877553 (+ 3078us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 12:16:02.878393 (+ 840us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 12:16:02.878399 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 12:16:02.878403 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0809 12:16:02.878531 (+ 128us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 12:16:02.879012 (+ 481us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 12:16:02.879018 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 12:16:02.879022 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0809 12:16:02.879361 (+ 339us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 12:16:02.879527 (+ 166us) server_negotiation.cc:1035] Waiting for connection context
0809 12:16:02.880096 (+ 569us) server_negotiation.cc:299] Negotiation successful
0809 12:16:02.880335 (+ 239us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":381,"thread_start_us":260,"threads_started":1}
I20260809 12:16:02.882054 25825 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 12:16:02.863626 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40422 (local address 127.0.222.60:37589)
0809 12:16:02.864287 (+ 661us) server_negotiation.cc:206] Beginning negotiation
0809 12:16:02.864291 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0809 12:16:02.864311 (+ 20us) server_negotiation.cc:407] Connection header received
0809 12:16:02.864350 (+ 39us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 12:16:02.864354 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 12:16:02.864406 (+ 52us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 12:16:02.864496 (+ 90us) server_negotiation.cc:226] Negotiated authn=SASL
0809 12:16:02.865574 (+ 1078us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 12:16:02.866824 (+ 1250us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 12:16:02.868831 (+ 2007us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 12:16:02.869526 (+ 695us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 12:16:02.869693 (+ 167us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 12:16:02.873354 (+ 3661us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 12:16:02.873359 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 12:16:02.873362 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0809 12:16:02.873392 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0809 12:16:02.878104 (+ 4712us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 12:16:02.879294 (+ 1190us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 12:16:02.879299 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 12:16:02.879302 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0809 12:16:02.879438 (+ 136us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 12:16:02.879898 (+ 460us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 12:16:02.879901 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 12:16:02.879903 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0809 12:16:02.880333 (+ 430us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 12:16:02.881369 (+ 1036us) server_negotiation.cc:1035] Waiting for connection context
0809 12:16:02.881505 (+ 136us) server_negotiation.cc:299] Negotiation successful
0809 12:16:02.881720 (+ 215us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":557,"thread_start_us":382,"threads_started":1}
I20260809 12:16:02.883239 25818 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.222.61" port: 42263 } has no permanent_uuid. Determining permanent_uuid...
I20260809 12:16:02.889061 25652 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P d343a8cfeb614de282f065a5966ca6cd: Bootstrap starting.
Aug 09 12:16:02 dist-test-slave-2w3w krb5kdc[25563](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786277761, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.222.62@KRBTEST.COM
I20260809 12:16:02.893868 25652 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P d343a8cfeb614de282f065a5966ca6cd: Neither blocks nor log segments found. Creating new log.
I20260809 12:16:02.895593 25652 log.cc:824] T 00000000000000000000000000000000 P d343a8cfeb614de282f065a5966ca6cd: Log is configured to *not* fsync() on all Append() calls
Aug 09 12:16:02 dist-test-slave-2w3w krb5kdc[25563](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786277762, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.222.60@KRBTEST.COM for kudu/127.0.222.61@KRBTEST.COM
I20260809 12:16:02.900061 25652 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P d343a8cfeb614de282f065a5966ca6cd: No bootstrap required, opened a new log
I20260809 12:16:02.901899 25822 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 12:16:02.882482 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59342 (local address 127.0.222.62:39441)
0809 12:16:02.883156 (+ 674us) server_negotiation.cc:206] Beginning negotiation
0809 12:16:02.883161 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0809 12:16:02.883180 (+ 19us) server_negotiation.cc:407] Connection header received
0809 12:16:02.883230 (+ 50us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 12:16:02.883236 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 12:16:02.883308 (+ 72us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 12:16:02.883410 (+ 102us) server_negotiation.cc:226] Negotiated authn=SASL
0809 12:16:02.885168 (+ 1758us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 12:16:02.886017 (+ 849us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 12:16:02.887910 (+ 1893us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 12:16:02.888316 (+ 406us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 12:16:02.888408 (+ 92us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 12:16:02.894639 (+ 6231us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 12:16:02.894646 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 12:16:02.894652 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0809 12:16:02.894696 (+ 44us) server_negotiation.cc:922] calling sasl_server_start()
0809 12:16:02.898239 (+ 3543us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 12:16:02.899186 (+ 947us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 12:16:02.899191 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 12:16:02.899194 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0809 12:16:02.899373 (+ 179us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 12:16:02.899873 (+ 500us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 12:16:02.899880 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 12:16:02.899884 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0809 12:16:02.900320 (+ 436us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 12:16:02.900691 (+ 371us) server_negotiation.cc:1035] Waiting for connection context
0809 12:16:02.901057 (+ 366us) server_negotiation.cc:299] Negotiation successful
0809 12:16:02.901325 (+ 268us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":549}
I20260809 12:16:02.904560 25824 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 12:16:02.885470 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.222.61:42263 (local address 127.0.0.1:47528)
0809 12:16:02.888652 (+ 3182us) negotiation.cc:107] Waiting for socket to connect
0809 12:16:02.888667 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0809 12:16:02.888792 (+ 125us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 12:16:02.890911 (+ 2119us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 12:16:02.890915 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 12:16:02.891312 (+ 397us) client_negotiation.cc:189] Negotiated authn=SASL
0809 12:16:02.891452 (+ 140us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 12:16:02.891460 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 12:16:02.892431 (+ 971us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 12:16:02.892436 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 12:16:02.893305 (+ 869us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 12:16:02.893317 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 12:16:02.894046 (+ 729us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 12:16:02.894052 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 12:16:02.894236 (+ 184us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 12:16:02.894863 (+ 627us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0809 12:16:02.894892 (+ 29us) client_negotiation.cc:656] calling sasl_client_start()
0809 12:16:02.898060 (+ 3168us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0809 12:16:02.901791 (+ 3731us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0809 12:16:02.901796 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0809 12:16:02.901800 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0809 12:16:02.902340 (+ 540us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0809 12:16:02.902754 (+ 414us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0809 12:16:02.902759 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0809 12:16:02.902763 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0809 12:16:02.902909 (+ 146us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0809 12:16:02.903589 (+ 680us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0809 12:16:02.903593 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0809 12:16:02.903751 (+ 158us) client_negotiation.cc:769] Sending connection context
0809 12:16:02.903886 (+ 135us) client_negotiation.cc:240] Negotiation successful
0809 12:16:02.904188 (+ 302us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":3080}
I20260809 12:16:02.904886 25737 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 12:16:02.885752 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47528 (local address 127.0.222.61:42263)
0809 12:16:02.887164 (+ 1412us) server_negotiation.cc:206] Beginning negotiation
0809 12:16:02.887171 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0809 12:16:02.888745 (+ 1574us) server_negotiation.cc:407] Connection header received
0809 12:16:02.888927 (+ 182us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 12:16:02.888931 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 12:16:02.889043 (+ 112us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 12:16:02.889146 (+ 103us) server_negotiation.cc:226] Negotiated authn=SASL
0809 12:16:02.891605 (+ 2459us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 12:16:02.892275 (+ 670us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 12:16:02.893452 (+ 1177us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 12:16:02.893878 (+ 426us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 12:16:02.894017 (+ 139us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 12:16:02.898279 (+ 4262us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 12:16:02.898289 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 12:16:02.898296 (+ 7us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0809 12:16:02.898343 (+ 47us) server_negotiation.cc:922] calling sasl_server_start()
0809 12:16:02.901653 (+ 3310us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 12:16:02.902469 (+ 816us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 12:16:02.902486 (+ 17us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 12:16:02.902488 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0809 12:16:02.902637 (+ 149us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 12:16:02.903083 (+ 446us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 12:16:02.903093 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 12:16:02.903095 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0809 12:16:02.903451 (+ 356us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 12:16:02.903624 (+ 173us) server_negotiation.cc:1035] Waiting for connection context
0809 12:16:02.904016 (+ 392us) server_negotiation.cc:299] Negotiation successful
0809 12:16:02.904245 (+ 229us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1266}
I20260809 12:16:02.908464 25818 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.222.60" port: 37589 } has no permanent_uuid. Determining permanent_uuid...
I20260809 12:16:02.913246 25652 raft_consensus.cc:348] T 00000000000000000000000000000000 P d343a8cfeb614de282f065a5966ca6cd [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d343a8cfeb614de282f065a5966ca6cd" member_type: VOTER last_known_addr { host: "127.0.222.62" port: 39441 } } peers { permanent_uuid: "39e4d93ef8de492bae8bf9b7677b0128" member_type: VOTER last_known_addr { host: "127.0.222.61" port: 42263 } } peers { permanent_uuid: "865df34449d0455398e47baaff333baf" member_type: VOTER last_known_addr { host: "127.0.222.60" port: 37589 } }
I20260809 12:16:02.913690 25652 raft_consensus.cc:729] T 00000000000000000000000000000000 P d343a8cfeb614de282f065a5966ca6cd [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d343a8cfeb614de282f065a5966ca6cd, State: Initialized, Role: FOLLOWER
I20260809 12:16:02.914299 25652 consensus_queue.cc:260] T 00000000000000000000000000000000 P d343a8cfeb614de282f065a5966ca6cd [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d343a8cfeb614de282f065a5966ca6cd" member_type: VOTER last_known_addr { host: "127.0.222.62" port: 39441 } } peers { permanent_uuid: "39e4d93ef8de492bae8bf9b7677b0128" member_type: VOTER last_known_addr { host: "127.0.222.61" port: 42263 } } peers { permanent_uuid: "865df34449d0455398e47baaff333baf" member_type: VOTER last_known_addr { host: "127.0.222.60" port: 37589 } }
Aug 09 12:16:02 dist-test-slave-2w3w krb5kdc[25563](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786277761, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.222.61@KRBTEST.COM
I20260809 12:16:02.915475 25829 sys_catalog.cc:455] T 00000000000000000000000000000000 P d343a8cfeb614de282f065a5966ca6cd [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d343a8cfeb614de282f065a5966ca6cd" member_type: VOTER last_known_addr { host: "127.0.222.62" port: 39441 } } peers { permanent_uuid: "39e4d93ef8de492bae8bf9b7677b0128" member_type: VOTER last_known_addr { host: "127.0.222.61" port: 42263 } } peers { permanent_uuid: "865df34449d0455398e47baaff333baf" member_type: VOTER last_known_addr { host: "127.0.222.60" port: 37589 } } }
I20260809 12:16:02.916388 25829 sys_catalog.cc:458] T 00000000000000000000000000000000 P d343a8cfeb614de282f065a5966ca6cd [sys.catalog]: This master's current role is: FOLLOWER
I20260809 12:16:02.917068 25652 sys_catalog.cc:564] T 00000000000000000000000000000000 P d343a8cfeb614de282f065a5966ca6cd [sys.catalog]: configured and running, proceeding with master startup.
Aug 09 12:16:02 dist-test-slave-2w3w krb5kdc[25563](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786277762, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.222.60@KRBTEST.COM for kudu/127.0.222.60@KRBTEST.COM
I20260809 12:16:02.925632 25737 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 12:16:02.904172 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47532 (local address 127.0.222.61:42263)
0809 12:16:02.905129 (+ 957us) server_negotiation.cc:206] Beginning negotiation
0809 12:16:02.905136 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0809 12:16:02.907233 (+ 2097us) server_negotiation.cc:407] Connection header received
0809 12:16:02.907400 (+ 167us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 12:16:02.907407 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 12:16:02.907502 (+ 95us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 12:16:02.907610 (+ 108us) server_negotiation.cc:226] Negotiated authn=SASL
0809 12:16:02.908954 (+ 1344us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 12:16:02.910079 (+ 1125us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 12:16:02.911674 (+ 1595us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 12:16:02.912066 (+ 392us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 12:16:02.912194 (+ 128us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 12:16:02.917164 (+ 4970us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 12:16:02.917172 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 12:16:02.917177 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0809 12:16:02.917216 (+ 39us) server_negotiation.cc:922] calling sasl_server_start()
0809 12:16:02.921256 (+ 4040us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 12:16:02.922000 (+ 744us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 12:16:02.922006 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 12:16:02.922010 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0809 12:16:02.922176 (+ 166us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 12:16:02.922707 (+ 531us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 12:16:02.922712 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 12:16:02.922716 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0809 12:16:02.923157 (+ 441us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 12:16:02.924775 (+ 1618us) server_negotiation.cc:1035] Waiting for connection context
0809 12:16:02.924927 (+ 152us) server_negotiation.cc:299] Negotiation successful
0809 12:16:02.925184 (+ 257us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":857,"thread_start_us":3343,"threads_started":1}
I20260809 12:16:02.925675 25824 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 12:16:02.909364 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.222.60:37589 (local address 127.0.0.1:40432)
0809 12:16:02.909546 (+ 182us) negotiation.cc:107] Waiting for socket to connect
0809 12:16:02.909565 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0809 12:16:02.909791 (+ 226us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 12:16:02.910181 (+ 390us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 12:16:02.910187 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 12:16:02.910670 (+ 483us) client_negotiation.cc:189] Negotiated authn=SASL
0809 12:16:02.910880 (+ 210us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 12:16:02.910893 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 12:16:02.912417 (+ 1524us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 12:16:02.912421 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 12:16:02.913843 (+ 1422us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 12:16:02.913861 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 12:16:02.915198 (+ 1337us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 12:16:02.915205 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 12:16:02.915414 (+ 209us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 12:16:02.916024 (+ 610us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0809 12:16:02.916061 (+ 37us) client_negotiation.cc:656] calling sasl_client_start()
0809 12:16:02.918779 (+ 2718us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0809 12:16:02.922485 (+ 3706us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0809 12:16:02.922490 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0809 12:16:02.922494 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0809 12:16:02.923140 (+ 646us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0809 12:16:02.923612 (+ 472us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0809 12:16:02.923618 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0809 12:16:02.923622 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0809 12:16:02.923798 (+ 176us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0809 12:16:02.924568 (+ 770us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0809 12:16:02.924572 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0809 12:16:02.924741 (+ 169us) client_negotiation.cc:769] Sending connection context
0809 12:16:02.924891 (+ 150us) client_negotiation.cc:240] Negotiation successful
0809 12:16:02.925225 (+ 334us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":57}
I20260809 12:16:02.925712 25825 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 12:16:02.909610 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40432 (local address 127.0.222.60:37589)
0809 12:16:02.909737 (+ 127us) server_negotiation.cc:206] Beginning negotiation
0809 12:16:02.909742 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0809 12:16:02.909757 (+ 15us) server_negotiation.cc:407] Connection header received
0809 12:16:02.909938 (+ 181us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 12:16:02.909945 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 12:16:02.910035 (+ 90us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 12:16:02.910139 (+ 104us) server_negotiation.cc:226] Negotiated authn=SASL
0809 12:16:02.911068 (+ 929us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 12:16:02.912236 (+ 1168us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 12:16:02.914009 (+ 1773us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 12:16:02.914736 (+ 727us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 12:16:02.914883 (+ 147us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 12:16:02.919039 (+ 4156us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 12:16:02.919050 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 12:16:02.919059 (+ 9us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0809 12:16:02.919105 (+ 46us) server_negotiation.cc:922] calling sasl_server_start()
0809 12:16:02.922355 (+ 3250us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 12:16:02.923289 (+ 934us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 12:16:02.923297 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 12:16:02.923304 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0809 12:16:02.923479 (+ 175us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 12:16:02.923939 (+ 460us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 12:16:02.923944 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 12:16:02.923947 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0809 12:16:02.924371 (+ 424us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 12:16:02.924518 (+ 147us) server_negotiation.cc:1035] Waiting for connection context
0809 12:16:02.925030 (+ 512us) server_negotiation.cc:299] Negotiation successful
0809 12:16:02.925354 (+ 324us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":127,"server-negotiator.queue_time_us":35}
I20260809 12:16:02.931622 888 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260809 12:16:02.932083 25818 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 865df34449d0455398e47baaff333baf: Bootstrap starting.
I20260809 12:16:02.935133 25625 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "00000000000000000000000000000000"
dest_uuid: "d343a8cfeb614de282f065a5966ca6cd"
from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:59342
I20260809 12:16:02.935500 25625 raft_consensus.cc:482] T 00000000000000000000000000000000 P d343a8cfeb614de282f065a5966ca6cd [term 0 FOLLOWER]: Starting forced leader election (received explicit request)
I20260809 12:16:02.935711 25625 raft_consensus.cc:3037] T 00000000000000000000000000000000 P d343a8cfeb614de282f065a5966ca6cd [term 0 FOLLOWER]: Advancing to term 1
I20260809 12:16:02.937147 25818 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 865df34449d0455398e47baaff333baf: Neither blocks nor log segments found. Creating new log.
I20260809 12:16:02.938357 25625 raft_consensus.cc:504] T 00000000000000000000000000000000 P d343a8cfeb614de282f065a5966ca6cd [term 1 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d343a8cfeb614de282f065a5966ca6cd" member_type: VOTER last_known_addr { host: "127.0.222.62" port: 39441 } } peers { permanent_uuid: "39e4d93ef8de492bae8bf9b7677b0128" member_type: VOTER last_known_addr { host: "127.0.222.61" port: 42263 } } peers { permanent_uuid: "865df34449d0455398e47baaff333baf" member_type: VOTER last_known_addr { host: "127.0.222.60" port: 37589 } }
I20260809 12:16:02.938823 25818 log.cc:824] T 00000000000000000000000000000000 P 865df34449d0455398e47baaff333baf: Log is configured to *not* fsync() on all Append() calls
I20260809 12:16:02.940827 25625 leader_election.cc:288] T 00000000000000000000000000000000 P d343a8cfeb614de282f065a5966ca6cd [CANDIDATE]: Term 1 election: Requested vote from peers 39e4d93ef8de492bae8bf9b7677b0128 (127.0.222.61:42263), 865df34449d0455398e47baaff333baf (127.0.222.60:37589)
I20260809 12:16:02.942725 25818 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 865df34449d0455398e47baaff333baf: No bootstrap required, opened a new log
I20260809 12:16:02.929286 25577 master_runner.cc:186] Error getting master registration for 127.0.222.61:42263: OK, instance_id { permanent_uuid: "39e4d93ef8de492bae8bf9b7677b0128" instance_seqno: 1786277762283682 } error { code: CATALOG_MANAGER_NOT_INITIALIZED status { code: SERVICE_UNAVAILABLE message: "Catalog manager is not initialized. State: Starting" } }
I20260809 12:16:02.947745 25577 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
W20260809 12:16:02.950773 25841 catalog_manager.cc:1558] T 00000000000000000000000000000000 P d343a8cfeb614de282f065a5966ca6cd: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260809 12:16:02.950968 25841 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260809 12:16:02.955453 25818 raft_consensus.cc:348] T 00000000000000000000000000000000 P 865df34449d0455398e47baaff333baf [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d343a8cfeb614de282f065a5966ca6cd" member_type: VOTER last_known_addr { host: "127.0.222.62" port: 39441 } } peers { permanent_uuid: "39e4d93ef8de492bae8bf9b7677b0128" member_type: VOTER last_known_addr { host: "127.0.222.61" port: 42263 } } peers { permanent_uuid: "865df34449d0455398e47baaff333baf" member_type: VOTER last_known_addr { host: "127.0.222.60" port: 37589 } }
I20260809 12:16:02.955988 25818 raft_consensus.cc:729] T 00000000000000000000000000000000 P 865df34449d0455398e47baaff333baf [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 865df34449d0455398e47baaff333baf, State: Initialized, Role: FOLLOWER
I20260809 12:16:02.956601 25818 consensus_queue.cc:260] T 00000000000000000000000000000000 P 865df34449d0455398e47baaff333baf [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d343a8cfeb614de282f065a5966ca6cd" member_type: VOTER last_known_addr { host: "127.0.222.62" port: 39441 } } peers { permanent_uuid: "39e4d93ef8de492bae8bf9b7677b0128" member_type: VOTER last_known_addr { host: "127.0.222.61" port: 42263 } } peers { permanent_uuid: "865df34449d0455398e47baaff333baf" member_type: VOTER last_known_addr { host: "127.0.222.60" port: 37589 } }
W20260809 12:16:02.959695 25602 tablet.cc:2369] T 00000000000000000000000000000000 P d343a8cfeb614de282f065a5966ca6cd: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260809 12:16:02.970867 25851 sys_catalog.cc:455] T 00000000000000000000000000000000 P 865df34449d0455398e47baaff333baf [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d343a8cfeb614de282f065a5966ca6cd" member_type: VOTER last_known_addr { host: "127.0.222.62" port: 39441 } } peers { permanent_uuid: "39e4d93ef8de492bae8bf9b7677b0128" member_type: VOTER last_known_addr { host: "127.0.222.61" port: 42263 } } peers { permanent_uuid: "865df34449d0455398e47baaff333baf" member_type: VOTER last_known_addr { host: "127.0.222.60" port: 37589 } } }
I20260809 12:16:02.971383 25851 sys_catalog.cc:458] T 00000000000000000000000000000000 P 865df34449d0455398e47baaff333baf [sys.catalog]: This master's current role is: FOLLOWER
I20260809 12:16:02.972762 25818 sys_catalog.cc:564] T 00000000000000000000000000000000 P 865df34449d0455398e47baaff333baf [sys.catalog]: configured and running, proceeding with master startup.
I20260809 12:16:02.972822 25790 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "d343a8cfeb614de282f065a5966ca6cd" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: true dest_uuid: "865df34449d0455398e47baaff333baf"
I20260809 12:16:02.973465 25790 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 865df34449d0455398e47baaff333baf [term 0 FOLLOWER]: Advancing to term 1
I20260809 12:16:02.977342 25790 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 865df34449d0455398e47baaff333baf [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d343a8cfeb614de282f065a5966ca6cd in term 1.
I20260809 12:16:02.978278 25597 leader_election.cc:302] T 00000000000000000000000000000000 P d343a8cfeb614de282f065a5966ca6cd [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 865df34449d0455398e47baaff333baf, d343a8cfeb614de282f065a5966ca6cd; no voters:
I20260809 12:16:02.979035 25829 raft_consensus.cc:2781] T 00000000000000000000000000000000 P d343a8cfeb614de282f065a5966ca6cd [term 1 FOLLOWER]: Leader election won for term 1
I20260809 12:16:02.991356 25825 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 12:16:02.962498 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40446 (local address 127.0.222.60:37589)
0809 12:16:02.962681 (+ 183us) server_negotiation.cc:206] Beginning negotiation
0809 12:16:02.962689 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0809 12:16:02.971023 (+ 8334us) server_negotiation.cc:407] Connection header received
0809 12:16:02.972897 (+ 1874us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 12:16:02.972902 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 12:16:02.973033 (+ 131us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 12:16:02.973128 (+ 95us) server_negotiation.cc:226] Negotiated authn=SASL
0809 12:16:02.975465 (+ 2337us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 12:16:02.976554 (+ 1089us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 12:16:02.978498 (+ 1944us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 12:16:02.979271 (+ 773us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 12:16:02.979365 (+ 94us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 12:16:02.983642 (+ 4277us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 12:16:02.983652 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 12:16:02.983658 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0809 12:16:02.983707 (+ 49us) server_negotiation.cc:922] calling sasl_server_start()
0809 12:16:02.987773 (+ 4066us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 12:16:02.988658 (+ 885us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 12:16:02.988666 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 12:16:02.988672 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0809 12:16:02.988848 (+ 176us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 12:16:02.989325 (+ 477us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 12:16:02.989331 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 12:16:02.989333 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0809 12:16:02.989715 (+ 382us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 12:16:02.990028 (+ 313us) server_negotiation.cc:1035] Waiting for connection context
0809 12:16:02.990646 (+ 618us) server_negotiation.cc:299] Negotiation successful
0809 12:16:02.990906 (+ 260us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":63}
I20260809 12:16:02.997609 25822 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 12:16:02.954912 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59356 (local address 127.0.222.62:39441)
0809 12:16:02.955143 (+ 231us) server_negotiation.cc:206] Beginning negotiation
0809 12:16:02.955149 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0809 12:16:02.956948 (+ 1799us) server_negotiation.cc:407] Connection header received
0809 12:16:02.957575 (+ 627us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 12:16:02.957585 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 12:16:02.957700 (+ 115us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 12:16:02.957795 (+ 95us) server_negotiation.cc:226] Negotiated authn=SASL
0809 12:16:02.966928 (+ 9133us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 12:16:02.968361 (+ 1433us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 12:16:02.979302 (+ 10941us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 12:16:02.980054 (+ 752us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 12:16:02.980199 (+ 145us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 12:16:02.985036 (+ 4837us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 12:16:02.985044 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 12:16:02.985052 (+ 8us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0809 12:16:02.985103 (+ 51us) server_negotiation.cc:922] calling sasl_server_start()
0809 12:16:02.989186 (+ 4083us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 12:16:02.990505 (+ 1319us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 12:16:02.990511 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 12:16:02.990514 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0809 12:16:02.990683 (+ 169us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 12:16:02.991436 (+ 753us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 12:16:02.991441 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 12:16:02.991443 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0809 12:16:02.991835 (+ 392us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 12:16:02.992114 (+ 279us) server_negotiation.cc:1035] Waiting for connection context
0809 12:16:02.993186 (+ 1072us) server_negotiation.cc:299] Negotiation successful
0809 12:16:02.993642 (+ 456us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":99}
I20260809 12:16:02.999595 25829 raft_consensus.cc:686] T 00000000000000000000000000000000 P d343a8cfeb614de282f065a5966ca6cd [term 1 LEADER]: Becoming Leader. State: Replica: d343a8cfeb614de282f065a5966ca6cd, State: Running, Role: LEADER
I20260809 12:16:03.000535 25829 consensus_queue.cc:237] T 00000000000000000000000000000000 P d343a8cfeb614de282f065a5966ca6cd [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d343a8cfeb614de282f065a5966ca6cd" member_type: VOTER last_known_addr { host: "127.0.222.62" port: 39441 } } peers { permanent_uuid: "39e4d93ef8de492bae8bf9b7677b0128" member_type: VOTER last_known_addr { host: "127.0.222.61" port: 42263 } } peers { permanent_uuid: "865df34449d0455398e47baaff333baf" member_type: VOTER last_known_addr { host: "127.0.222.60" port: 37589 } }
I20260809 12:16:03.002518 25737 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 12:16:02.955194 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47538 (local address 127.0.222.61:42263)
0809 12:16:02.955376 (+ 182us) server_negotiation.cc:206] Beginning negotiation
0809 12:16:02.955383 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0809 12:16:02.967813 (+ 12430us) server_negotiation.cc:407] Connection header received
0809 12:16:02.968178 (+ 365us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 12:16:02.968189 (+ 11us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 12:16:02.968313 (+ 124us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 12:16:02.968419 (+ 106us) server_negotiation.cc:226] Negotiated authn=SASL
0809 12:16:02.972888 (+ 4469us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 12:16:02.974103 (+ 1215us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 12:16:02.985047 (+ 10944us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 12:16:02.986065 (+ 1018us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 12:16:02.986206 (+ 141us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 12:16:02.995111 (+ 8905us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 12:16:02.995118 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 12:16:02.995123 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0809 12:16:02.995167 (+ 44us) server_negotiation.cc:922] calling sasl_server_start()
0809 12:16:02.998842 (+ 3675us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 12:16:02.999716 (+ 874us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 12:16:02.999723 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 12:16:02.999726 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0809 12:16:02.999910 (+ 184us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 12:16:03.000371 (+ 461us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 12:16:03.000376 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 12:16:03.000379 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0809 12:16:03.000767 (+ 388us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 12:16:03.001003 (+ 236us) server_negotiation.cc:1035] Waiting for connection context
0809 12:16:03.001602 (+ 599us) server_negotiation.cc:299] Negotiation successful
0809 12:16:03.002159 (+ 557us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":66}
I20260809 12:16:03.010537 25732 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.222.60" port: 37589 } attempt: 5
I20260809 12:16:03.019222 25854 sys_catalog.cc:455] T 00000000000000000000000000000000 P d343a8cfeb614de282f065a5966ca6cd [sys.catalog]: SysCatalogTable state changed. Reason: New leader d343a8cfeb614de282f065a5966ca6cd. Latest consensus state: current_term: 1 leader_uuid: "d343a8cfeb614de282f065a5966ca6cd" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d343a8cfeb614de282f065a5966ca6cd" member_type: VOTER last_known_addr { host: "127.0.222.62" port: 39441 } } peers { permanent_uuid: "39e4d93ef8de492bae8bf9b7677b0128" member_type: VOTER last_known_addr { host: "127.0.222.61" port: 42263 } } peers { permanent_uuid: "865df34449d0455398e47baaff333baf" member_type: VOTER last_known_addr { host: "127.0.222.60" port: 37589 } } }
I20260809 12:16:03.024802 25854 sys_catalog.cc:458] T 00000000000000000000000000000000 P d343a8cfeb614de282f065a5966ca6cd [sys.catalog]: This master's current role is: LEADER
Aug 09 12:16:03 dist-test-slave-2w3w krb5kdc[25563](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786277762, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.222.61@KRBTEST.COM for kudu/127.0.222.60@KRBTEST.COM
I20260809 12:16:03.037246 25857 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260809 12:16:03.039759 25735 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 12:16:03.011389 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.222.60:37589 (local address 127.0.0.1:40448)
0809 12:16:03.011772 (+ 383us) negotiation.cc:107] Waiting for socket to connect
0809 12:16:03.011798 (+ 26us) client_negotiation.cc:174] Beginning negotiation
0809 12:16:03.011961 (+ 163us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 12:16:03.012357 (+ 396us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 12:16:03.012365 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 12:16:03.013134 (+ 769us) client_negotiation.cc:189] Negotiated authn=SASL
0809 12:16:03.013452 (+ 318us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 12:16:03.013471 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 12:16:03.016095 (+ 2624us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 12:16:03.016105 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 12:16:03.018235 (+ 2130us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 12:16:03.018259 (+ 24us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 12:16:03.019455 (+ 1196us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 12:16:03.019465 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 12:16:03.019859 (+ 394us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 12:16:03.020745 (+ 886us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0809 12:16:03.020795 (+ 50us) client_negotiation.cc:656] calling sasl_client_start()
0809 12:16:03.026306 (+ 5511us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0809 12:16:03.033319 (+ 7013us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0809 12:16:03.033327 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0809 12:16:03.033334 (+ 7us) client_negotiation.cc:760] calling sasl_client_step()
0809 12:16:03.034219 (+ 885us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0809 12:16:03.034872 (+ 653us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0809 12:16:03.034881 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0809 12:16:03.034888 (+ 7us) client_negotiation.cc:760] calling sasl_client_step()
0809 12:16:03.035162 (+ 274us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0809 12:16:03.038663 (+ 3501us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0809 12:16:03.038668 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0809 12:16:03.038857 (+ 189us) client_negotiation.cc:769] Sending connection context
0809 12:16:03.039040 (+ 183us) client_negotiation.cc:240] Negotiation successful
0809 12:16:03.039348 (+ 308us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":186}
I20260809 12:16:03.041743 25857 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260809 12:16:03.041955 25825 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 12:16:03.011452 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40448 (local address 127.0.222.60:37589)
0809 12:16:03.011663 (+ 211us) server_negotiation.cc:206] Beginning negotiation
0809 12:16:03.011669 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0809 12:16:03.011876 (+ 207us) server_negotiation.cc:407] Connection header received
0809 12:16:03.012103 (+ 227us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 12:16:03.012109 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 12:16:03.012197 (+ 88us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 12:16:03.012318 (+ 121us) server_negotiation.cc:226] Negotiated authn=SASL
0809 12:16:03.013643 (+ 1325us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 12:16:03.014868 (+ 1225us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 12:16:03.018425 (+ 3557us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 12:16:03.019259 (+ 834us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 12:16:03.019410 (+ 151us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 12:16:03.026622 (+ 7212us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 12:16:03.026634 (+ 12us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 12:16:03.026642 (+ 8us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0809 12:16:03.026704 (+ 62us) server_negotiation.cc:922] calling sasl_server_start()
0809 12:16:03.033124 (+ 6420us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 12:16:03.034386 (+ 1262us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 12:16:03.034397 (+ 11us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 12:16:03.034404 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0809 12:16:03.034640 (+ 236us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 12:16:03.035333 (+ 693us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 12:16:03.035340 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 12:16:03.035344 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0809 12:16:03.038351 (+ 3007us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 12:16:03.040533 (+ 2182us) server_negotiation.cc:1035] Waiting for connection context
0809 12:16:03.040773 (+ 240us) server_negotiation.cc:299] Negotiation successful
0809 12:16:03.041199 (+ 426us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":84}
I20260809 12:16:03.058694 25732 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 39e4d93ef8de492bae8bf9b7677b0128: Bootstrap starting.
I20260809 12:16:03.067718 25732 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 39e4d93ef8de492bae8bf9b7677b0128: Neither blocks nor log segments found. Creating new log.
I20260809 12:16:03.069386 25732 log.cc:824] T 00000000000000000000000000000000 P 39e4d93ef8de492bae8bf9b7677b0128: Log is configured to *not* fsync() on all Append() calls
I20260809 12:16:03.073935 25732 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 39e4d93ef8de492bae8bf9b7677b0128: No bootstrap required, opened a new log
W20260809 12:16:03.075098 25765 tablet.cc:2369] T 00000000000000000000000000000000 P 865df34449d0455398e47baaff333baf: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260809 12:16:03.088238 25790 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 865df34449d0455398e47baaff333baf [term 1 FOLLOWER]: Refusing update from remote peer d343a8cfeb614de282f065a5966ca6cd: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260809 12:16:03.089572 25854 consensus_queue.cc:1035] T 00000000000000000000000000000000 P d343a8cfeb614de282f065a5966ca6cd [LEADER]: Connected to new peer: Peer: permanent_uuid: "865df34449d0455398e47baaff333baf" member_type: VOTER last_known_addr { host: "127.0.222.60" port: 37589 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260809 12:16:03.093000 25732 raft_consensus.cc:348] T 00000000000000000000000000000000 P 39e4d93ef8de492bae8bf9b7677b0128 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d343a8cfeb614de282f065a5966ca6cd" member_type: VOTER last_known_addr { host: "127.0.222.62" port: 39441 } } peers { permanent_uuid: "39e4d93ef8de492bae8bf9b7677b0128" member_type: VOTER last_known_addr { host: "127.0.222.61" port: 42263 } } peers { permanent_uuid: "865df34449d0455398e47baaff333baf" member_type: VOTER last_known_addr { host: "127.0.222.60" port: 37589 } }
I20260809 12:16:03.093541 25732 raft_consensus.cc:729] T 00000000000000000000000000000000 P 39e4d93ef8de492bae8bf9b7677b0128 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 39e4d93ef8de492bae8bf9b7677b0128, State: Initialized, Role: FOLLOWER
I20260809 12:16:03.094164 25732 consensus_queue.cc:260] T 00000000000000000000000000000000 P 39e4d93ef8de492bae8bf9b7677b0128 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d343a8cfeb614de282f065a5966ca6cd" member_type: VOTER last_known_addr { host: "127.0.222.62" port: 39441 } } peers { permanent_uuid: "39e4d93ef8de492bae8bf9b7677b0128" member_type: VOTER last_known_addr { host: "127.0.222.61" port: 42263 } } peers { permanent_uuid: "865df34449d0455398e47baaff333baf" member_type: VOTER last_known_addr { host: "127.0.222.60" port: 37589 } }
I20260809 12:16:03.115650 25863 sys_catalog.cc:455] T 00000000000000000000000000000000 P 39e4d93ef8de492bae8bf9b7677b0128 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d343a8cfeb614de282f065a5966ca6cd" member_type: VOTER last_known_addr { host: "127.0.222.62" port: 39441 } } peers { permanent_uuid: "39e4d93ef8de492bae8bf9b7677b0128" member_type: VOTER last_known_addr { host: "127.0.222.61" port: 42263 } } peers { permanent_uuid: "865df34449d0455398e47baaff333baf" member_type: VOTER last_known_addr { host: "127.0.222.60" port: 37589 } } }
I20260809 12:16:03.116216 25863 sys_catalog.cc:458] T 00000000000000000000000000000000 P 39e4d93ef8de492bae8bf9b7677b0128 [sys.catalog]: This master's current role is: FOLLOWER
I20260809 12:16:03.118842 25705 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "d343a8cfeb614de282f065a5966ca6cd" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: true dest_uuid: "39e4d93ef8de492bae8bf9b7677b0128"
I20260809 12:16:03.119237 25704 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 39e4d93ef8de492bae8bf9b7677b0128 [term 0 FOLLOWER]: Advancing to term 1
I20260809 12:16:03.123605 25704 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 39e4d93ef8de492bae8bf9b7677b0128 [term 1 FOLLOWER]: Refusing update from remote peer d343a8cfeb614de282f065a5966ca6cd: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260809 12:16:03.124740 25854 consensus_queue.cc:1035] T 00000000000000000000000000000000 P d343a8cfeb614de282f065a5966ca6cd [LEADER]: Connected to new peer: Peer: permanent_uuid: "39e4d93ef8de492bae8bf9b7677b0128" member_type: VOTER last_known_addr { host: "127.0.222.61" port: 42263 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260809 12:16:03.128655 25851 sys_catalog.cc:455] T 00000000000000000000000000000000 P 865df34449d0455398e47baaff333baf [sys.catalog]: SysCatalogTable state changed. Reason: New leader d343a8cfeb614de282f065a5966ca6cd. Latest consensus state: current_term: 1 leader_uuid: "d343a8cfeb614de282f065a5966ca6cd" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d343a8cfeb614de282f065a5966ca6cd" member_type: VOTER last_known_addr { host: "127.0.222.62" port: 39441 } } peers { permanent_uuid: "39e4d93ef8de492bae8bf9b7677b0128" member_type: VOTER last_known_addr { host: "127.0.222.61" port: 42263 } } peers { permanent_uuid: "865df34449d0455398e47baaff333baf" member_type: VOTER last_known_addr { host: "127.0.222.60" port: 37589 } } }
I20260809 12:16:03.129119 25851 sys_catalog.cc:458] T 00000000000000000000000000000000 P 865df34449d0455398e47baaff333baf [sys.catalog]: This master's current role is: FOLLOWER
I20260809 12:16:03.129714 25705 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 39e4d93ef8de492bae8bf9b7677b0128 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d343a8cfeb614de282f065a5966ca6cd in term 1.
I20260809 12:16:03.136832 25732 sys_catalog.cc:564] T 00000000000000000000000000000000 P 39e4d93ef8de492bae8bf9b7677b0128 [sys.catalog]: configured and running, proceeding with master startup.
I20260809 12:16:03.137641 25860 mvcc.cc:205] Tried to move back new op lower bound from 7316593717596561408 to 7316593717258285056. Current Snapshot: MvccSnapshot[applied={T|T < 7316593717596561408}]
I20260809 12:16:03.145545 25829 sys_catalog.cc:455] T 00000000000000000000000000000000 P d343a8cfeb614de282f065a5966ca6cd [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "d343a8cfeb614de282f065a5966ca6cd" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d343a8cfeb614de282f065a5966ca6cd" member_type: VOTER last_known_addr { host: "127.0.222.62" port: 39441 } } peers { permanent_uuid: "39e4d93ef8de492bae8bf9b7677b0128" member_type: VOTER last_known_addr { host: "127.0.222.61" port: 42263 } } peers { permanent_uuid: "865df34449d0455398e47baaff333baf" member_type: VOTER last_known_addr { host: "127.0.222.60" port: 37589 } } }
I20260809 12:16:03.146001 25829 sys_catalog.cc:458] T 00000000000000000000000000000000 P d343a8cfeb614de282f065a5966ca6cd [sys.catalog]: This master's current role is: LEADER
I20260809 12:16:03.150321 25865 mvcc.cc:205] Tried to move back new op lower bound from 7316593717596561408 to 7316593717258285056. Current Snapshot: MvccSnapshot[applied={T|T < 7316593717596561408}]
I20260809 12:16:03.155334 25857 catalog_manager.cc:1347] Generated new cluster ID: 6412f4377ffd4262871cb4998b2fd7bb
I20260809 12:16:03.155470 25857 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260809 12:16:03.159323 25851 sys_catalog.cc:455] T 00000000000000000000000000000000 P 865df34449d0455398e47baaff333baf [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "d343a8cfeb614de282f065a5966ca6cd" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d343a8cfeb614de282f065a5966ca6cd" member_type: VOTER last_known_addr { host: "127.0.222.62" port: 39441 } } peers { permanent_uuid: "39e4d93ef8de492bae8bf9b7677b0128" member_type: VOTER last_known_addr { host: "127.0.222.61" port: 42263 } } peers { permanent_uuid: "865df34449d0455398e47baaff333baf" member_type: VOTER last_known_addr { host: "127.0.222.60" port: 37589 } } }
I20260809 12:16:03.159597 25851 sys_catalog.cc:458] T 00000000000000000000000000000000 P 865df34449d0455398e47baaff333baf [sys.catalog]: This master's current role is: FOLLOWER
I20260809 12:16:03.160634 25863 sys_catalog.cc:455] T 00000000000000000000000000000000 P 39e4d93ef8de492bae8bf9b7677b0128 [sys.catalog]: SysCatalogTable state changed. Reason: New leader d343a8cfeb614de282f065a5966ca6cd. Latest consensus state: current_term: 1 leader_uuid: "d343a8cfeb614de282f065a5966ca6cd" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d343a8cfeb614de282f065a5966ca6cd" member_type: VOTER last_known_addr { host: "127.0.222.62" port: 39441 } } peers { permanent_uuid: "39e4d93ef8de492bae8bf9b7677b0128" member_type: VOTER last_known_addr { host: "127.0.222.61" port: 42263 } } peers { permanent_uuid: "865df34449d0455398e47baaff333baf" member_type: VOTER last_known_addr { host: "127.0.222.60" port: 37589 } } }
I20260809 12:16:03.160969 25863 sys_catalog.cc:458] T 00000000000000000000000000000000 P 39e4d93ef8de492bae8bf9b7677b0128 [sys.catalog]: This master's current role is: FOLLOWER
I20260809 12:16:03.164057 25854 sys_catalog.cc:455] T 00000000000000000000000000000000 P d343a8cfeb614de282f065a5966ca6cd [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "d343a8cfeb614de282f065a5966ca6cd" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d343a8cfeb614de282f065a5966ca6cd" member_type: VOTER last_known_addr { host: "127.0.222.62" port: 39441 } } peers { permanent_uuid: "39e4d93ef8de492bae8bf9b7677b0128" member_type: VOTER last_known_addr { host: "127.0.222.61" port: 42263 } } peers { permanent_uuid: "865df34449d0455398e47baaff333baf" member_type: VOTER last_known_addr { host: "127.0.222.60" port: 37589 } } }
I20260809 12:16:03.164898 25854 sys_catalog.cc:458] T 00000000000000000000000000000000 P d343a8cfeb614de282f065a5966ca6cd [sys.catalog]: This master's current role is: LEADER
I20260809 12:16:03.175083 25863 sys_catalog.cc:455] T 00000000000000000000000000000000 P 39e4d93ef8de492bae8bf9b7677b0128 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "d343a8cfeb614de282f065a5966ca6cd" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d343a8cfeb614de282f065a5966ca6cd" member_type: VOTER last_known_addr { host: "127.0.222.62" port: 39441 } } peers { permanent_uuid: "39e4d93ef8de492bae8bf9b7677b0128" member_type: VOTER last_known_addr { host: "127.0.222.61" port: 42263 } } peers { permanent_uuid: "865df34449d0455398e47baaff333baf" member_type: VOTER last_known_addr { host: "127.0.222.60" port: 37589 } } }
I20260809 12:16:03.175395 25863 sys_catalog.cc:458] T 00000000000000000000000000000000 P 39e4d93ef8de492bae8bf9b7677b0128 [sys.catalog]: This master's current role is: FOLLOWER
I20260809 12:16:03.178817 25867 mvcc.cc:205] Tried to move back new op lower bound from 7316593717596561408 to 7316593717258285056. Current Snapshot: MvccSnapshot[applied={T|T < 7316593717596561408}]
I20260809 12:16:03.193881 25857 catalog_manager.cc:1370] Generated new certificate authority record
I20260809 12:16:03.197408 25857 catalog_manager.cc:1504] Loading token signing keys...
I20260809 12:16:03.209924 25887 catalog_manager.cc:795] Waiting for catalog manager background task thread to start: Service unavailable: Catalog manager is not initialized. State: Starting
I20260809 12:16:03.219657 25857 catalog_manager.cc:5868] T 00000000000000000000000000000000 P d343a8cfeb614de282f065a5966ca6cd: Generated new TSK 0
I20260809 12:16:03.221174 25857 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260809 12:16:03.216732 25740 master_runner.cc:186] Error getting master registration for 127.0.222.61:42263: OK, instance_id { permanent_uuid: "39e4d93ef8de492bae8bf9b7677b0128" instance_seqno: 1786277762283682 } error { code: CATALOG_MANAGER_NOT_INITIALIZED status { code: SERVICE_UNAVAILABLE message: "Catalog manager is not initialized. State: Starting" } }
I20260809 12:16:03.230073 25740 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20260809 12:16:03.261829 25891 catalog_manager.cc:1259] Loaded cluster ID: 6412f4377ffd4262871cb4998b2fd7bb
I20260809 12:16:03.261960 25891 catalog_manager.cc:1552] T 00000000000000000000000000000000 P 39e4d93ef8de492bae8bf9b7677b0128: loading cluster ID for follower catalog manager: success
I20260809 12:16:03.266304 25891 catalog_manager.cc:1574] T 00000000000000000000000000000000 P 39e4d93ef8de492bae8bf9b7677b0128: acquiring CA information for follower catalog manager: success
I20260809 12:16:03.269207 25891 catalog_manager.cc:1602] T 00000000000000000000000000000000 P 39e4d93ef8de492bae8bf9b7677b0128: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
W20260809 12:16:03.341308 25825 negotiation.cc:337] Failed RPC negotiation. Trace:
0809 12:16:03.335226 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40464 (local address 127.0.222.60:37589)
0809 12:16:03.335404 (+ 178us) server_negotiation.cc:206] Beginning negotiation
0809 12:16:03.335410 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0809 12:16:03.339476 (+ 4066us) server_negotiation.cc:407] Connection header received
0809 12:16:03.339731 (+ 255us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 12:16:03.339736 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 12:16:03.339836 (+ 100us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 12:16:03.339912 (+ 76us) server_negotiation.cc:226] Negotiated authn=SASL
0809 12:16:03.341015 (+ 1103us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:40464: BlockingRecv error: recv got EOF from 127.0.0.1:40464 (error 108)
Metrics: {"server-negotiator.queue_time_us":66}
W20260809 12:16:03.341360 25737 negotiation.cc:337] Failed RPC negotiation. Trace:
0809 12:16:03.328589 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47548 (local address 127.0.222.61:42263)
0809 12:16:03.328806 (+ 217us) server_negotiation.cc:206] Beginning negotiation
0809 12:16:03.328812 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0809 12:16:03.337805 (+ 8993us) server_negotiation.cc:407] Connection header received
0809 12:16:03.339478 (+ 1673us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 12:16:03.339485 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 12:16:03.339578 (+ 93us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 12:16:03.339658 (+ 80us) server_negotiation.cc:226] Negotiated authn=SASL
0809 12:16:03.341117 (+ 1459us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:47548: BlockingRecv error: recv got EOF from 127.0.0.1:47548 (error 108)
Metrics: {"server-negotiator.queue_time_us":70}
W20260809 12:16:03.343394 25822 negotiation.cc:337] Failed RPC negotiation. Trace:
0809 12:16:03.328271 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59370 (local address 127.0.222.62:39441)
0809 12:16:03.328537 (+ 266us) server_negotiation.cc:206] Beginning negotiation
0809 12:16:03.328544 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0809 12:16:03.339197 (+ 10653us) server_negotiation.cc:407] Connection header received
0809 12:16:03.341711 (+ 2514us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 12:16:03.341717 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 12:16:03.341809 (+ 92us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 12:16:03.341883 (+ 74us) server_negotiation.cc:226] Negotiated authn=SASL
0809 12:16:03.343156 (+ 1273us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:59370: BlockingRecv error: recv got EOF from 127.0.0.1:59370 (error 108)
Metrics: {"server-negotiator.queue_time_us":126}
I20260809 12:16:03.355280 25737 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 12:16:03.347458 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47550 (local address 127.0.222.61:42263)
0809 12:16:03.347626 (+ 168us) server_negotiation.cc:206] Beginning negotiation
0809 12:16:03.347632 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0809 12:16:03.348159 (+ 527us) server_negotiation.cc:407] Connection header received
0809 12:16:03.348365 (+ 206us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 12:16:03.348371 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 12:16:03.348577 (+ 206us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 12:16:03.348738 (+ 161us) server_negotiation.cc:226] Negotiated authn=TOKEN
0809 12:16:03.349202 (+ 464us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 12:16:03.350388 (+ 1186us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 12:16:03.352423 (+ 2035us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 12:16:03.353115 (+ 692us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 12:16:03.353251 (+ 136us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 12:16:03.353824 (+ 573us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0809 12:16:03.354544 (+ 720us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0809 12:16:03.354692 (+ 148us) server_negotiation.cc:1035] Waiting for connection context
0809 12:16:03.354833 (+ 141us) server_negotiation.cc:299] Negotiation successful
0809 12:16:03.354920 (+ 87us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":62}
W20260809 12:16:03.364890 25825 negotiation.cc:337] Failed RPC negotiation. Trace:
0809 12:16:03.362683 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40474 (local address 127.0.222.60:37589)
0809 12:16:03.362881 (+ 198us) server_negotiation.cc:206] Beginning negotiation
0809 12:16:03.362887 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0809 12:16:03.363370 (+ 483us) server_negotiation.cc:407] Connection header received
0809 12:16:03.363567 (+ 197us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 12:16:03.363572 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 12:16:03.363687 (+ 115us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 12:16:03.363814 (+ 127us) server_negotiation.cc:226] Negotiated authn=SASL
0809 12:16:03.364676 (+ 862us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:40474: BlockingRecv error: recv got EOF from 127.0.0.1:40474 (error 108)
Metrics: {"server-negotiator.queue_time_us":61}
I20260809 12:16:03.376248 25737 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 12:16:03.369328 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47566 (local address 127.0.222.61:42263)
0809 12:16:03.369497 (+ 169us) server_negotiation.cc:206] Beginning negotiation
0809 12:16:03.369503 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0809 12:16:03.369956 (+ 453us) server_negotiation.cc:407] Connection header received
0809 12:16:03.370130 (+ 174us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 12:16:03.370135 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 12:16:03.370255 (+ 120us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 12:16:03.370370 (+ 115us) server_negotiation.cc:226] Negotiated authn=TOKEN
0809 12:16:03.370800 (+ 430us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 12:16:03.372027 (+ 1227us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 12:16:03.374067 (+ 2040us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 12:16:03.374829 (+ 762us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 12:16:03.374997 (+ 168us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 12:16:03.375453 (+ 456us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0809 12:16:03.375630 (+ 177us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0809 12:16:03.375727 (+ 97us) server_negotiation.cc:1035] Waiting for connection context
0809 12:16:03.375890 (+ 163us) server_negotiation.cc:299] Negotiation successful
0809 12:16:03.375971 (+ 81us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":58}
W20260809 12:16:03.387043 25825 negotiation.cc:337] Failed RPC negotiation. Trace:
0809 12:16:03.384815 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40486 (local address 127.0.222.60:37589)
0809 12:16:03.384991 (+ 176us) server_negotiation.cc:206] Beginning negotiation
0809 12:16:03.384998 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0809 12:16:03.385552 (+ 554us) server_negotiation.cc:407] Connection header received
0809 12:16:03.385757 (+ 205us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 12:16:03.385763 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 12:16:03.385855 (+ 92us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 12:16:03.385954 (+ 99us) server_negotiation.cc:226] Negotiated authn=SASL
0809 12:16:03.386779 (+ 825us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:40486: BlockingRecv error: recv got EOF from 127.0.0.1:40486 (error 108)
Metrics: {"server-negotiator.queue_time_us":62}
I20260809 12:16:03.404175 25737 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 12:16:03.397155 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47568 (local address 127.0.222.61:42263)
0809 12:16:03.397346 (+ 191us) server_negotiation.cc:206] Beginning negotiation
0809 12:16:03.397353 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0809 12:16:03.397428 (+ 75us) server_negotiation.cc:407] Connection header received
0809 12:16:03.397736 (+ 308us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 12:16:03.397741 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 12:16:03.397836 (+ 95us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 12:16:03.397931 (+ 95us) server_negotiation.cc:226] Negotiated authn=TOKEN
0809 12:16:03.398583 (+ 652us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 12:16:03.399904 (+ 1321us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 12:16:03.401910 (+ 2006us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 12:16:03.402596 (+ 686us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 12:16:03.402733 (+ 137us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 12:16:03.403255 (+ 522us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0809 12:16:03.403439 (+ 184us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0809 12:16:03.403536 (+ 97us) server_negotiation.cc:1035] Waiting for connection context
0809 12:16:03.403711 (+ 175us) server_negotiation.cc:299] Negotiation successful
0809 12:16:03.403809 (+ 98us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":61}
W20260809 12:16:03.421324 25825 negotiation.cc:337] Failed RPC negotiation. Trace:
0809 12:16:03.417129 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40488 (local address 127.0.222.60:37589)
0809 12:16:03.417338 (+ 209us) server_negotiation.cc:206] Beginning negotiation
0809 12:16:03.417346 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0809 12:16:03.419615 (+ 2269us) server_negotiation.cc:407] Connection header received
0809 12:16:03.419822 (+ 207us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 12:16:03.419828 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 12:16:03.419932 (+ 104us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 12:16:03.420064 (+ 132us) server_negotiation.cc:226] Negotiated authn=SASL
0809 12:16:03.421055 (+ 991us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:40488: BlockingRecv error: recv got EOF from 127.0.0.1:40488 (error 108)
Metrics: {"server-negotiator.queue_time_us":74}
I20260809 12:16:03.442181 25737 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 12:16:03.430870 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47582 (local address 127.0.222.61:42263)
0809 12:16:03.431167 (+ 297us) server_negotiation.cc:206] Beginning negotiation
0809 12:16:03.431174 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0809 12:16:03.435602 (+ 4428us) server_negotiation.cc:407] Connection header received
0809 12:16:03.435807 (+ 205us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 12:16:03.435813 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 12:16:03.435936 (+ 123us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 12:16:03.436047 (+ 111us) server_negotiation.cc:226] Negotiated authn=TOKEN
0809 12:16:03.436540 (+ 493us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 12:16:03.437704 (+ 1164us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 12:16:03.439757 (+ 2053us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 12:16:03.440476 (+ 719us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 12:16:03.440613 (+ 137us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 12:16:03.441359 (+ 746us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0809 12:16:03.441497 (+ 138us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0809 12:16:03.441594 (+ 97us) server_negotiation.cc:1035] Waiting for connection context
0809 12:16:03.441771 (+ 177us) server_negotiation.cc:299] Negotiation successful
0809 12:16:03.441870 (+ 99us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":163}
W20260809 12:16:03.451603 25825 negotiation.cc:337] Failed RPC negotiation. Trace:
0809 12:16:03.449834 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40500 (local address 127.0.222.60:37589)
0809 12:16:03.450070 (+ 236us) server_negotiation.cc:206] Beginning negotiation
0809 12:16:03.450076 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0809 12:16:03.450115 (+ 39us) server_negotiation.cc:407] Connection header received
0809 12:16:03.450187 (+ 72us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 12:16:03.450193 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 12:16:03.450308 (+ 115us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 12:16:03.450418 (+ 110us) server_negotiation.cc:226] Negotiated authn=SASL
0809 12:16:03.451366 (+ 948us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:40500: BlockingRecv error: recv got EOF from 127.0.0.1:40500 (error 108)
Metrics: {"server-negotiator.queue_time_us":118}
I20260809 12:16:03.476622 25737 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 12:16:03.468681 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47594 (local address 127.0.222.61:42263)
0809 12:16:03.468904 (+ 223us) server_negotiation.cc:206] Beginning negotiation
0809 12:16:03.468911 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0809 12:16:03.469524 (+ 613us) server_negotiation.cc:407] Connection header received
0809 12:16:03.469755 (+ 231us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 12:16:03.469778 (+ 23us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 12:16:03.469883 (+ 105us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 12:16:03.469984 (+ 101us) server_negotiation.cc:226] Negotiated authn=TOKEN
0809 12:16:03.470493 (+ 509us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 12:16:03.471764 (+ 1271us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 12:16:03.473909 (+ 2145us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 12:16:03.474671 (+ 762us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 12:16:03.474826 (+ 155us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 12:16:03.475710 (+ 884us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0809 12:16:03.475856 (+ 146us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0809 12:16:03.475970 (+ 114us) server_negotiation.cc:1035] Waiting for connection context
0809 12:16:03.476131 (+ 161us) server_negotiation.cc:299] Negotiation successful
0809 12:16:03.476229 (+ 98us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":82}
W20260809 12:16:03.486708 25825 negotiation.cc:337] Failed RPC negotiation. Trace:
0809 12:16:03.484355 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40514 (local address 127.0.222.60:37589)
0809 12:16:03.484545 (+ 190us) server_negotiation.cc:206] Beginning negotiation
0809 12:16:03.484552 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0809 12:16:03.485100 (+ 548us) server_negotiation.cc:407] Connection header received
0809 12:16:03.485306 (+ 206us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 12:16:03.485312 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 12:16:03.485419 (+ 107us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 12:16:03.485527 (+ 108us) server_negotiation.cc:226] Negotiated authn=SASL
0809 12:16:03.486477 (+ 950us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:40514: BlockingRecv error: recv got EOF from 127.0.0.1:40514 (error 108)
Metrics: {"server-negotiator.queue_time_us":68}
I20260809 12:16:03.516538 25737 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 12:16:03.509146 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47608 (local address 127.0.222.61:42263)
0809 12:16:03.509364 (+ 218us) server_negotiation.cc:206] Beginning negotiation
0809 12:16:03.509372 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0809 12:16:03.509966 (+ 594us) server_negotiation.cc:407] Connection header received
0809 12:16:03.510233 (+ 267us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 12:16:03.510247 (+ 14us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 12:16:03.510343 (+ 96us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 12:16:03.510456 (+ 113us) server_negotiation.cc:226] Negotiated authn=TOKEN
0809 12:16:03.511008 (+ 552us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 12:16:03.512221 (+ 1213us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 12:16:03.514219 (+ 1998us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 12:16:03.514942 (+ 723us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 12:16:03.515117 (+ 175us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 12:16:03.515663 (+ 546us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0809 12:16:03.515849 (+ 186us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0809 12:16:03.515958 (+ 109us) server_negotiation.cc:1035] Waiting for connection context
0809 12:16:03.516144 (+ 186us) server_negotiation.cc:299] Negotiation successful
0809 12:16:03.516252 (+ 108us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":74}
W20260809 12:16:03.530040 25825 negotiation.cc:337] Failed RPC negotiation. Trace:
0809 12:16:03.523657 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40528 (local address 127.0.222.60:37589)
0809 12:16:03.523870 (+ 213us) server_negotiation.cc:206] Beginning negotiation
0809 12:16:03.523878 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0809 12:16:03.528311 (+ 4433us) server_negotiation.cc:407] Connection header received
0809 12:16:03.528545 (+ 234us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 12:16:03.528554 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 12:16:03.528658 (+ 104us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 12:16:03.528796 (+ 138us) server_negotiation.cc:226] Negotiated authn=SASL
0809 12:16:03.529745 (+ 949us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:40528: BlockingRecv error: recv got EOF from 127.0.0.1:40528 (error 108)
Metrics: {"server-negotiator.queue_time_us":73}
I20260809 12:16:03.574152 25737 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 12:16:03.566783 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47622 (local address 127.0.222.61:42263)
0809 12:16:03.567031 (+ 248us) server_negotiation.cc:206] Beginning negotiation
0809 12:16:03.567037 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0809 12:16:03.567632 (+ 595us) server_negotiation.cc:407] Connection header received
0809 12:16:03.567854 (+ 222us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 12:16:03.567861 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 12:16:03.567966 (+ 105us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 12:16:03.568082 (+ 116us) server_negotiation.cc:226] Negotiated authn=TOKEN
0809 12:16:03.568566 (+ 484us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 12:16:03.569677 (+ 1111us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 12:16:03.571685 (+ 2008us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 12:16:03.572441 (+ 756us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 12:16:03.572581 (+ 140us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 12:16:03.573117 (+ 536us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0809 12:16:03.573303 (+ 186us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0809 12:16:03.573406 (+ 103us) server_negotiation.cc:1035] Waiting for connection context
0809 12:16:03.573620 (+ 214us) server_negotiation.cc:299] Negotiation successful
0809 12:16:03.573706 (+ 86us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":80}
W20260809 12:16:03.587165 25825 negotiation.cc:337] Failed RPC negotiation. Trace:
0809 12:16:03.584721 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40530 (local address 127.0.222.60:37589)
0809 12:16:03.584937 (+ 216us) server_negotiation.cc:206] Beginning negotiation
0809 12:16:03.584944 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0809 12:16:03.585469 (+ 525us) server_negotiation.cc:407] Connection header received
0809 12:16:03.585656 (+ 187us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 12:16:03.585662 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 12:16:03.585767 (+ 105us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 12:16:03.585888 (+ 121us) server_negotiation.cc:226] Negotiated authn=SASL
0809 12:16:03.586908 (+ 1020us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:40530: BlockingRecv error: recv got EOF from 127.0.0.1:40530 (error 108)
Metrics: {"server-negotiator.queue_time_us":73}
I20260809 12:16:03.661593 25737 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 12:16:03.654970 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47630 (local address 127.0.222.61:42263)
0809 12:16:03.655256 (+ 286us) server_negotiation.cc:206] Beginning negotiation
0809 12:16:03.655263 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0809 12:16:03.655636 (+ 373us) server_negotiation.cc:407] Connection header received
0809 12:16:03.655859 (+ 223us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 12:16:03.655863 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 12:16:03.655938 (+ 75us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 12:16:03.656036 (+ 98us) server_negotiation.cc:226] Negotiated authn=TOKEN
0809 12:16:03.656654 (+ 618us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 12:16:03.657491 (+ 837us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 12:16:03.659564 (+ 2073us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 12:16:03.660211 (+ 647us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 12:16:03.660335 (+ 124us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 12:16:03.660849 (+ 514us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0809 12:16:03.660972 (+ 123us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0809 12:16:03.661083 (+ 111us) server_negotiation.cc:1035] Waiting for connection context
0809 12:16:03.661230 (+ 147us) server_negotiation.cc:299] Negotiation successful
0809 12:16:03.661316 (+ 86us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":71}
W20260809 12:16:03.670035 25825 negotiation.cc:337] Failed RPC negotiation. Trace:
0809 12:16:03.667803 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40532 (local address 127.0.222.60:37589)
0809 12:16:03.668065 (+ 262us) server_negotiation.cc:206] Beginning negotiation
0809 12:16:03.668073 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0809 12:16:03.668382 (+ 309us) server_negotiation.cc:407] Connection header received
0809 12:16:03.668606 (+ 224us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 12:16:03.668611 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 12:16:03.668700 (+ 89us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 12:16:03.668837 (+ 137us) server_negotiation.cc:226] Negotiated authn=SASL
0809 12:16:03.669742 (+ 905us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:40532: BlockingRecv error: recv got EOF from 127.0.0.1:40532 (error 108)
Metrics: {"server-negotiator.queue_time_us":74}
I20260809 12:16:03.807212 25737 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 12:16:03.801869 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47642 (local address 127.0.222.61:42263)
0809 12:16:03.802141 (+ 272us) server_negotiation.cc:206] Beginning negotiation
0809 12:16:03.802149 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0809 12:16:03.802414 (+ 265us) server_negotiation.cc:407] Connection header received
0809 12:16:03.802646 (+ 232us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 12:16:03.802650 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 12:16:03.802741 (+ 91us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 12:16:03.802852 (+ 111us) server_negotiation.cc:226] Negotiated authn=TOKEN
0809 12:16:03.803313 (+ 461us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 12:16:03.804029 (+ 716us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 12:16:03.805390 (+ 1361us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 12:16:03.805799 (+ 409us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 12:16:03.806062 (+ 263us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 12:16:03.806471 (+ 409us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0809 12:16:03.806578 (+ 107us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0809 12:16:03.806680 (+ 102us) server_negotiation.cc:1035] Waiting for connection context
0809 12:16:03.806829 (+ 149us) server_negotiation.cc:299] Negotiation successful
0809 12:16:03.806913 (+ 84us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":95}
W20260809 12:16:03.815265 25825 negotiation.cc:337] Failed RPC negotiation. Trace:
0809 12:16:03.813406 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40548 (local address 127.0.222.60:37589)
0809 12:16:03.813643 (+ 237us) server_negotiation.cc:206] Beginning negotiation
0809 12:16:03.813650 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0809 12:16:03.813952 (+ 302us) server_negotiation.cc:407] Connection header received
0809 12:16:03.814091 (+ 139us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 12:16:03.814095 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 12:16:03.814164 (+ 69us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 12:16:03.814249 (+ 85us) server_negotiation.cc:226] Negotiated authn=SASL
0809 12:16:03.815050 (+ 801us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:40548: BlockingRecv error: recv got EOF from 127.0.0.1:40548 (error 108)
Metrics: {"server-negotiator.queue_time_us":62}
I20260809 12:16:04.080231 25737 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 12:16:04.075110 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47650 (local address 127.0.222.61:42263)
0809 12:16:04.075295 (+ 185us) server_negotiation.cc:206] Beginning negotiation
0809 12:16:04.075300 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0809 12:16:04.075664 (+ 364us) server_negotiation.cc:407] Connection header received
0809 12:16:04.075829 (+ 165us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 12:16:04.075833 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 12:16:04.075924 (+ 91us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 12:16:04.076013 (+ 89us) server_negotiation.cc:226] Negotiated authn=TOKEN
0809 12:16:04.076405 (+ 392us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 12:16:04.077121 (+ 716us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 12:16:04.078416 (+ 1295us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 12:16:04.078879 (+ 463us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 12:16:04.079036 (+ 157us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 12:16:04.079458 (+ 422us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0809 12:16:04.079575 (+ 117us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0809 12:16:04.079685 (+ 110us) server_negotiation.cc:1035] Waiting for connection context
0809 12:16:04.079877 (+ 192us) server_negotiation.cc:299] Negotiation successful
0809 12:16:04.079956 (+ 79us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":62}
W20260809 12:16:04.088253 25825 negotiation.cc:337] Failed RPC negotiation. Trace:
0809 12:16:04.086288 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40558 (local address 127.0.222.60:37589)
0809 12:16:04.086501 (+ 213us) server_negotiation.cc:206] Beginning negotiation
0809 12:16:04.086508 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0809 12:16:04.086703 (+ 195us) server_negotiation.cc:407] Connection header received
0809 12:16:04.086919 (+ 216us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 12:16:04.086923 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 12:16:04.087109 (+ 186us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 12:16:04.087242 (+ 133us) server_negotiation.cc:226] Negotiated authn=SASL
0809 12:16:04.088044 (+ 802us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:40558: BlockingRecv error: recv got EOF from 127.0.0.1:40558 (error 108)
Metrics: {"server-negotiator.queue_time_us":69}
I20260809 12:16:04.214733 25887 catalog_manager.cc:1259] Loaded cluster ID: 6412f4377ffd4262871cb4998b2fd7bb
I20260809 12:16:04.214937 25887 catalog_manager.cc:1552] T 00000000000000000000000000000000 P 865df34449d0455398e47baaff333baf: loading cluster ID for follower catalog manager: success
I20260809 12:16:04.217727 25887 catalog_manager.cc:1574] T 00000000000000000000000000000000 P 865df34449d0455398e47baaff333baf: acquiring CA information for follower catalog manager: success
I20260809 12:16:04.219908 25887 catalog_manager.cc:1602] T 00000000000000000000000000000000 P 865df34449d0455398e47baaff333baf: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260809 12:16:04.613564 26021 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 12:16:04.608396 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47666 (local address 127.0.222.61:42263)
0809 12:16:04.608961 (+ 565us) server_negotiation.cc:206] Beginning negotiation
0809 12:16:04.608966 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0809 12:16:04.608987 (+ 21us) server_negotiation.cc:407] Connection header received
0809 12:16:04.609176 (+ 189us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 12:16:04.609180 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 12:16:04.609266 (+ 86us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 12:16:04.609369 (+ 103us) server_negotiation.cc:226] Negotiated authn=TOKEN
0809 12:16:04.609758 (+ 389us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 12:16:04.610520 (+ 762us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 12:16:04.611806 (+ 1286us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 12:16:04.612265 (+ 459us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 12:16:04.612369 (+ 104us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 12:16:04.612815 (+ 446us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0809 12:16:04.612917 (+ 102us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0809 12:16:04.612987 (+ 70us) server_negotiation.cc:1035] Waiting for connection context
0809 12:16:04.613146 (+ 159us) server_negotiation.cc:299] Negotiation successful
0809 12:16:04.613233 (+ 87us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":456,"thread_start_us":311,"threads_started":1}
I20260809 12:16:04.625113 26030 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 12:16:04.619440 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40566 (local address 127.0.222.60:37589)
0809 12:16:04.619976 (+ 536us) server_negotiation.cc:206] Beginning negotiation
0809 12:16:04.619980 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0809 12:16:04.620061 (+ 81us) server_negotiation.cc:407] Connection header received
0809 12:16:04.620206 (+ 145us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 12:16:04.620210 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 12:16:04.620368 (+ 158us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 12:16:04.620469 (+ 101us) server_negotiation.cc:226] Negotiated authn=TOKEN
0809 12:16:04.620926 (+ 457us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 12:16:04.621660 (+ 734us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 12:16:04.622897 (+ 1237us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 12:16:04.623403 (+ 506us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 12:16:04.623499 (+ 96us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 12:16:04.623966 (+ 467us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0809 12:16:04.624489 (+ 523us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0809 12:16:04.624604 (+ 115us) server_negotiation.cc:1035] Waiting for connection context
0809 12:16:04.624669 (+ 65us) server_negotiation.cc:299] Negotiation successful
0809 12:16:04.624734 (+ 65us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":428,"thread_start_us":315,"threads_started":1}
I20260809 12:16:04.648928 26030 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 12:16:04.630749 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40582 (local address 127.0.222.60:37589)
0809 12:16:04.630982 (+ 233us) server_negotiation.cc:206] Beginning negotiation
0809 12:16:04.630990 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0809 12:16:04.642864 (+ 11874us) server_negotiation.cc:407] Connection header received
0809 12:16:04.643099 (+ 235us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 12:16:04.643111 (+ 12us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 12:16:04.643195 (+ 84us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 12:16:04.643305 (+ 110us) server_negotiation.cc:226] Negotiated authn=TOKEN
0809 12:16:04.643926 (+ 621us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 12:16:04.644644 (+ 718us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 12:16:04.647065 (+ 2421us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 12:16:04.647541 (+ 476us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 12:16:04.647634 (+ 93us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 12:16:04.648107 (+ 473us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0809 12:16:04.648251 (+ 144us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0809 12:16:04.648343 (+ 92us) server_negotiation.cc:1035] Waiting for connection context
0809 12:16:04.648512 (+ 169us) server_negotiation.cc:299] Negotiation successful
0809 12:16:04.648600 (+ 88us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":61}
I20260809 12:16:04.649101 26036 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 12:16:04.630767 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59384 (local address 127.0.222.62:39441)
0809 12:16:04.641862 (+ 11095us) server_negotiation.cc:206] Beginning negotiation
0809 12:16:04.641871 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0809 12:16:04.643091 (+ 1220us) server_negotiation.cc:407] Connection header received
0809 12:16:04.643406 (+ 315us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 12:16:04.643411 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 12:16:04.643573 (+ 162us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 12:16:04.643761 (+ 188us) server_negotiation.cc:226] Negotiated authn=TOKEN
0809 12:16:04.644322 (+ 561us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 12:16:04.645085 (+ 763us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 12:16:04.646773 (+ 1688us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 12:16:04.647252 (+ 479us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 12:16:04.647400 (+ 148us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 12:16:04.647964 (+ 564us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0809 12:16:04.648448 (+ 484us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0809 12:16:04.648558 (+ 110us) server_negotiation.cc:1035] Waiting for connection context
0809 12:16:04.648766 (+ 208us) server_negotiation.cc:299] Negotiation successful
0809 12:16:04.648847 (+ 81us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":10919,"thread_start_us":11112,"threads_started":1}
I20260809 12:16:04.650516 26021 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 12:16:04.631102 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47680 (local address 127.0.222.61:42263)
0809 12:16:04.631340 (+ 238us) server_negotiation.cc:206] Beginning negotiation
0809 12:16:04.631348 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0809 12:16:04.638406 (+ 7058us) server_negotiation.cc:407] Connection header received
0809 12:16:04.639173 (+ 767us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 12:16:04.639184 (+ 11us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 12:16:04.639316 (+ 132us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 12:16:04.639401 (+ 85us) server_negotiation.cc:226] Negotiated authn=TOKEN
0809 12:16:04.640394 (+ 993us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 12:16:04.641945 (+ 1551us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 12:16:04.646349 (+ 4404us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 12:16:04.647009 (+ 660us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 12:16:04.647118 (+ 109us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 12:16:04.648256 (+ 1138us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0809 12:16:04.648391 (+ 135us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0809 12:16:04.648503 (+ 112us) server_negotiation.cc:1035] Waiting for connection context
0809 12:16:04.650142 (+ 1639us) server_negotiation.cc:299] Negotiation successful
0809 12:16:04.650239 (+ 97us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":56}
I20260809 12:16:04.654354 888 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskgH5Pjm/build/asan/bin/kudu pid 25577
I20260809 12:16:04.655354 25646 generic_service.cc:196] Checking for leaks (request via RPC)
I20260809 12:16:04.769377 888 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskgH5Pjm/build/asan/bin/kudu with pid 25577
W20260809 12:16:04.795162 25676 connection.cc:535] server connection from 127.0.0.1:47516 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
I20260809 12:16:04.795701 888 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskgH5Pjm/build/asan/bin/kudu pid 25660
I20260809 12:16:04.796808 25727 generic_service.cc:196] Checking for leaks (request via RPC)
I20260809 12:16:04.913717 888 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskgH5Pjm/build/asan/bin/kudu with pid 25660
I20260809 12:16:04.938426 888 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskgH5Pjm/build/asan/bin/kudu pid 25740
I20260809 12:16:04.939612 25812 generic_service.cc:196] Checking for leaks (request via RPC)
I20260809 12:16:05.050778 888 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskgH5Pjm/build/asan/bin/kudu with pid 25740
2026-08-09T12:16:05Z chronyd exiting
[ OK ] ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys/1 (5337 ms)
[----------] 2 tests from ConnectToFollowerMasterTest (10362 ms total)
[----------] Global test environment tear-down
[==========] 56 tests from 4 test suites ran. (382911 ms total)
[ PASSED ] 55 tests.
[ FAILED ] 1 test, listed below:
[ FAILED ] SecurityITest.TestJwtMiniCluster
1 FAILED TEST